Try the Gateway Console

Client & Problem Overview

The client runs a multi-region B2B software marketplace app platform supporting ~2,800 engineers globally. Their core product provides complex workflow automation integrations, connecting enterprise clients to CRM databases, ITSM ticket runners, and third-party SaaS pipelines.

As autonomous AI agents and developer copilots became integral to their core product offerings, the client faced an integration bottleneck. The engineering team was tasked with building custom, ad-hoc API integrations for each new tool. Because each tool had unique schemas, endpoint configurations, and authentication paradigms, onboarding a single third-party tool took an average of 14 business days.

Furthermore, developer agents executed unmonitored commands directly against databases and internal services. Without centralized audit logs, credential isolation, or prompt scanning, the security team lacked visibility into what data was being accessed, who authorized the execution, and whether outbound data contained sensitive Customer Identifiable Information (PII).

Challenges

To resolve this integration deadlock, the engineering platform team identified four primary technical blockers:

1. Schema Fragmentation: Every internal service and third-party connector exposed arbitrary endpoints (REST, gRPC, Webhooks). Engineers had to manually write mapping layers for LLM context structures, leading to fragile integrations.
2. Credential Proliferation: AI agents required direct access to sensitive developer databases and third-party API keys. Storing these credentials locally on developer clients or sharing them across shared agent sessions violated security protocols.
3. Lack of Rate Limiting: Destructive loop patterns in autonomous agents regularly triggered API rate limits on destination endpoints, leading to service disruption for human users.
4. Audit and Compliance Gaps: Traditional API logs recorded raw payload requests but failed to correlate human session authorization tokens with corresponding agent tool invocations, creating compliance gaps for SOC 2 Type II processing integrity audits.

Solution Approach

The platform engineering team deployed a unified Model Context Protocol (MCP) Gateway. MCP, standardizing client-server tool context sharing under mcp-jsonrpc/2.0, decoupled LLM reasoning agents from source code connections.

The gateway serves as a secure proxy layer. Instead of allowing agents to interface directly with database hosts, agents communicate exclusively with the gateway using Server-Sent Events (SSE). The gateway acts as a dynamic registry, mapping requested tool calls to the correct backend host, validating input arguments against JSON schemas, and scrubbing data before it leaves the corporate perimeter.

To drive developer self-service, the team introduced a developer portal where engineers can register their own custom tool schemas in under 5 minutes. The gateway automatically imports the tool schema, registers it in the active routing table, and exposes the tool to permissioned developer agent sessions.

Architecture

The gateway is built on a two-tier high-availability cluster deployed across multiple regions using Kubernetes and FastAPI.

Requests flow from LLM developer agents to the FastAPI Gateway edge. The gateway queries a highly available Redis cache layer to resolve the destination host registry and route mapping.

Two-Layer Authentication

To secure the boundary between human intent and automated tool execution, the architecture enforces a strict two-layer authentication scheme:

1. User session context validation: The human developer initiates the agent session using their Okta OIDC credential. The gateway asserts that this user has authorization to perform actions in the target workspace.
2. Agent principal check: The agent itself executes tool requests using short-lived JWT tokens provisioned for the specific session ID, mapping the exact call context back to the human initiator.

Unified Deployment Pipeline

New schemas must be vetted before promotion to the live production registry. The client built an automated validation pipeline:

Whenever an engineer commits a new tool configuration, the pipeline boots a isolated sandbox container, executes static schema validation, runs command validation filters, and registers the endpoints.

Observability and Audit Loops

Every tool invocation triggers a synchronous logging audit cycle. Latency curves, invocation volumes, and error counts are monitored in real time.

Outbound payloads are passed through a PII scrubber where values like API keys, emails, and database keys are redacted and replaced with secure, non-reversible hashes.

Implementation Steps

The rollout followed a four-step phased roadmap:

1. Gateway Bootstrap: Fast API nodes were deployed to Kubernetes using ingress controllers optimized for long-lived Server-Sent Events (SSE) connections.
2. Registry Mapping: Existing databases and ticket runner scripts were wrapped in small, lightweight Python-based MCP servers using the official SDK.
3. Credential Scoping: All database and API access keys were centralized in HashiCorp Vault. The gateway was mapped as the sole principal capable of retrieving these credentials.
4. Audit Log Hook: Outbound payload scrubbing filters were coded and hooked into the gateway middleware to intercept all JSON-RPC responses.

Here is a snippet of the custom validation middleware used to intercept outgoing JSON-RPC tools packages in Python:

class="tok-cm"># mcp_gateway_middleware.py
import re
from fastapi import Request, Response
from starlette.middleware.base import BaseHTTPMiddleware

class MCPScrubbingMiddleware(BaseHTTPMiddleware):
    class="tok-kw">def __init__(self, app, pii_patterns=None):
        super().__init__(app)
        self.pii_patterns = pii_patterns or [
            r&class="tok-cm">#039;class="tok-str">"api_key"\s*:\s*class="tok-str">"[^"]+class="tok-str">"',
            r&class="tok-cm">#039;"emailclass="tok-str">"\s*:\s*"[^class="tok-str">"]+"class="tok-str">',
            r&class="tok-cm">#039;"password"\s*:\s*"[^"]+"'
        ]

    async class="tok-kw">def dispatch(self, request: Request, call_next):
        response = await call_next(request)
        if response.media_type == class="tok-str">"application/json-rpc":
            body = bclass="tok-str">""
            async for chunk in response.body_iterator:
                body += chunk
            decoded_body = body.decode(class="tok-str">"utf-8")
            
            class="tok-cm"># Apply regex scrubbing logic
            for pattern in self.pii_patterns:
                decoded_body = re.sub(pattern, &class="tok-cm">#039;class="tok-str">"redacted": class="tok-str">"[SECURE_HASH]"', decoded_body)
                
            return Response(
                content=decoded_body.encode(class="tok-str">"utf-8"),
                status_code=response.status_code,
                headers=dict(response.headers),
                media_type=response.media_type
            )
        return response

In Go, we validated incoming JSON schemas recursively before allowing them to pass to execution hosts:

// schema_validator.go
package main

import (
	"encoding/json"
	"fmt"
	"github.com/xeipuuv/gojsonschema"
)

type MCPRequest struct {
	JSONRPC string          `json:"jsonrpc"`
	Method  string          `json:"method"`
	Params  json.RawMessage `json:"params"`
	ID      int             `json:"id"`
}

func ValidateParams(schemaStr string, params json.RawMessage) error {
	schemaLoader := gojsonschema.NewStringLoader(schemaStr)
	documentLoader := gojsonschema.NewBytesLoader(params)

	result, err := gojsonschema.Validate(schemaLoader, documentLoader)
	if err != nil {
		return err
	}

	if !result.Valid() {
		var errs string
		for _, desc := range result.Errors() {
			errs += fmt.Sprintf("- %s\n", desc.String())
		}
		return fmt.Errorf("schema validation failed:\n%s", errs)
	}
	return nil
}

Tech Stack

The operational components of the platform gateway registry are tabulated below:

Results & Outcomes

The implementation yielded measurable success across financial, operational, and security vectors:

• Onboarding Speed: Third-party integration cycles collapsed from 14 business days down to 6 hours. Instead of manually coding mapping logic, platform teams self-served tool schemas through the developer portal catalog.
• Developer Engagement: Portal monthly active users (MAU) surged from 12% to 61% by month 6, as engineers adopted pre-approved, security-cleared agent connections.
• Invocations Scale: Steady state monthly active invocations reached 48,212 sessions with an average end-to-end routing latency of 26ms (p95).
• Loop Prevention: Automated rate-limiting policies prevented cascading loop failures in developer copilots, eliminating agent-driven database outages.

Key Learnings

1. Decouple Logic from Transport: SSE proved more resilient than raw WebSockets for multi-region tool routing because standard HTTP headers simplified OAuth integration.
2. PII Redaction is a Day-One Requirement: Outbound telemetry streams must be filtered. If raw database values are allowed to bleed into centralized log collectors, auditing compliance becomes impossible.
3. Limit Loop Depth Programmatically: Developer agents will occasionally loop. Enforcing request-per-minute (RPM) limits at the gateway layer protects backend endpoints from sudden traffic spikes.

FAQ