Blog

AI Gateway on Networking Hardware: Access Control for Small Private GPU Clusters

1. Small Private GPU Clusters: A Practical Starting Point

Building a private AI environment doesn’t necessarily mean deploying a massive GPU cluster. Open-weight models have become more widely available, and inference frameworks continue to mature, giving enterprises more options for running AI services on their own infrastructure.

For many enterprise applications, the goal is fairly straightforward: deploy existing models to support internal workflows. A knowledge assistant that retrieves information from company documents, a customer support system that generates responses, or an AI agent that interacts with internal tools can all benefit from private inference. Depending on model size, concurrency requirements, and performance targets, a cluster of two to four GPU servers can be a practical place to start.

Running models on private infrastructure also gives organizations greater control over where workloads execute and how the environment is maintained. That matters when AI applications handle sensitive business data, connect to internal databases, or require predictable access to dedicated computing resources.

But getting a model to respond to an API request is only the first step. Once multiple applications and departments begin sharing the same GPU resources, the questions extend well beyond whether inference is working. Who can access a particular model? How much capacity should each application be allowed to consume? What happens when a model moves to another server, or an automated agent suddenly starts submitting requests at a much higher rate than expected?

These are AI service access and governance challenges, and they show up long before an inference deployment grows to data-center scale.

2. From Model Endpoints to a Unified AI Entry Point

Consider an enterprise running several inference services across three GPU servers. One hosts a general-purpose large language model, another runs a specialized business model, and the third provides additional inference capacity.

Without a shared entry point, application teams have to maintain separate model endpoints, credentials, and backend configurations. Even a routine operational change, such as moving an inference service to another GPU server or adding a model replica, can force several application teams to update their configurations at the same time.

Access policies are equally difficult to manage. An internal knowledge assistant may only need a general-purpose chat service, while a financial application requires access to a separately controlled model. Maintaining those rules at individual inference endpoints creates more operational work and makes consistent policy enforcement harder.

Resource consumption presents another challenge. Inference requests can differ dramatically in their GPU compute and memory requirements. A short question may return a response almost immediately, while analyzing a lengthy document can involve thousands of input tokens and a much longer generation process. An AI agent caught in a retry loop can also generate enough requests to affect other applications sharing the cluster.

An AI Gateway establishes a common control point in front of these inference services. Applications connect through a consistent API interface, while the Gateway handles authentication, authorization, token-based rate limits, and model routing before forwarding requests to approved backends.

The external interface doesn’t need to expose the physical deployment behind it. An application, for example, can request a logical service called enterprise-chat without knowing the IP address of the server hosting the model. The Gateway maps that service to an approved backend model pool according to configured policies. If an application depends on a particular model or version, that requirement can remain part of its service contract rather than being silently overridden.

Logical service mapping also lays the foundation for Policy-Based Model Routing. Applications and departments can access model services according to their permissions and business requirements, while administrators maintain the mapping between those services and the available backend models and instances. Changes to the underlying deployment no longer have to trigger configuration changes across every consuming application.

For infrastructure teams, the result is a clearer boundary between business applications and inference resources, with access permissions, usage quotas, and request routing managed at a common entry point.

 Figure 1. Unified AI Access and Governance

3. Where the AI Gateway Runs: Hardware and Software Options

Where should the Gateway run?

Deploying a dedicated CPU server is a conventional choice. In smaller environments, administrators may instead run the Gateway in a container on an existing GPU server for convenience. That works, but it ties the AI service entry point to the resources and maintenance schedule of a single compute node. Rebooting that server for a GPU driver upgrade, for example, could interrupt API access even while the remaining inference servers continue operating normally.

Moving the Gateway onto separate hardware avoids this dependency. But if the deployment already includes a networking device that can accommodate the workload, adding another general-purpose server may be unnecessary.

Asteraix is exploring a more compact approach: running AI Gateway software directly on ET series Appliance that connects the inference environment. Access governance remains physically separate from GPU compute nodes, while network connectivity and application-layer Gateway services share the same hardware platform.

Two hardware options can support this architecture.

Figure 2. Two Hardware Options for Hosting an AI Gateway

ET Series Appliances

ET Series Appliances can be deployed at the network boundary between enterprise applications and a private GPU cluster. By running AI Gateway software in a compatible environment, the same device can provide network connectivity and act as a controlled entry point for model services.

For organizations already planning to deploy an ET appliance, this approach can eliminate the need for a separate server dedicated to AI service management, as long as the device has sufficient hardware resources for the intended workload.

COMe102-Based Smart Switch

Asteraix’s upcoming COMe102-based smart switch, equipped with eight 200G interfaces, offers another hardware option.

The key difference is where the device sits in the network topology. When the switch also interconnects backend GPU servers, it can forward traffic between compute nodes while running the AI Gateway application on its onboard COMe102 compute module. Network switching and AI service governance can reside on the same device.

That position in the network also opens up a useful operational perspective. Depending on hardware and software support, interface counters, link utilization, and congestion indicators can help administrators understand the network conditions between connected GPU servers.

ET appliances and COMe102-based switches are two parallel hardware choices. They address different physical network layouts, and neither depends on the other.

Software Deployment Options

Hardware is one decision. The software deployment model is another, with two approaches to consider.

Option 1: General-Purpose Linux Deployment

For teams looking to bring an AI service entry point online quickly, the open-source ecosystem already offers a broad selection of AI Gateways, API Gateway extensions, and model routing projects. These can run in compatible Linux, Docker, or Kubernetes environments without requiring a particular network operating system.

Several approaches have emerged:

  • Cloud-Native Envoy Extensions: Envoy AI Gateway, Higress, and kgateway build on the Envoy ecosystem, with a focus on unified traffic entry points, policy management, and security governance in Kubernetes environments.

  • Traditional API Gateway Evolution: Kong AI Gateway and Apache APISIX extend established API Gateway architectures with AI service management capabilities, allowing organizations to reuse existing microservices management and multi-tenant infrastructure.

  • High-Performance Rust/Go Routers: Bifrost, SGLang Router, Hydra, and Praxis emphasize lightweight deployment and efficient request handling, targeting environments with demanding throughput or latency requirements.

  • Python Application-Layer Ecosystem: LiteLLM and OneAPI (NewAPI) provide multi-model API compatibility and unified access, making them relevant to internal development, model integration, and rapid prototyping.

  • Compute- and KV Cache-Aware Routing: llm-d Router, AIBrix, and vLLM Router focus more closely on inference instance selection, KV Cache affinity, and request scheduling within GPU inference clusters.

These projects do not all cover the same functions. Some offer broader Gateway governance capabilities; others concentrate on inference request distribution and compute scheduling. The appropriate choice depends on the existing infrastructure, model serving architecture, and requirements for access control, token quotas, concurrency management, and inference scheduling.

The main advantage of this approach is the ability to draw from an established open-source software ecosystem while keeping the networking hardware and AI Gateway software relatively independent.

Option 2: AsterNOS-VPP Integration

The second approach is to integrate AI Gateway capabilities into Asteraix’s AsterNOS-VPP operating software.

AsterNOS-VPP already uses a containerized service architecture. The proposed integration follows that model by packaging the required AI Gateway functions into dedicated Docker containers, running alongside the network operating system’s existing service modules.

Under this design, AsterNOS-VPP continues to handle network connectivity, high-performance packet forwarding, and network-layer access control. AI/API traffic identified for Layer 7 inspection and governance can be steered to the appropriate AI Gateway Docker container for processing.

Once an HTTP/HTTPS request reaches the Gateway container, TLS termination or offloading can be handled as required by the deployment. The Gateway can then inspect HTTP headers, API paths, and model identifiers to enforce authentication, authorization, request rate limits, token quotas, and concurrency policies. Authorized requests are forwarded to the appropriate GPU inference services according to the model routing rules, with generated responses streamed back to the calling applications.

Here, the AI Gateway joins AsterNOS-VPP as another Docker-based service module rather than replacing its existing networking functions. Layer 2 and Layer 3 forwarding, ACLs, VRFs, and other network capabilities retain their respective roles, delivering connectivity and network security for the AI services.

Figure 3. AI Gateway Software Deployment Options

4. AI Governance and Network Security

When several internal teams share a private GPU cluster, access control becomes a practical engineering requirement. Exposing model endpoints across the corporate network may be convenient during early testing, but it offers little control over application permissions, isolation, or compute budgets.

An AI Gateway at the inference service entry point gives administrators a central place to enforce access policies and manage AI-specific requirements such as model selection and token consumption.

Identity, Access, and Model Policies

The first questions are who is making a request and which AI services that caller is allowed to use.

API keys and JWT-based authentication can identify calling applications or users, while TLS protects requests in transit. Authorization policies then determine which model services each identity can access.

An internal knowledge assistant, for example, might be permitted to use enterprise-chat, while an application handling sensitive business workflows connects to a separate service with stricter access requirements. Requests outside the caller’s permissions can be rejected before they reach the GPU cluster.

Model routing must respect the same authorization framework. Rather than allowing applications to connect directly to arbitrary inference endpoints, the Gateway selects approved model pools according to configured service mappings and access rules.

There is an important distinction here: model routing should never mean that the Gateway can silently substitute one model for another. If an upstream application depends on a particular model, version, or capability, those requirements belong in its service contract. Logical service names simplify backend operations, but the behavior and constraints that applications rely on must be preserved.

Managing Tokens, Rates, and Concurrency

AI inference requests vary widely in resource consumption, so conventional API rate limiting doesn’t tell the whole story.

A short request containing a few dozen tokens has very different compute requirements from one that submits a lengthy document and asks for a detailed report. Token-based controls, including Tokens Per Minute (TPM) and application-level quotas, offer a more accurate way to account for those differences.

Large language model APIs also commonly use Server-Sent Events (SSE) to stream generated text back to clients. A single streaming request may remain active for tens of seconds or longer, holding a connection and consuming backend compute resources throughout the inference process. Even when the rate of incoming requests looks modest, several long-running requests can keep a small inference cluster under sustained load.

Combining Requests Per Minute (RPM), Tokens Per Minute (TPM), token quotas, and concurrent request limits gives administrators multiple ways to manage shared inference capacity. Different applications can operate under separate usage policies instead of competing for the same compute pool without restrictions.

These Gateway-level controls govern which workloads are admitted to the cluster and under what limits. GPU scheduling, continuous batching, and memory management remain the responsibility of the backend inference engines.

Network Security Around the GPU Cluster

Even a Gateway with strong authentication and authorization offers limited protection if users can still connect directly to inference ports on backend GPU servers.

Network segmentation and access control rules can close those unintended access paths, requiring authorized business applications to reach model services through the managed Gateway entry point.

In an AsterNOS-VPP integrated deployment, existing network capabilities can complement the AI service policies. The AI Gateway determines whether a caller is authorized to use a model service. The underlying network, using functions such as ACLs and VRFs, controls which hosts can reach the backend infrastructure.

Application-level authorization and network-level reachability serve different purposes. Both are needed to establish an effective access boundary.

Controlling Outbound AI Traffic with GeoSite and GeoIP

A private AI cluster may need to do more than respond to internal applications. During normal operation, it may also access external model APIs, model repositories, embedding services, or third-party tools.

Controlling who can reach internal inference services addresses only one side of the security problem. In environments that allow outbound connections, administrators also need to decide which external destinations their systems are permitted to contact.

AsterNOS-VPP includes GeoSite and GeoIP traffic classification capabilities for this purpose. GeoSite classifies domains by website and application categories, as well as geographic groupings. GeoIP classifies IP addresses based on geographic location. Classification results can be applied directly to Access Control Lists (ACLs) and Policy-Based Routing (PBR).

Depending on the protocol and identification method, domain information can be extracted from several sources: the Host field in HTTP requests, Server Name Indication (SNI) during TLS handshakes, destination information carried by SOCKS5 and HTTP proxy protocols, and DNS query names when DNS Proxy is enabled.

Administrators can use these classification mechanisms to define network policies based on predefined domain groups.

For example, an enterprise could allow its GPU cluster to connect only to approved external AI service domains while blocking access to unauthorized services. Policy-Based Routing could also direct selected AI-related outbound traffic through a designated, compliant network egress.

GeoSite and GeoIP thus add another layer of control over outbound connections from AI infrastructure. The network restricts reachability to destination domains and IP addresses, while the AI Gateway manages caller identity, model permissions, and token consumption. Together, the two layers cover both network communication and application access policies.

Observability, Auditing, and AI-Specific Security

A unified AI Gateway also makes it easier to collect operational data across multiple model services.

Request logs, authentication results, response latency, backend health, and available token usage records help administrators understand how inference resources are being used. If an application’s compute demand suddenly spikes or authorization failures become frequent, Gateway logs offer a clear starting point for troubleshooting.

The same data supports longer-term capacity planning. Which logical model services generate the most traffic? How does resource consumption vary between applications? Where do requests most often fail? Answers to these questions help infrastructure teams decide whether the current deployment has enough capacity and when adjustments or expansion may be necessary.

Logging itself must follow appropriate data protection practices. API credentials and prompts containing confidential business information should not be captured indiscriminately or stored in plaintext.

Additional AI-specific security functions can be integrated as requirements evolve. Prompt-injection detection, sensitive data scanning, and input/output guardrails are possible directions, but they require dedicated detection engines or external policy services. Deploying an AI Gateway alone does not provide comprehensive protection against every attack targeting large language models.

For a small private cluster, the immediate priorities remain model access control, token consumption management, and visibility into shared AI services. Those capabilities also establish a foundation for more advanced security policies later.

5. One AI Gateway, Three GPU Servers: A Practical Example

To see how these pieces fit together, take a deployment with three GPU servers supporting several internal teams.

Two servers host independent replicas of the same general-purpose large language model. The third runs a different model for specialized tasks. An internal knowledge assistant, a document processing system, and several AI agents all need access to these models.

Instead of configuring individual GPU server IP addresses and credentials in every application, the infrastructure team exposes a unified AI service endpoint through an Asteraix networking platform.

When the knowledge assistant sends a request to enterprise-chat, the AI Gateway checks its credentials, verifies that the application is authorized to use the service, and applies the configured request rate and token limits. It then maps enterprise-chat to the appropriate backend model pool and selects an available replica according to the load-balancing policy.

The selected GPU server performs the inference. As tokens are generated, the response streams through the Gateway to the calling application. The Gateway records the request status and available usage data, while the application remains unaware of which physical server processed the request.

The document processing system can use a different logical service, such as enterprise-analysis, with its own permissions, model mappings, and usage quotas. Both applications share the same Gateway entry point but operate under separate governance policies.

Network controls further restrict which systems can reach the backend infrastructure. For example, AsterNOS-VPP can use ACLs and VRFs to isolate the inference network and prevent unauthorized hosts from bypassing the Gateway to connect directly to backend GPU servers.

If the cluster needs to access external model repositories or AI APIs, GeoSite/GeoIP rules can restrict the destination domains and IP ranges it is allowed to reach. PBR policies can also direct selected outbound traffic through designated network egress paths.

Backend changes are handled through the Gateway as well. When the infrastructure team adds an inference replica or moves a service to another GPU server, it can update the model mappings and backend configuration without changing the logical endpoint used by applications, as long as the new instance meets the established service requirements.

For hardware deployment, the architecture can use an ET series appliance at the network boundary of the cluster or a COMe102-based smart switch. The latter can interconnect the three GPU servers through high-speed switching interfaces while running the AI Gateway on its onboard compute module.

With the smart switch, supported network telemetry can also offer visibility into link health and traffic conditions between connected GPU servers.

The core inference workflow remains the same with either hardware option: applications request logical model services, the Gateway enforces access and routing policies, and GPU servers handle the actual inference computation.

Before moving into production, the design must also account for workload capacity, failure handling, and the Gateway’s own availability. Those decisions depend on the selected hardware platform and the requirements of the deployment.

6. Looking Ahead: Network-Aware Routing and Hybrid Inference

Hosting an AI Gateway on networking hardware brings model access governance closer to the network entry point of a private inference cluster. It also creates a way to correlate network conditions with AI service metrics.

An smart switch, for example, can expose network-level information that is difficult for a standalone Gateway server to observe directly. Link utilization, interface error counters, and congestion indicators can be analyzed alongside service-level metrics from inference engines. With the necessary hardware and software support, this information could help operators pinpoint performance bottlenecks and eventually inform network-aware model and instance routing.

Policy-based hybrid inference is another direction worth considering. An AI Gateway could enforce data-handling policies that keep confidential workloads on approved private models while allowing sanitized or less-sensitive requests to use authorized external cloud AI services. If local inference capacity becomes constrained, eligible workloads could also be routed to external compute resources, provided the relevant data protection and compliance requirements are met.

Both network-aware routing and hybrid inference go beyond basic model access control and request proxying. Implementing them requires reliable telemetry, clearly defined policies, and coordination with the systems responsible for inference scheduling and security decisions.

ET series appliances and COMe102-based smart switch platform offer two hardware options for hosting an AI Gateway at the network boundary or within the cluster’s switching layer. The software deployment path can then be selected according to platform compatibility and the requirements of the environment.

For organizations operating only a few GPU servers, a unified AI service entry point can already solve several practical problems. As more business applications begin sharing the same compute pool, model permissions, token quotas, outbound access controls, and network isolation become part of the infrastructure team’s everyday operational responsibilities.

Keep reading