The AI Factory Is Specializing. The Network Must Too.
Last week, I joined Dylan Patel on stage at the RAISE Summit in Paris, one of Europe’s premier AI conferences. Thousands of founders, researchers, investors, hyperscalers and infrastructure builders gathered to discuss where AI is headed next. The conversations on stage were sharp, but the discussions in the hallways were just as revealing.
One theme surfaced again and again.
The industry is no longer asking how to build bigger AI clusters. It is asking how to build the right AI infrastructure for each workload. Whether I was speaking with cloud providers, model developers, enterprise AI teams or infrastructure vendors, the conversation kept shifting toward specialization, economics and measurable business outcomes.
It reinforced something I have been thinking about for some time. The AI factory is specializing. The network must too. Market imperatives and ROI considerations are compelling AI infrastructure to shift from general-purpose, one-size-fits-all architectures to specialized ones.
Today’s conversation is about designing the system around the use case, and the metric that determines whether that use case succeeds. A factory optimized for highest commodity token throughput is not the same as one optimized for real-time voice, interactive coding, long-context reasoning, or multi-agent workflows. The model may be similar. The infrastructure required to deliver the experience is not.
For the past several years, training shaped the reference architecture: huge accelerator pools, rail-optimized fabrics, and synchronized collectives. Inference initially inherited that infrastructure because it was already available.
That was the right starting point. It is increasingly the wrong default.
The analogy I keep returning to is the transition from homesteads to cities. Early homesteads combined the home, workplace, warehouse, and production facility. As activity scaled, those functions separated into factories, apartments, hotels, warehouses, and hospitals—each designed for a different occupant, operating model, and financial profile.
AI infrastructure is following the same path. Training clusters are the industrial zones. High-throughput inference factories are the high-density buildings, optimized for volume and utilization. Premium, high-interactivity factories are more like luxury towers: more expensive to build, but engineered to a different set of requirements. In the case of high interactivity speed, these include speed, responsiveness, and context.
The useful lesson is not the analogy itself. It is that architecture follows intended use, business case, and ROI.
One inference market, many factories
Inference is fragmenting into distinct use cases. Batch processing, commodity chat, coding assistants, real-time voice, enterprise copilots, long-context reasoning, and agentic systems do not want the same infrastructure.
The ranges below are directional, not public API prices or universal benchmarks. Actual results depend on the model, context length, quantization, hardware, batching, utilization, cache-hit rate, power cost, and target service level.
| Factory | Cost/1M | Tokens/s/user |
|---|---|---|
| Throughput | $.07–$.25 | 2–25 |
| General-purpose | $.25–$2 | 15–70 |
| High-interactivity | $.50–$20 | 30–150 |
Tokens/s/user (or agent) rises with interactivity because premium factories optimize per-user responsiveness, not aggregate batch throughput.
What is striking is that optimization can improve metrics by one or in some cases two order of magnitude!
A high-throughput factory can queue requests, build large batches, and trade individual responsiveness for aggregate efficiency. A high-interactivity factory may use smaller batches, reserve capacity, maintain more memory per session, provide more network headroom, and build in greater redundancy. Customers are willing to pay a lot more for tokens, and in return receive a faster and more predictable experience.
That is also the risk of the general-purpose cluster: too expensive to beat the throughput specialist and too slow to beat the premium provider. In trying to serve every workload, it risks being competitive in none.
High-interactivity inference puts the network on the critical path
High-interactivity inference is where specialization becomes most visible, and where the network becomes inseparable from application performance.
Users experience time to first token, the cadence of the tokens that follow, and the total time to a useful answer. A cluster can show strong aggregate throughput while feeling slow to each individual user. TTFT, inter-token latency, token rate per user, end-to-end latency, and tail performance all matter.
Consider what happens when one request reaches an advanced AI application.
It first crosses the front-end network: edge services, security layers, load balancers, authentication, rate limiting, and the API gateway. Before a model has processed a token, congestion or retries here can already increase time to first token.
The request then reaches an orchestrator. A simple application may invoke one model. An agentic application may send the query to a planner, which divides it into multiple branches: one agent retrieves enterprise data, another calls a reasoning model, another uses a coding or domain-specific model, another invokes a tool, and another verifies the result.
One request has now become a distributed workflow, each traversing the network independently. Every branch adds service-to-service traffic, scheduling decisions, and another opportunity for tail latency. The final response is often governed by the slowest required branch.
For each model, the assembled prompt enters that model’s own token space and is routed independently. A cache-aware router may try to place it on a worker that already holds reusable context, avoiding redundant computation.
The system then looks for the relevant KV cache. It may be in accelerator memory, host DRAM, local NVMe, or a distributed storage tier, accessed through storage network. If it exists, it must be found and retrieved. If it does not, the prompt goes through prefill to generate it. Modern serving systems can extend KV capacity beyond GPU memory through CPU RAM and disk, putting storage directly on the serving path.
Prefill processes the input context and creates the KV state used during generation. For a large model, the work may span several accelerators: scale-up networking within a system or rack-scale domain, and scale-out networking across nodes or racks.
In a disaggregated design, prefill and decode run in separate pools. The KV state must be transferred or exposed to a decode worker. That transfer can be large, bursty, and point-to-point. If it is delayed, the decode accelerator waits, TTFT rises, and expensive capacity sits idle. NVIDIA Dynamo and vLLM describe this basic sequence: prefill generates the KV cache, the cache moves to decode, and decode streams the response.
Decode then generates output sequentially. Each step reads model weights and KV state, produces the next token, updates the cache, and repeats. If the model is sharded, those steps may repeatedly traverse scale-up or scale-out fabrics. A small delay during prefill may be paid once; a delay inside distributed decode can be paid token after token, reducing tokens per second per user and creating uneven streaming.
The outputs return to the orchestrator, which may merge agent results, run a verifier, call another model, apply policy, and reconstruct the final response before streaming it back through the gateway.
A single query may therefore traverse the front-end network, service and control plane, storage network, scale-up fabric, scale-out fabric, and sometimes a scale-across network connecting clusters or regions. A disruption in any one can surface as the same user-visible symptom: slower first response, lower token rate, jitter, or a missed SLA.
Networking in advanced inference carries requests, context, cache state, model-parallel traffic, scheduling information, agent coordination, and the response itself.
Architecture is underwriting
An AI factory is a financially engineered asset.
A provider secures land and power, purchases accelerators, builds memory, storage, and networking systems, and models the revenue the capacity can generate. Every technical choice affects both the service the factory can deliver and the return it can earn.
A low-cost producer should not copy a premium provider. A power-constrained provider may optimize for tokens per watt. A sovereign provider may optimize for control and data residency. A premium, high-interactivity token producer will likely build a complex, optimized machine akin to a Formula One race car. Its cost per token will be higher, but customers will be paying significantly more - potentially an order of magnitude or two - for their tokens.
This should force first-principles questions. Does pure inference require the same rail-optimized fabric as training? Where is non-blocking bandwidth justified? Which networks can converge, and which require isolation? How should storage change now that KV cache is on the live serving path? How much spare capacity is needed to protect tail latency?
There is no universal design or optimization because there is no universal inference business. Once the factory is engineered around a particular metric and financial envelope, the operator must be able to observe whether it is actually delivering, optimize towards this metric, and intervene before a small impairment cascades across the request path.
What specialized factories require from the network
This is the problem we are building Aria Networks around. We call the approach Deep Networking: treating the network not as an isolated transport layer, but as an intelligent system spanning hardware, telemetry, hosts, workloads, and operations.
The purpose is not another dashboard. It is to connect network behavior to workload performance and business outcomes. Aria’s architecture brings network, host, and AI-workload signals into a shared context and applies specialized, layer-appropriate intelligence. Three capabilities are essential.
Fine-grain telemetry
You cannot underwrite - or optimize - what you cannot measure.
Average utilization is not enough. High-interactivity inference can be damaged by events that disappear inside coarse intervals: microbursts, transient queue buildup, packet loss, retransmissions, optical degradation, flow imbalance, or brief synchronization problems among model shards.
Those signals must be observed at the granularity at which they affect TTFT, KV-transfer time, inter-token latency, and tokens per second per user—and correlated with NIC, host, accelerator, storage, scheduler, and inference-engine state.
Telemetry is not about collecting more data. It is about creating the fact base needed to determine whether the factory is operating inside the envelope that was underwritten.
End-to-end visibility and unified control
Fine-grain data from one layer, or one network is still insufficient.
The same latency spike could originate at the gateway, in cache-aware routing, during storage retrieval, on the prefill-to-decode path, inside a scale-up domain, across the scale-out fabric, or in an overloaded decode worker. Separate dashboards, clocks, data models, and operating teams expose fragments and are not useful to determine causes.
Deep Networking requires a unified operational model across the request path: front end, storage, scale-up, scale-out, hosts, and workload. Signals need to share a timeline and context so operators can locate the issue, distinguish cause from symptom, and reason about remediation.
Visibility alone is also insufficient. Control must be coordinated so that routing, congestion management, workload placement, failover, and remediation do not work at cross-purposes. The network has to be observable end to end and operable as a whole.
An agentic hierarchy at every layer
The scale and speed of these systems exceed what humans can operate through alerts and dashboards alone.
The answer is not one generic agent above the network. It is a hierarchy of specialized agents, each working at the layer and timescale appropriate to the problem.
At the bottom is the network’s “reptile brain”: fast, bounded reflexes close to the hardware that respond to immediate conditions. Above that are domain agents that understand congestion, routing, optics, hosts, storage, KV movement, and workload behavior. They correlate evidence, test hypotheses, identify root cause, and recommend or execute controlled remediation.
At the highest layer, strategic agents reason across the factory’s objective function. They can help an operator understand the cost of a transient drop in interactivity, whether a recurring problem reflects configuration or architecture, and help compare strategies to remediate the issue.
The role of the agent evolves from reflex, to diagnosis, to reasoning, and ultimately to partnership with the operator. Humans remain in control, but work with systems that can investigate continuously and at machine scale. Aria’s approach similarly emphasizes specialized agents grounded in shared telemetry and guided action rather than disconnected generic chat.
The three pillars reinforce one another. Telemetry provides evidence. Unified, end to end context and control make the system understandable and operable. Agents turn understanding into timely action.
Without fine-grain telemetry, the agents are guessing. Without end-to-end context, they may optimize one layer while harming another. Without an agentic operating model, the volume and velocity of the data overwhelm the humans expected to use it.
The network must match the factory
The AI market is moving from generic architectures to purpose-built architectures. But a specialized factory cannot be operated with a standard network built for another era.
If the business is underwritten around low-cost tokens, the network must expose and remove wasted capacity. If it is underwritten around tokens per watt, the network must keep accelerators doing useful work inside a fixed power envelope. If it is underwritten around premium interactivity, the network must protect the entire critical path - from the API gateway, through cache and model execution, back to the user.
The answer is one programmable network platform that can support different objective functions simultaneously, applying different policies, priorities, and optimization strategies according to the workload.
This is where the Deep Networking approach becomes essential. Fine-grain telemetry provides the evidence; unified, end-to-end control provides a common operating model; and agents turn intent into continuous, adaptive optimization. The network can adapt dynamically - protecting latency-sensitive inference, maximizing throughput for batch workloads, or improving accelerator efficiency.
The central question is no longer simply, “How big is the cluster?” It is: What experience is this AI factory designed to deliver - and can its network measure, reason, adapt, and protect that experience end to end?
That’s the impetus behind Aria Networks, Inc.
