How to Route AI Requests to the Right Model Like NVIDIA NeMo Switchyard and Runway's Model Router (2026)

August 31, 2026

Written by

Aerin Kim

How to Route AI Requests to the Right Model Like NVIDIA NeMo Switchyard and Runway's Model Router (2026)

NVIDIA's NeMo Switchyard and Runway's Model Router both shipped in August 2026. Here is how AI model routing actually works, from the research behind it to real production architecture.

Every AI product built in 2026 runs on more than one model, and almost nobody talks about how the request actually decides which model it lands on. A coding agent might need a frontier reasoning model for a hard refactor and a small, fast model for reformatting a file. A creative platform generating a video ad needs a script model, an image model, a video model, and a voice model, each one a completely different piece of software with a different cost, a different latency profile, and a different failure mode. Picking the right model for each piece of work, automatically, at the moment the request happens, is called model routing, and in August 2026 it went from an internal trick used by a handful of large labs to a documented, productized layer that two very different companies shipped in the same week.

On August 11, 2026, NVIDIA released Nemotron 3.5 Lightning alongside NeMo Switchyard, an open source library built specifically to route each step of an AI agent's workflow to whichever model actually fits that step [1] [2]. Around the same time, Runway's developer platform, Runway Dev, went wide with its own Model Router, a single API endpoint that picks between video, image, and audio generation models based on a cost, latency, or quality preference you set once [3] [4]. Two companies, two completely different product categories, agentic coding tools and generative media, arriving at the same architectural answer within days of each other is not a coincidence. It is a sign that routing has become the layer every serious AI system needs, the same way load balancers became something every serious web service needed once a single server stopped being enough.

This post walks through why that happened, what the research behind it actually says, how NeMo Switchyard and Runway's Model Router are built under the hood, and how to think about routing in your own AI or creative pipeline, whether that pipeline is a coding agent, a customer support bot, or a platform generating video, images, and music from a single idea the way Miraflow AI does.

model-routing-explained-nemo-switchyard-runway-router-2026-hero.png

Step 1: Why Model Routing Became a Real Engineering Problem

For most of the language model era, the default architecture was simple: pick your best available model, send every request to it, and pay whatever that costs. That approach breaks down for three separate reasons once a system has to run at real production scale.

Cost. A frontier reasoning model can cost ten to fifty times more per token than a small, efficient model tuned for a narrow task. Sending every request, including trivial ones like formatting a date or classifying a support ticket, through the most expensive model available is not a technical decision, it is a budget decision, and it is usually the wrong one. NVIDIA's own framing of Nemotron 3.5 Lightning is explicit about this: agentic systems spend most of their token budget on repetitive execution steps, tool calls, formatting, short lookups, not on the hard reasoning step that actually needs a frontier model, so paying frontier prices for execution work is pure waste [1].

Latency. A long-running agent that calls a model dozens of times per task accumulates latency fast. If every one of those calls goes through a large model with a multi-second time-to-first-token, the whole workflow becomes unusable for anything interactive. Nemotron 3.5 Lightning's whole design brief is the opposite end of that trade-off: NVIDIA reports roughly 4x faster output speed and about 30 percent faster completion on a 10,000-task agentic benchmark called PinchBench compared with similarly sized models, specifically because most agent steps do not need frontier-model depth, they need frontier-model speed at a fraction of the size [5] [6].

Capability fit. This is the one that generative media makes obvious in a way pure text systems do not. No single model is simultaneously the best video generator, the best image generator, and the best audio generator, and the best model in each category changes every few months as new releases ship. Runway's own explanation of why it built a router leads with exactly this problem: a config tied to a specific workflow, a fast in-app preview versus a paid final export, needs a different model each time, and hardcoding a specific model name into application code means rewriting that code every time a better model ships [3].

Put together, these three pressures describe an optimization problem, not a single best answer. Every request has a cost budget, a latency budget, and a capability requirement, and the model that best satisfies all three changes from request to request. That is precisely the problem a router is built to solve automatically, instead of a developer hardcoding one model choice for an entire application.

model-routing-explained-nemo-switchyard-runway-router-2026-cost-latency-quality.png

Step 2: The Research Behind Routing, From FrugalGPT to RouteLLM

Model routing is not a 2026 invention. It has a real academic lineage that is worth understanding before looking at how NVIDIA and Runway implemented it, because the production systems shipping today are direct descendants of ideas published starting in 2023.

FrugalGPT was one of the earliest and most cited papers to frame this explicitly as a cost optimization problem. Researchers at Stanford showed that a cascade, trying a cheap model first and only escalating to an expensive model when the cheap one's output looked unreliable, could match GPT-4 level accuracy on several benchmarks while cutting API cost by up to 98 percent in some configurations [7]. The core insight is that most queries in a real workload are easy for a cheap model to get right, and the hard part is reliably detecting which queries are the exceptions that need a stronger model.

RouteLLM, published in 2024, took a more direct approach: instead of a cascade, train a lightweight classifier on real human preference data to predict, before generation even starts, whether a strong model or a weak model is likely to produce a response a user would prefer. The RouteLLM authors reported that their best router could match GPT-4 level response quality while routing over 85 percent of queries to a much cheaper open-weight model, cutting costs by more than 75 percent on some benchmarks without materially hurting output quality [8]. Their key finding, that a small, well-trained classifier is enough to make this decision accurately most of the time, is the direct intellectual ancestor of the router logic sitting inside both NeMo Switchyard and Runway's Model Router today.

Amazon Bedrock's Intelligent Prompt Routing, announced in late 2024, was one of the first times this research left the paper stage and shipped as a managed cloud feature: a router that picks between models in the same family based on predicted response quality for each incoming prompt, aiming to cut costs by up to 30 percent with no perceptible quality loss. It proved the same idea works as a hosted product, not just a research benchmark.

What changed between 2024 and August 2026 is scope. The 2024-era research and Bedrock's initial feature focused narrowly on text-only LLM routing within a single provider's model family. NeMo Switchyard generalizes the idea to full multi-step agent workflows across any provider, and Runway's Model Router generalizes it further still, to routing across entirely different modalities, video, image, and audio, not just different sizes of the same kind of model [2] [3]. The underlying math, classify the request, estimate the cost and quality trade-off, pick accordingly, is the same lineage of idea scaled up to a much messier real-world surface.

model-routing-explained-nemo-switchyard-runway-router-2026-research-cascade.png

A worked example makes the FrugalGPT result concrete. The paper's headline result is up to 98 percent cost reduction on some benchmark configurations while matching GPT-4-level accuracy [7]. Picture an agent workload running 100,000 requests a day where a frontier model costs roughly 40 times more per call than a small execution-layer model. Sending every request through the frontier model at that ratio is the expensive default most teams start with. A cascade that resolves 90 percent of requests correctly on the first, cheap pass and escalates only the remaining 10 percent to the frontier model cuts total spend by roughly 85 to 90 percent relative to the all-frontier baseline, without ever serving a worse answer than the frontier model would have given on the escalated slice, since those are exactly the requests that get escalated. That is the mechanism behind FrugalGPT's headline number, not a black box, a cascade only pays frontier prices for the fraction of traffic that actually needs frontier capability.

RouteLLM's contribution on top of that is replacing "try cheap, escalate on failure" with "predict up front whether cheap will work," which avoids paying for the cheap model's wasted attempt on requests that were always going to need escalation. Both ideas show up inside NeMo Switchyard and Runway's Model Router, Switchyard's policy engine leans toward the predictive RouteLLM style, using session context to decide up front, while a naive router without that context still benefits from a FrugalGPT-style fallback path, which is exactly why Step 8 below treats a fallback path as non-negotiable rather than optional.

If you want to see the classifier-based routing idea in a runnable form before looking at the production systems, here is a minimal version of the logic RouteLLM-style routers are built around:

python
/code
# Minimal RouteLLM-style routing logic: a lightweight classifier decides
# whether a request needs a strong model or can go to a cheap one.

def route_request(prompt: str, classifier, threshold: float = 0.5) -> str:
    """Returns 'strong' or 'weak' based on a predicted win-rate score."""
    win_rate_if_weak = classifier.predict_win_rate(prompt)
    if win_rate_if_weak >= threshold:
        return "weak"   # cheap model is predicted to perform acceptably
    return "strong"      # escalate to the expensive frontier model

# Example usage
decision = route_request(
    prompt="Summarize this changelog into three bullet points.",
    classifier=trained_preference_classifier,
    threshold=0.5,
)
model_name = "nemotron-3.5-lightning" if decision == "weak" else "frontier-reasoning-model"

Step 3: Inside NVIDIA NeMo Switchyard's Architecture

NeMo Switchyard is open source, and its own architecture documentation is unusually direct about what it actually is: a proxy that sits between an application's inference calls and the models that fulfill them, functioning as a routing layer rather than a model itself [9]. Four pieces make up the system.

A provider-agnostic SDK. The library, called switchyard-libsy, defines the set of models available to a system using semantic names rather than hardcoded provider strings. An application asks for something like "the fast execution model" or "the reasoning model," and a mapping layer resolves that semantic name to a concrete provider endpoint and model ID behind the scenes. This is the detail that actually matters for maintainability: swapping the model behind "the fast execution model" from one vendor's checkpoint to another does not require touching the application code that calls it, only the mapping configuration [9].

Format translation. Every inbound request gets decoded into a provider-neutral internal representation before any routing decision happens, then re-encoded into whatever format the selected target model actually expects. This is what lets Switchyard route between models from different vendors that do not share an API schema, without the application ever needing to know that translation is happening [9] [10].

Session-aware state. Unlike a stateless load balancer, Switchyard can carry routing context across an entire agent session. If an earlier turn already established that this session's task benefits from a specific model's tool-calling behavior, that context is available to influence the routing decision on later turns instead of every call being evaluated in isolation [10].

Policy-driven selection. At runtime, a router evaluates each request against the available context and a configured policy, then sends the work to whichever model best satisfies that policy's requirements, constraints, and cost target. NVIDIA's own description of the routing behavior is that heavy reasoning steps get routed to large frontier models while repetitive, high-volume execution steps get pushed to small, fast local models like Nemotron 3.5 Lightning [10] [11].

The Register's coverage of the release framed the business motivation plainly: enterprise AI spend has been climbing faster than the value teams are getting out of it, largely because so much routine agent work has been running through models sized for much harder problems, and Switchyard is NVIDIA's answer to that specific waste [11].

Here is what a minimal Switchyard-style routing config looks like conceptually, based on the documented architecture, defining semantic model names and a policy that maps request types to those names:

json
/code
{
  "models": {
    "fast-execution": {
      "provider": "nvidia-nim",
      "model_id": "nemotron-3.5-lightning-30b-a3b",
      "max_cost_per_1k_tokens": 0.02
    },
    "deep-reasoning": {
      "provider": "openai",
      "model_id": "frontier-reasoning-model",
      "max_cost_per_1k_tokens": 1.50
    }
  },
  "routing_policy": {
    "default_target": "fast-execution",
    "escalate_when": [
      "tool_call_failed_twice",
      "task_type == 'multi_step_planning'",
      "requires_long_context_reasoning"
    ]
  }
}

And here is a real, runnable call to Nemotron 3.5 Lightning itself, the model Switchyard is most often configured to route execution-layer work toward, through OpenRouter's hosted API:

bash
/code
curl https://openrouter.ai/api/v1/chat/completions \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/nemotron-3.5-lightning",
    "messages": [
      {"role": "user", "content": "Reformat this JSON log line into a one-sentence summary."}
    ]
  }'

model-routing-explained-nemo-switchyard-runway-router-2026-switchyard-architecture.png

Step 4: Nemotron 3.5 Lightning, the Model Switchyard Was Built to Feed

Understanding Switchyard's design goal is easier once you know what Nemotron 3.5 Lightning actually is, since the two shipped together deliberately.

Nemotron 3.5 Lightning is a 30 billion parameter mixture-of-experts model with only 3 billion parameters active per token, small enough to run on a single consumer GPU, and licensed for free commercial use and modification without a separate agreement [6] [12]. Its architecture is a hybrid: interleaved Mamba-2 state-space layers and mixture-of-experts feedforward blocks, with a smaller number of traditional attention layers mixed in, a pattern that traces back to the original Mixture-of-Experts formulation for neural networks [13] and the state-space Mamba architecture published in late 2023 [14], combined the way NVIDIA's own Nemotron 3 family has been doing across its recent releases [5].

Two specific numbers make the case for why this model is exactly what an execution-layer router wants to route toward. On PinchBench, NVIDIA's 10,000-task agentic benchmark, Nemotron 3.5 Lightning scores 85.37, with NVIDIA claiming roughly 4x the throughput and 30 to 35 percent faster task completion than comparably sized Qwen models on the same benchmark [5]. On more general reasoning benchmarks it posts SWE-bench Verified 51.56, GPQA Diamond 75.44, and MMLU Pro 81.94, respectable for a 3B-active-parameter model but clearly not frontier-tier next to a 500B-plus dense or MoE reasoning model [5] [15]. That gap is the point. Nemotron 3.5 Lightning is not trying to win a leaderboard against frontier reasoning models, it is trying to be the fastest, cheapest model that is still good enough for the execution steps a router hands it, and a router only works well if the small model it escalates away from is genuinely strong at the narrow thing it is asked to do.

Part of that speed comes from a built-in multi-token predictor and support for speculative decoding through two NVIDIA-built variants, D-Flash and D-Spark, both derived from techniques DeepSeek popularized for accelerating autoregressive generation without changing the model's output distribution [5]. Speculative decoding itself traces back to two independent 2023 papers, one from Google Research and one from DeepMind, both showing that a small draft model can propose several tokens ahead and a larger model can verify them in a single forward pass, cutting wall-clock generation time by 2 to 3x with no change to output quality [16] [17]. Layering that technique on top of an already-small MoE model is what gets Nemotron 3.5 Lightning to the throughput numbers NVIDIA is advertising.

model-routing-explained-nemo-switchyard-runway-router-2026-nemotron-architecture.png

If you want a standalone image to illustrate this branching-switch idea for your own deck or post, here is a Nano Banana style prompt built around the same switchyard metaphor used throughout this piece:

Pastel editorial illustration, a single glowing signal splitting through a mechanical switch mechanism into three separate labeled paths of different widths, the widest path leading to a small fast gear icon, the narrowest leading to a large glowing star icon representing a frontier model, soft pastel palette of dusty blue and coral, clean editorial linework, no readable text, no logos, no people, gender neutral, scientifically accurate depiction of a branching switch mechanism.

Step 5: Inside Runway's Model Router for Generative Media

Runway's Model Router solves a version of the same problem in a domain where the stakes of picking the wrong model are more visible, because the output is an image, a video, or a piece of audio a human is going to look at or listen to directly.

The router works in two steps, and Runway's own documentation lays this out precisely. First, it narrows the full model catalog down to only the models that could satisfy a given request: models you have explicitly enabled, models that support the specific capabilities the request needs, and models that would stay under any price cap you have configured for that modality. Second, among whatever survives that filter, it picks the single best option according to a preference you set once, cost, latency, or quality, rather than a preference you re-specify on every call [3].

Configuration happens once, not per request. Inside the Runway Dev portal, you set your optimization preference, name the config, set a price cap, and allow or deny specific models or providers, then save that as a reusable config tied to a specific use case, a fast, cheap config for in-app live previews and a separate, higher-quality config for a paid final export are Runway's own example of two configs living side by side for the same product [3]. From that point on, the application calls one endpoint with the config ID attached and never has to name a specific model again.

What makes this router structurally different from a text-only LLM router like RouteLLM is that it operates across modalities inside one system. The same config can serve video, image, and audio requests, since modality is implied by which endpoint you call, /v1/generate/video, /v1/generate/image, or /v1/generate/audio, rather than being a separate router per modality [3]. The response comes back with metadata identifying exactly which model was actually used and why it was selected, which matters for debugging and for understanding cost after the fact, not just at request time [3] [18].

Here is what a Runway Model Router config looks like as a real request body, based on Runway's own documented API shape:

json
/code
{
  "config_name": "fast-preview",
  "optimize_for": "latency",
  "price_cap_usd": 0.05,
  "allowed_models": ["*"],
  "denied_providers": [],
  "modality_defaults": {
    "video": { "optimize_for": "latency" },
    "image": { "optimize_for": "quality" },
    "audio": { "optimize_for": "cost" }
  }
}

And a matching call to the router endpoint using that config, instead of naming a specific model directly:

bash
/code
curl https://api.dev.runwayml.com/v1/generate/video \
  -H "Authorization: Bearer $RUNWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "router_config_id": "fast-preview",
    "prompt": "A slow dolly-in on a coffee cup steaming on a wooden table, morning light",
    "duration_seconds": 4
  }'

model-routing-explained-nemo-switchyard-runway-router-2026-runway-router-flow.png

Laid side by side, the four routing systems this post covers, two research systems and two production systems, share the same basic shape but differ in scope and where the routing decision actually happens.

System	Scope	Decision method	Cross-provider	Cross-modality
FrugalGPT (2023)	Text LLM cost cascade	Cascade with escalation on low confidence	Yes	No
RouteLLM (2024)	Text LLM quality routing	Trained preference classifier	Yes	No
NeMo Switchyard (2026)	Full agent workflows	Policy engine with session context	Yes	No (text/tool calls)
Runway Model Router (2026)	Generative media	Constraint filter + preference scoring	Yes	Yes (video, image, audio)

The pattern in that table is worth sitting with for a moment. Every system decides between a smaller number of candidates using some notion of a policy or a trained classifier, and every system except a plain cascade evaluates that decision before generation starts rather than after. The two 2026 production systems, Switchyard and Runway's Model Router, both add a fourth dimension the 2023 and 2024 research did not need to handle, format and provider translation, because a real production router has to talk to models that were never designed to share an interface.

Step 6: Building a Minimal Router Yourself

You do not need NeMo Switchyard's full session-state machinery or Runway's model catalog to benefit from routing. Most teams get a meaningful chunk of the cost and latency win from a much simpler heuristic router, and understanding the minimal version makes the production systems easier to reason about.

The simplest useful router has three parts: a cheap classification step that estimates task difficulty, a policy that maps difficulty bands to specific models, and a fallback path for when the cheap model's output looks unreliable, which is the FrugalGPT cascade idea from Step 2 applied directly [7]. Here is a working example in Python that routes a coding-agent-style workload between a small local model and a frontier model based on a simple complexity heuristic:

python
/code
# A minimal heuristic router for a coding-agent-style workflow.
# Routes based on a cheap complexity estimate, with a fallback for
# low-confidence outputs, following the FrugalGPT cascade pattern.

import re

def estimate_complexity(task: str) -> str:
    long_task = len(task.split()) > 120
    mentions_multi_file = bool(re.search(r"\b(refactor|multiple files|architecture)\b", task, re.I))
    if long_task or mentions_multi_file:
        return "hard"
    return "easy"

def route(task: str) -> str:
    complexity = estimate_complexity(task)
    return "frontier-reasoning-model" if complexity == "hard" else "nemotron-3.5-lightning"

def run_with_fallback(task: str, low_confidence_threshold: float = 0.4):
    model = route(task)
    result, confidence = call_model(model, task)
    if model != "frontier-reasoning-model" and confidence < low_confidence_threshold:
        # Cascade escalation, same idea FrugalGPT introduced in 2023
        result, confidence = call_model("frontier-reasoning-model", task)
    return result

This kind of router will not match RouteLLM's benchmark numbers, since it has no trained classifier behind it, but it captures the same shape of decision that both NeMo Switchyard and Runway's Model Router are making at a larger scale: look at what the request actually needs, and only pay for more capability than that when the request genuinely calls for it. The trained-classifier version, the RouteLLM approach from Step 2, is the natural next step once you have logged enough real request and outcome data to train one [8].

model-routing-explained-nemo-switchyard-runway-router-2026-minimal-router-diagram.png

Step 7: Where Routing Fits Into a Real Creative Production Pipeline

The clearest way to see why this matters outside of a pure coding-agent context is to walk through a full creative workflow that already, whether or not the product calls it "routing," has to make the exact model-selection decision Switchyard and Runway's Model Router formalize.

Turning one idea into a finished piece of content is not one model's job. A script needs a language model tuned for writing and structure. A scene needs an image or video generation model tuned for visual fidelity and prompt adherence. A voiceover needs a text-to-speech model tuned for natural intonation. A thumbnail needs an image model tuned for composition and text legibility at a small size. Background music needs a music generation model entirely separate from all of the above. This is exactly the idea-to-script-to-visual-to-video-to-thumbnail-to-music pipeline that platforms like Miraflow AI run under the hood, and it is structurally the same multi-model orchestration problem NeMo Switchyard solves for agent execution steps and Runway's Model Router solves for generative media requests, just expressed as a full content pipeline instead of a single API call [3].

A platform that hardcodes one model for every step of that pipeline runs into the exact three problems from Step 1. It overpays by sending simple formatting or short-clip work through a large, expensive model. It gets slow, since a heavy model on every step compounds latency across a five- or six-stage pipeline. And it caps its own quality ceiling, since the single model it committed to will never be the single best choice for scriptwriting, image generation, video generation, voice, and music all at once, because no such model exists. Routing each stage to the model built specifically for that stage, the way Text2Shorts generates a script and then hands scene visuals to a dedicated visual generation step, or the way the AI Image Generator and cinematic AI video generator are separate, purpose-built tools rather than one model doing everything badly, is the same underlying architecture idea this entire post has been describing at the infrastructure level.

model-routing-explained-nemo-switchyard-runway-router-2026-pipeline-routing.png

Walk through a single request to see why this matters in practice. A creator types one topic into Text2Shorts and asks for a 45-second vertical video. The stage that turns that topic into a script is a language task, structure, hooks, pacing, the kind of work a routing-aware system would send to a model tuned for writing rather than one tuned for pixel-level visual fidelity. The stage that turns each script beat into a scene is a visual generation task with an entirely different quality bar, prompt adherence, composition, lighting consistency across scenes. The voiceover stage cares about prosody and natural pacing, not visual fidelity or writing structure at all. A thumbnail generated afterward optimizes for something narrower still, legibility and contrast at a small size, closer to the constraint-driven filtering step in Runway's two-stage router than to open-ended generation. Four stages, four different optimization targets, and treating them as one undifferentiated "make content" step rather than four routed sub-problems is exactly the mistake Step 1 describes at the infrastructure level, just visible here as a finished video that looks slightly worse in every dimension at once instead of being excellent in any one of them.

Here is a Wan-style video generation prompt built around the same switchyard visual, useful as a standalone hero clip for a post or presentation about this exact idea:

Wan-style cinematic video prompt: slow overhead drone shot gliding above a glowing miniature railway switchyard at dusk, single bright light pulse traveling down the main track and splitting smoothly across branching tracks toward three distinct softly lit terminal stations, gentle blue and amber lighting, smooth continuous camera motion, no readable text, no logos, no people, 6 second loopable clip.

For a creator building an AI clipping workflow specifically, the same logic applies one level down. AI Clipping has to transcribe a long video, score moments for virality, crop for vertical format, and generate captions, four genuinely different sub-tasks that benefit from different specialized processing rather than one model handling every step with equal quality. Understanding routing as a general pattern, not a feature specific to NVIDIA or Runway, makes it obvious why more of the tools creators already use are built this way under the hood, even when the product itself never uses the word "router."

Step 8: Production Best Practices and Common Mistakes

A few patterns separate a router that actually saves money and latency from one that quietly makes a system worse.

Log every routing decision, not just the model's output. Without a record of which model handled which request and why, you cannot audit whether the router is making good decisions or debug a quality regression after the fact. Runway's router returns model-selection metadata with every response specifically so this is possible without extra instrumentation [3].

Set a real fallback path, not just a preferred model. Every router implementation covered in this post, FrugalGPT's cascade, RouteLLM's classifier, Switchyard's policy engine, and Runway's constraint filter, includes an explicit path for when the first-choice model is unavailable or its output looks wrong [7] [8] [9] [3]. A router with no fallback is a single point of failure disguised as a cost optimization.

Do not route on cost alone. The FrugalGPT results and RouteLLM's benchmarks are impressive specifically because they hold output quality roughly constant while cutting cost, not because they cut cost at the expense of quality [7] [8]. A router tuned purely to minimize spend, with no quality signal in the loop, will degrade user-facing output quality in ways that cost far more in churn than the routing saved in API spend.

Revisit routing policies as new models ship. The entire reason Runway built a router instead of hardcoding a model name into its API is that the best available model changes every few months [3]. A routing policy configured once and never revisited slowly drifts from optimal as better, cheaper models become available for tasks the policy still sends to an older, more expensive option.

Watch for latency added by the router itself. A routing decision that requires its own model call, a classifier inference, adds latency before the actual work even starts. NeMo Switchyard's design keeps this overhead intentionally small by using a lightweight policy evaluation rather than a heavy model call for the routing decision itself [9], and any router you build yourself should hold itself to the same bar.

model-routing-explained-nemo-switchyard-runway-router-2026-best-practices.png

Step 9: Production Architecture for Putting a Router in Front of Your Stack

Everything above explains why routing works and how the two major August 2026 systems are built. Actually operating one in production adds a few architectural concerns that research papers and vendor documentation tend to underweight.

Treat the router as a critical-path service, not a side utility. Every request in a routed system passes through the router before it reaches a model, which means a router outage is a full outage, not a degraded-mode fallback. Both NeMo Switchyard's proxy design and Runway's Model Router are built to sit directly in the request path for this reason, so they need the same uptime and monitoring discipline as the models behind them, not less [9] [3].

Instrument routing decisions as a first-class metric. Track, at minimum, which model handled each request, the routing policy or classifier confidence behind that choice, the resulting latency, and whether a fallback or escalation fired. Runway's router returns exactly this metadata with every response so it never has to be reconstructed after the fact [3], and any router you build should log the equivalent even if the underlying platform does not hand it to you automatically.

Canary new models through the router before making them the default. Since a routing policy is just configuration, not application code, a new model release can be tested by routing a small percentage of real traffic to it and comparing outcomes against the incumbent, before flipping the default. This is the single biggest operational advantage of a router-based architecture over one where a model name is hardcoded directly into application logic, swapping models becomes a config change instead of a code change and a redeploy.

Separate the routing policy from the model catalog. Keep the list of available models, and how to reach each one, as a distinct piece of configuration from the rules that decide which model a given request should hit. This is precisely the separation Switchyard's semantic-name mapping layer enforces [9], and it is what makes a routing policy portable across model generations instead of needing a rewrite every time a new model ships.

Conclusion

Model routing stopped being a research curiosity the moment two very different companies shipped production routing layers in the same week for two very different reasons. NVIDIA built NeMo Switchyard because agentic AI workloads waste enormous budget sending repetitive execution steps through frontier-priced models, and paired it with Nemotron 3.5 Lightning, a model purpose-built to be the fast, cheap target that router sends work toward. Runway built its Model Router because no single generative media model is the best choice for every video, image, and audio request, and the best option changes too often to hardcode. Both systems are direct descendants of research that started with FrugalGPT's cost-aware cascades in 2023 and RouteLLM's learned classifiers in 2024, scaled up from routing between sizes of the same text model to routing across entire agent workflows and entire generative media modalities. The pattern underneath all of it is the same one creative platforms already apply without necessarily calling it routing: send each piece of work to the model actually built for that piece of work, and let a system, not a hardcoded default, make that call automatically. Whether you are building an agent, a content pipeline, or generating your next video with Miraflow AI, the underlying question is identical, and 2026 is the year the tooling to answer it well finally caught up to the problem.

Frequently Asked Questions

What is AI model routing in simple terms? Model routing is the practice of automatically sending each individual request to whichever AI model best fits that specific request's cost, speed, and capability requirements, instead of sending every request through the same single model regardless of how simple or complex it is.

Is NeMo Switchyard only for NVIDIA models? No. Switchyard is provider-agnostic by design, its SDK maps semantic model names to whatever provider endpoint you configure behind them, so it can route across models from multiple vendors, not just NVIDIA's own Nemotron family, though Nemotron 3.5 Lightning is the model NVIDIA specifically built and released alongside it as the default fast execution target.

How is Runway's Model Router different from a text-based LLM router like RouteLLM? RouteLLM and similar text routers choose between different sizes or providers of language models for a single modality, text generation. Runway's Model Router operates across modalities entirely, the same router and config can serve video, image, and audio generation requests, with modality determined by which API endpoint is called rather than a separate router per media type.

Does using a router always save money? It saves money when the workload actually contains a meaningful mix of easy and hard requests, which most real production workloads do. A workload where every single request genuinely requires the most capable available model will not see much benefit from routing, since there is nothing cheap to route the easy cases toward.

Do I need to build my own router, or should I use an existing one? For most teams, starting with an existing system, NeMo Switchyard for agent workflows or Runway's Model Router for generative media, is faster than building a classifier-based router from scratch. The minimal Python example in Step 6 is a useful way to understand the underlying logic before adopting a full production router, not necessarily a replacement for one at real scale.

How does this connect to how a platform like Miraflow AI works? Any platform that turns one idea into a finished video, image, or piece of music is running multiple specialized models across its pipeline rather than one model doing every step, which is the same multi-model orchestration problem this entire post describes, just expressed as a creative pipeline instead of an agent workflow or a single generative media API call.