In this article
Do not choose one AI model for every task, and do not optimise for the cheapest request. Route each task class to the path with the lowest cost per accepted outcome, then escalate uncertain or high-harm cases to a stronger model or human review. This reduces cost without buying false savings at the expense of business quality.
A cheap call that produces a rejected answer and triggers rework can cost more than a higher-priced call that succeeds once. Routing economics therefore needs four inputs: call cost, acceptance rate, retry cost and the cost of an error that reaches a customer or operational process.
Define acceptance before building the router
Classify work by required outcome, not merely by department. Summarising an internal document is not equivalent to recommending customer credit, even if both use the same interface. Give each class a measurable acceptance test: required fields completed, claims grounded in authorised sources, pricing policy followed, arithmetic validated, or a defined human sample approved.
The NIST AI RMF Measure playbook calls for performance and assurance criteria to be assessed in conditions similar to deployment. Its Manage playbook calls for ongoing monitoring of components connected to pretrained models and remediation when risk tolerances are exceeded. Routing is one operational application of that logic; NIST does not prescribe a particular model or vendor.
Build three decision layers
- Risk gate: financial, legal or sensitive-data decisions go to a stronger or human-approved path even when the prompt appears simple.
- Complexity gate: use observable features such as context length, document count, calculations and tool use rather than a vague difficulty label.
- Confidence gate: test the output after execution. If acceptance rules fail or sources conflict, escalate once through a defined path instead of bouncing indefinitely.
Do not let a model be the sole judge of its own answer. Prefer deterministic checks where possible, source comparison, human sampling and safe stops when evidence is insufficient. Apply authorisation before retrieval before the request reaches any generation path.
A worked hypothetical
Assumptions: a company processes 100,000 tasks per month: 70,000 simple, 25,000 moderate and 5,000 high-risk. An efficient path costs QAR 0.03 per attempt and achieves acceptance rates of 92%, 72% and 45% across those classes. An advanced path costs QAR 0.18 and achieves 98%, 91% and 86%. These are hypothetical assumptions, not provider prices or reported company results.
Sending every task to the advanced path would cost QAR 18,000 in model calls. Under the proposed policy, simple and moderate work starts on the efficient path and rejected outputs escalate; high-risk work goes directly to the advanced path.
- Simple: 70,000 × QAR 0.03 = QAR 2,100, then 5,600 rejects × QAR 0.18 = QAR 1,008.
- Moderate: 25,000 × QAR 0.03 = QAR 750, then 7,000 rejects × QAR 0.18 = QAR 1,260.
- High-risk: 5,000 × QAR 0.18 = QAR 900.
Model cost becomes QAR 6,018. Add a hypothetical QAR 1,500 for routing, evaluation and monitoring, and the total is QAR 7,518 rather than QAR 18,000—a difference of QAR 10,482. Savings alone are not enough. Under the assumptions, the routed process produces about 98,558 accepted outcomes after escalation, or roughly QAR 0.076 per accepted outcome. Human-review cost and actual failure loss still need to be included.
Acceptance is not generic accuracy
An answer can read well and still fail commercially because it omitted a required field or used an expired policy. Build a dashboard by task class showing first-pass acceptance, escalation rate, cost per accepted outcome, tail latency, errors reaching users, and human-review cost.
Connect that view to the economics of human exception queues, release gates based on error cost, and the AI incident log. Routing does not replace those controls; it decides where they are applied.
Limits and failure modes
A router fails when task distribution changes but thresholds remain fixed, self-reported confidence substitutes for an independent test, or routing latency exceeds the savings. Teams may also choose the “cheap” path because its budget looks better while shifting failure cost to customer service or compliance. The process owner—not the technology team alone—must therefore define an accepted outcome.
Add failure cost to the equation
Expected route cost is not the call fee. A practical approximation is: execution cost + rejection probability × rework cost + escaped-error probability × harm cost. A low-volume class can be the most expensive when one error freezes a major order or exposes sensitive information.
Review thresholds when the model, system instructions, knowledge base or customer mix changes—not only on a calendar. Keep one fixed benchmark sample and one recent sample representing live demand. If savings fall while latency or manual workload improves, show the complete trade-off rather than reporting a single cost-reduction percentage that hides quality.
Decision for the first pilot
Start with two classes: one high-volume, low-harm task and one low-volume, high-harm task. Run the router in shadow mode for two weeks without changing the live path, then compare cost, acceptance, escalation and latency. Activate routing gradually only if it reduces cost per accepted outcome without increasing errors that reach customers.
Contact on WhatsApp
