One Default AI Model Is Not Enough

Conversational models

The default mental model for "AI in a product" is usually singular: there's an AI, and it answers things. In practice, even a fairly small application ends up needing several different models doing several different jobs, and treating them as one interchangeable "the AI" is where a lot of avoidable confusion comes from.

Start with the obvious one: the model doing the actual talking, answering questions, reasoning through a problem, holding a conversation. This is usually the model people picture when they think "AI assistant," and it's the one most worth spending on, because conversational quality is what a user directly experiences.

Structured-output models

A separate job, easy to overlook: generating output that has to conform to a strict, machine-readable shape rather than free-flowing prose. A model can be an excellent conversationalist and a mediocre fit for reliably returning valid, schema-constrained data every time -- these are genuinely different skills, and not every model is strong at both. Systems that need dependable structured output often do better treating it as its own assignment, with its own model choice, rather than assuming whatever answers chat questions well will also format data correctly on demand.

Embedding models

A third, quieter job: turning text into the numerical representations that power search and retrieval. Embedding models don't converse at all -- they exist purely to make "find the relevant thing" work well. This is arguably the job most likely to be silently miscategorised as part of "the AI," when it's really infrastructure for retrieval, with its own quality and consistency concerns, independent of whichever model happens to be generating conversational replies at the same time.

Evaluation or judge models

A less obvious role, but an increasingly important one as AI-assisted work becomes serious enough to need checking: a model whose job is assessing another model's output. Using AI to help evaluate AI is a real and useful pattern -- comparing responses, checking for issues, scoring quality against a rubric -- but it only works if the evaluating model is chosen deliberately for that job, not simply whatever happened to be the default already assigned to something else.

Capability versus assignment

Here's the distinction that tends to get lost: a model being capable of a role and a model being assigned to it are two different facts, and conflating them causes real operational confusion. A model might support structured output perfectly well without currently being the one anything relies on for it. Enabling a model makes it available. It doesn't make it responsible for anything until something is actually pointed at it. Keeping those two ideas visibly separate -- what a model can do versus what it's currently doing -- is most of what it takes to make a multi-model setup understandable rather than mysterious.

Cost, latency, and reliability

Different jobs also have genuinely different tolerances. A conversational reply that takes an extra second is barely noticeable; a structured-output call blocking a workflow is more latency-sensitive. A model that's excellent but expensive might be the right choice for occasional deep analysis and the wrong choice for something called on every single request. None of this argues for always picking the cheapest or fastest option -- it argues for treating each role's requirements as its own decision, rather than inheriting whatever tradeoff was made for a different job entirely.

There's also a more basic lesson underneath all of this, learned the unglamorous way: it is entirely possible to configure the wrong provider because two names looked similar enough at a glance. A multi-model setup only works if it's genuinely easy to see, at a glance, which provider and which model is actually assigned to which job -- because the moment that's unclear, mistakes like that stop being rare.

Why administrators must see which model serves which role

Put all of this together and the practical requirement is simple to state and easy to get wrong in an interface: someone configuring the system should be able to look at it and immediately answer "which model handles conversation, which handles structured output, which handles retrieval, and which of those assignments would this change actually affect?" -- without having to infer it from ambiguous labels or dig through multiple settings pages to piece together a mental model that should have just been shown to them.

One model can carry an application a surprisingly long way. But the moment a second job shows up -- structured output, retrieval, evaluation -- pretending it's still "the AI, singular" stops being simplicity and starts being a blind spot. Naming the jobs, and making it obvious which model is doing which one, is what keeps a multi-model system legible instead of accidentally risky.