Agent

Choosing Brains: the Model Bench

12 min read

An agent harness is, at bottom, a loop wrapped around a language model: gather context, ask the model, act on the answer, repeat. Everything else in this series — planners, skills, tool calls, huddles — rides on that one call. So the first real design question is deceptively simple: which model answers? This article is about Pixie's answer: not one model but a bench of them, one switch that decides who plays, and one rule that nothing ever benches the whole team.

This is the first deep dive of the series, sitting directly under Agent — which explains why a relationship-care app ships an agent at all, and why its cognition is local-first. Two Mechanism articles sit beneath this one: Mechanism: On-Device Curation and Mechanism: External Routing & Fallback. You do not need any machine learning here; we treat every model as a black box with a complete() method. What we care about is the harness: how a model is chosen, what happens when the chosen one is unavailable, and who gets to decide.

1. The tension: small and here vs. big and elsewhere

Every agent that runs on a phone faces the same fork.

On-device models are private by construction — the prompt never leaves the hardware — and they are always there: no network, no account, no key, no server having a bad day. But a model that fits in a phone's memory is small, and small models are weaker at planning, at following long instructions, and at calling tools correctly.

External models — a big open-weights model on your own computer, or a frontier model behind an API key — are dramatically more capable. But each comes with a prerequisite (a running machine, a valid key) and a cost (your prompt travels somewhere else).

Most apps resolve this fork by picking a side. Pixie's stance, inherited from the local-first trust decision in Agent, is different: the fork is the user's to resolve, per device, at any time — and the app's job is to make every choice safe to make. Concretely that means three things, and they are the spine of this article:

A bench of model sources, from an always-available floor up to bring-your-own-frontier.
A single routing truth — one Settings switch that alone decides where cognition runs.
Graceful degradation — every choice, when its prerequisite is missing, falls back to on-device instead of failing.

The bench has three seats. Let us take them in order of increasing ambition.

2. The floor: Apple's built-in model

The first seat is the one that is always occupied. Every iPhone that runs Pixie ships with Apple's built-in system language model, exposed through the FoundationModels framework (SystemLanguageModel driven through a LanguageModelSession). Pixie wraps it in a type called AppleFoundationProvider, which implements the app's one model-facing interface, LLMProvider — essentially "take a list of messages and some tool definitions, return a completion (or a stream of chunks)." Every brain on the bench, local or remote, hides behind that same interface, which is what makes swapping them a routing decision rather than a rewrite.

The built-in model matters for two reasons that have nothing to do with its raw quality:

It has no prerequisite. Nothing to download, nothing to configure, no key to paste. In Settings it is described exactly this way: "The built-in on-device model. Private, always available."
It is the floor everything else lands on. As we will see in section 6, every other arm of the router names this provider as its fallback. When any prerequisite anywhere is missing, the agent thinks here.

One honest wrinkle, because it shapes the deeper articles: FoundationModels has its own native idea of a tool (a Swift Tool protocol with typed, @Generable arguments), which is a different shape from the generic JSON-schema tool definitions the rest of the harness passes around. AppleFoundationProvider does not pretend otherwise — generic tools are dropped with a debug log, and instead the provider attaches native FoundationModels tools for the integrations the user has switched on (today: a Calendar tool, gated on the Settings → Agent → Context toggle). How the harness gets reliable tool use out of small models anyway — by splitting the work across cooperating sessions — is the subject of The L1–L2–L3 Harness.

3. Seat two: curated community models

The second seat is for users who want a stronger on-device brain and are willing to spend a download and some gigabytes for it. Pixie can run open-weights models from the mlx-community organization on Hugging Face — models converted to run on Apple silicon via the MLX runtime — entirely on the phone, through the same AppleFoundationProvider seam (the provider builds its session against the chosen MLX model when it is selected and downloaded, and against the system model otherwise).

The interesting design problem here is not running the models; it is choosing which ones to offer. The mlx-community org contains thousands of repositories, most of which would be a trap in an agent harness: base checkpoints with no chat template, text-only models that cannot read images, models that cannot call tools. Pixie's catalog (MLXModelHubCatalog) fetches the org and parses it into a four-level tree —

family (Qwen) → series (Qwen 2.5) → size (1.5B · Coder) → quant (4-bit)

— where each leaf is one downloadable repo, and then applies a curation bar: the browser only surfaces models that can do vision and tool calling, because the skills-and-plan loop needs both to work the same way it does on the Apple system model. A model that cannot call tools is not a smaller brain in this harness; it is a dead end. Whether a repo qualifies is itself heuristic (Hugging Face metadata has no reliable "can call tools" flag, so the catalog keys off instruct/chat-tuned naming), the quantization tiers (4-bit, 8-bit, BF16) trade download size against fidelity, and the device itself gets a vote — a model can be marked "Needs an iPhone on iOS 27 with enough memory." All of that filtering machinery is its own story: Mechanism: On-Device Curation.

Two footnotes worth knowing. First, the curation bar is a default, not a cage: there is an escape hatch to add any mlx-community/<repo> by name, on the user's own judgment. Second, the built-in Apple model is deliberately not listed in this browser — it is its own radio row, not one entry among thousands, because it plays a structurally different role: it is the floor.

4. Seat three: External — a category, not a provider

The third seat is where most apps would put "the cloud." Pixie instead puts a category. The Settings row reads: "Ollama on your Pixie Homebase, or your own OpenAI / Claude API key." The common thread is ownership — every brain in this category is something the user brings:

Your own Ollama runtime. The Homebase — Pixie on your Mac, a full peer of your phone — can run Ollama, a local server for large open-weights models. Pixie discovers it by probing the default loopback port (with manual entry for a non-default one), and tests before persisting: a candidate endpoint that fails its probe is reported, never stored. The Homebase advertises its installed models to your other devices over the sealed sibling channel — the end-to-end-encrypted relay lane between your own devices, which the server cannot read (that channel belongs to the sibling Relay series) — and your handset can then route inference requests to it. Even the pull list on the Homebase is curated the same way the MLX browser is: a short list of known tool-calling-capable tags, plus free-text for anything else.
Your own OpenAI or Claude key. Paste an API key and calls go straight from the device to that provider — there is no Pixie server in the middle of the path. Keys live in the device Keychain, are displayed only obfuscated (sk-proj•••••), and each one has a Test button that fires a real authenticated request that bills zero tokens (a GET /v1/models; a 200 proves the key). You can hold a ring of keys per provider; the first key in the ring is the active one.
A Homebase someone shared with you. A friend can grant your account access to their Homebase's models. On your side that pane is deliberately read-only — you select a model with a checkmark, and nothing more; their machine, their management. The grant-and-revoke machinery is Mechanism: Sharing a Homebase.

Inside the External pane, a segmented dropdown picks which of these arms is live. That dropdown is the reason External is one radio row rather than three: from the router's point of view there is one question — "is cognition leaving this device's on-device models?" — and one category answers it. Which arm, and how each is wired and probed, is the second deep dive under this article: Mechanism: External Routing & Fallback.

5. One routing truth

Three sources, several arms, key rings, adverts, shared machines — this could easily become a pile of interacting toggles where nobody can answer "so where does my prompt actually go?" Pixie collapses it to one type and one switch.

The type is LLMRouter: the single point that decides which LLMProvider every agent call site gets. Chat, briefs, huddles, the planner — none of them know about Ollama or keys or adverts; they ask the router for a provider and get one. The switch is the model-source radio in Settings → Agent: three rows — Apple Intelligence, Community, External — of which exactly one is selected. The router's resolve() reads that radio first, and the radio alone decides the route:

radio                 prerequisite                        route
──────────────────────────────────────────────────────────────────────────
Apple Intelligence    (none)                              on-device, system model
Community             chosen model downloaded & ready     on-device, your MLX pick
External · Ollama     verified runtime, or a fresh        your own Ollama
                      Homebase advert (< 10 min old)
External · OpenAI     a key in the Keychain               OpenAI, your key
External · Claude     a key in the Keychain               Anthropic, your key

any row, prerequisite missing  ──────────────────────▶    on-device

Two disciplines keep this switch trustworthy.

Checkmarks are configuration; the radio is routing. Everything inside a configure pane — picking an MLX quant, selecting an Ollama model, saving a key — sets a checkmark. It never flips the radio. This is a deliberate consent boundary: pasting an OpenAI key should not silently start sending your conversations to OpenAI; it should make the External row ready, and only your explicit tap on that row changes where cognition runs. The inverse courtesy also holds — selecting a source whose configuration is not finished yet (Community with no model chosen, External that cannot route) auto-opens its pane so you can finish, and until you do, the router simply resolves on-device.

The decision is made per call, not at construction. The router re-resolves every time a call site asks for a provider, so a key you just added or a radio you just flipped applies on the very next send — no restart, no rebuilt view models, no stale provider captured in a closure somewhere. (The router even honors a retired setting: an old install whose radio still says the short-lived value "openai" keeps routing correctly, mapped into the External category. Stored user intent outlives UI redesigns.)

6. The rule: every arm degrades to on-device

Now the principle the whole bench is built around: the agent never fails to route. There is no state of the settings, the network, or the world in which asking for a provider returns an error. Instead, every arm names its prerequisite, and a missing prerequisite means one thing — fall back on-device:

External · Claude ──(no key)──────────────────┐
External · OpenAI ──(no key)──────────────────┤
External · Ollama ──(no runtime, no fresh ────┼──▶  your on-device pick ──(not ready)──▶  Apple system model
                     advert, or Homebase      │
                     revoked)                 │
Community ────────────────────────────────────┘

Note that the ladder has two rungs even after the fallback: the router falls back to the user's on-device catalog choice, and that provider in turn falls back to the built-in system model if the chosen MLX model is not downloaded or the device cannot run it. The floor is genuinely the floor.

The fallback also runs at two times, because failures do not politely wait for resolve():

At resolve time, the router checks prerequisites before choosing a route. The Ollama arm is the instructive case: a handset only routes to the Homebase while the Homebase's model advert is fresh, so a Mac that has gone to sleep is skipped instantly instead of costing a doomed timeout on every send. Freshness turns "fail slowly, every time" into "fall back instantly" — the windows and the reasoning behind them are in Mechanism: External Routing & Fallback.
Inside the provider, for the races that resolve-time checks cannot close: the runtime forgotten, the advert expired, the shared Homebase revoked between the routing decision and the actual call. A call site cannot re-route mid-call, so the remote providers carry the on-device provider as an internal fallback — the race falls back, it does not trap. (Even a corrupted stored value degrades safely: an unreadable "whose Homebase" record is treated as "my own" rather than stranding routing.)

Why insist on this? Because of what the agent is for. Pixie's agent nudges you before a friend's birthday, summarizes a huddle, drafts the message you asked for. An error modal — "configure a model to continue" — at any of those moments converts a relationship tool into an IT ticket. A smaller answer beats no answer, always.

And the tradeoff, stated honestly: graceful degradation means capability can drop without the current operation failing — your carefully chosen 70B model naps with the Mac, and the next reply quietly comes from a phone-sized brain. Silent-but-invisible would be a dark pattern, so the state is surfaced where you would look for it: the Model section in Settings leads with a live banner naming the resolved route ("External · qwen2.5:7b", "On-device · Apple Intelligence"), and the External panes flag a stale Homebase in orange — "Not heard from recently — inference falls back on-device until it's back." The system never lies about where it is thinking; it just refuses to stop thinking while you fix it.

7. Where the deep dives go

This article gave you the bench and the switch. Two mechanism articles sit directly below it:

Mechanism: On-Device Curation — how the catalog decides which mlx-community models qualify (the tool-calling and vision gates, quant tiers, device capability checks), and why the built-in Apple model is always the floor.
Mechanism: External Routing & Fallback — Ollama on the Homebase, the cloud key rings and their zero-token probes, the LLMRouter seam in detail, and the full fallback ladder including the mid-call races.

Sideways and up: The Homebase is the machine that makes the Ollama arm interesting, and Mechanism: Inference over the Relay is how a handset's prompt reaches it without any server reading it. And the question this article deliberately postponed — how can the small on-device floor run a real agent at all? — is exactly where the story goes next.

References

Apple, FoundationModels framework — SystemLanguageModel, LanguageModelSession, and the native Tool protocol.
Hugging Face, mlx-community — the open-weights model organization the Community catalog is built from; and MLX, the Apple-silicon array framework those models run on.
Ollama — the local model runtime the External category's self-hosted arm connects to.

Next: The L1–L2–L3 Harness.