HomeTechnical ArchitecturePhilosophyContactRequest access
All articles

Agent

Mechanism: External Routing & Fallback

14 min read

Every time Pixie's agent is about to think — a chat reply, a brief, a huddle turn — it first has to answer a deceptively simple question: which model answers this one? This article is about the one function that decides, the settings that feed it, and the rule that makes the whole thing trustworthy: no matter what is misconfigured, expired, or unplugged, the agent always lands on a model. It just might be a smaller one.

This is a mechanism deep-dive under Choosing Brains: the Model Bench, which explains why Pixie offers several brains at all. Its sibling, Mechanism: On-Device Curation, covers the on-device half — how local models are vetted and why the built-in Apple model is always available. Here we take the other half: the external arms — a runtime on your own computer, a cloud key you bring, or a friend's shared Homebase — and the fallback discipline that keeps them safe to depend on.

1. One seam, one question

Pixie's agent code never talks to a model directly. Every call site — the chat view model, the brief generator, the planner — asks for an LLMProvider: a small protocol with a complete() (and a streaming variant) that takes messages, tool definitions, and options, and returns a response. Concrete providers exist for each brain: AppleFoundationProvider (on-device), OllamaProvider (your own runtime), OpenAIProvider and AnthropicProvider (your own cloud keys), and two relay-routed ones we will meet shortly.

The thing that hands out providers is LLMRouter — a single class, and deliberately the only place in the app where "which model?" is decided. Call sites either hold a reference to it or reach it through a process-wide pointer (LLMRouter.current, set once at app startup), and ask:

let provider = router.provider(model: modelName)

Two design choices in that one line carry most of the weight.

First, the decision is made at every call, not at construction. The router does not cache a provider when a view model is created; it re-reads the user's settings each time provider(model:) runs. Add an API key, flip the model radio, verify a runtime — the very next message routes differently, with no view-model rebuilding, no restart, no "apply" button.

Second, there is exactly one decision point. When routing lives in one function, you can state invariants about it and actually mean them. The invariant this article builds to: every arm of the router, when its prerequisite is missing or its remote end fails, lands on the on-device provider — never on an error. That is only a checkable claim because there is one place to check.

Inside provider(model:), the first step is a call to resolve(), which reduces all the settings to one of four routes:

enum Route {
    case onDevice     // Apple system model or a local MLX model
    case external     // a model on an Ollama runtime (yours, or a shared Homebase)
    case openai       // your own OpenAI key
    case anthropic    // your own Anthropic (Claude) key
}

The rest of the article is how resolve() picks, and what each route does when reality disagrees with the settings.

2. The radio is the routing truth

In Settings → Agent there is a radio group — the model-source radio — with three positions: Apple Intelligence (the built-in system model), Community (a locally downloaded MLX model; see Mechanism: On-Device Curation), and External. Whatever the radio says is the routing truth. There is no heuristic that second-guesses it, no "smart" auto-upgrade to a bigger model behind your back. You chose a brain; the router honors the choice — up to availability.

External is not a single brain but a category. Inside its pane, a runtime dropdown picks which external brain: Ollama (a model server running on your own computer — the default), OpenAI, or Claude. So resolve() is essentially this decision tree:

model-source radio
├─ "apple" / "community"  → .onDevice
├─ "external"             → runtime dropdown
│    ├─ "openai"  → key present?          yes → .openai     no → .onDevice
│    ├─ "claude"  → key present?          yes → .anthropic  no → .onDevice
│    └─ "ollama"  → arm available now?    yes → .external   no → .onDevice
└─ (never set)            → legacy derivation (below)

Notice the shape of every branch: the user's intent is one input, but a prerequisite check is the other. Selecting Claude with no Anthropic key on file does not produce a "please configure" error at send time — it produces .onDevice. The radio expresses what you want; the router only routes there when the arm can actually carry the call. Settings shows a banner nudging you to finish configuring (needsConfiguration backs it), but the agent itself never refuses to answer.

Two legacy branches keep old installs working. Early builds had a four-position radio with OpenAI as its own top-level choice; a stored "openai" radio value still routes (key present → .openai, else .onDevice). And if the radio was never set at all — an install predating the radio entirely — the router derives a sensible answer from older settings: a working Ollama arm wins, then an explicit local-model preference, then a stored OpenAI key, then on-device. The lesson generalizes: a settings migration is a routing problem. If an update makes yesterday's stored preferences meaningless, users silently lose the brain they chose. Cheap aliases avoid that.

3. Bring your own keys: rings with an active first key

A word on why cloud keys exist at all. Pixie ships no platform key — there is no "Pixie cloud inference" the app quietly bills or proxies through. If your words go to OpenAI or Anthropic, it is because you pasted your key, under your account and rate limits. The .openai and .anthropic routes are strictly bring-your-own.

People who bring keys often bring several — a personal key, a work key, one nearing its quota. CloudKeyStore manages each provider's keys as a ring: an ordered list stored in the device Keychain (the OS's encrypted secret store) as a JSON array under <legacyKey>.list. The rules are small and strict:

  • Keys are trimmed and never empty; duplicates are dropped (first occurrence wins, order otherwise preserved).
  • The first key in the ring is the active key, and it is mirrored into the original single-key Keychain slot.
  • An empty ring deletes both slots.
  • An install from before rings had a single key in the legacy slot; it is read as a one-element ring. No migration step, no data loss.

That mirror trick is the interesting part. The router and the providers were written when there was one key per provider, and they still read exactly one: hasBYOKey() and hasAnthropicKey() check the legacy slot, and each provider pulls the same slot at call time. Rings were added around them — reorder the ring in Settings and the new first key lands in the slot everyone already reads. The consumers never changed. When you can add a feature by changing what a storage location contains rather than who reads it, the blast radius stays tiny.

Two more details make keys pleasant to live with. Each provider takes its key as a deferred closure (apiKeyProvider), evaluated per request — rotate a key and the next call uses it, mid-conversation. And the Settings pane's "Test" button is backed by a zero-token probe: AnthropicProvider.validateKey issues a GET /v1/models, which answers 200 for a good key and 401 for a bad one without billing a single token. In the UI, keys render masked — the conventional prefix stays legible, the secret does not: sk-ant-•••••.

The providers behind these routes are thin, faithful API clients. AnthropicProvider speaks the Messages API directly — x-api-key and anthropic-version headers, tools carried as input_schema, tool calls coming back as tool_use blocks whose arguments are a JSON object (unlike OpenAI's, which arrive as a JSON string to parse). OpenAIProvider speaks Chat Completions. Both plug into the same agent loop as every other provider, tools and all.

4. The Ollama arm: whose Homebase?

The dropdown's default — Ollama — is the most interesting arm, because "a model on your own computer" turns out to mean three different network shapes.

The anchor concept is the Homebase: the Mac in your account that acts as a full peer and hosts big models (the whole story is The Homebase). On it runs Ollama, an open-source local model server. Pixie's ExternalRuntimeStore records where that runtime lives — with a test-before-persist gate. An endpoint is stored only after it has answered a version probe, listed its installed models, and — if any completion-capable model is installed — actually served a trial completion. A URL that never proved it can do inference never becomes a route. And a previously verified endpoint is not forgotten just because a later probe fails (maybe Ollama isn't launched right now); configuration and availability are deliberately separate facts.

One storage rule matters more than it looks: the endpoint is device-only and never rides Pixie's cross-device settings sync. The default endpoint is http://127.0.0.1:11434 — the loopback address. If the Mac synced that string to your phone, the phone would try to reach an Ollama server on itself. "My machine runs a runtime" is a machine-local fact, so it stays machine-local.

The Ollama arm also needs a selected model tag (like llama3.2:3b) — no selection, no route. With a selection in hand, provider(model:) walks the shapes in order:

.external, model tag selected
├─ selection points at a JOINED Homebase (someone else's)
│    ├─ that Homebase fresh?   → CrossAccountInferenceProvider   (fallback: on-device)
│    └─ stale                  → on-device
├─ verified local endpoint     → OllamaProvider  (loopback HTTP)
├─ handset + fresh sibling advert
│                              → RemoteInferenceProvider          (fallback: on-device)
└─ otherwise                   → on-device

Shape one: the model is next door. On the Homebase itself, the verified endpoint is loopback HTTP, and the provider is OllamaProvider. It is deliberately a thin wrapper: Ollama exposes an OpenAI-compatible surface at /v1/chat/completions, same streaming format, same tool-call shapes — so OllamaProvider simply configures the battle-tested OpenAIProvider engine with a different base URL, the model tag, and a dummy key Ollama ignores. We could have written a native client against Ollama's own /api/*; instead that API is used only for management (probing, listing, pulling models), and the inference path reuses code that already survived production. Fewer codepaths, fewer bugs.

Shape two: the model is across the room. Your phone has no local runtime, but your account's Homebase does. RemoteInferenceProvider ships the request to the Homebase over the sealed sibling channel — the same end-to-end-encrypted sync_delta frames your devices already use to sync with each other, so the relay in the middle carries only ciphertext. The request/accept/chunk/done round-trip gets its own article, Mechanism: Inference over the Relay.

Shape three: the model is in someone else's house. A friend can share their Homebase with you (see Mechanism: Sharing a Homebase). The external selection records not just the model tag but whose Homebase serves it: a field that is either "own" or the host device's public key. A joined-Homebase selection routes through CrossAccountInferenceProvider over the sealed 1:1 contact channel — and, as the tree shows, it is checked first, so choosing a friend's model wins even on a machine that has its own runtime. If the stored host value is corrupt (not valid hex, wrong length), it silently degrades to "own" — a damaged preference must not strand routing.

One platform asymmetry: shape two is fenced off on Mac Catalyst (the Mac build). The Homebase does not route to a sibling advert, because the Homebase is the host — if its own runtime is gone, the honest answer is on-device, not a network hop to itself. The guest side of shape three has no such fence; any device can be a guest.

5. Freshness: don't route into a dead desktop

Shapes two and three share a failure mode: the desktop at the far end can be asleep, offline, or powered off, and the router cannot cheaply ask "are you there?" before every message. Its evidence is adverts — small signed announcements a Homebase sends ("here are my models"), remembered on the receiving device with a timestamp.

ExternalModelsAdvertStore holds the last advert from your own Homebase, and the router's gate is blunt: the advert must be less than 10 minutes old (and non-empty). The number is not about model lists changing — it is about a specific bad experience. The remote-inference protocol gives the Homebase 10 seconds to accept a request before timing out. Without a freshness gate, a phone whose desktop went to sleep an hour ago would eat that 10-second wait on every single send before falling back. A stale advert means "probably unreachable," so the router falls back before the doomed wait, and chat stays snappy.

JoinedHomebaseStore gates shape three the same way but with a 24-hour window, and the difference is principled. Sibling adverts are heartbeat-like; cross-account adverts are event-driven — sent on grant, on model changes, on runtime re-verification — with no periodic pulse, though every inbound frame from the host refreshes the clock (noteHostAlive). A 10-minute window would mark a perfectly healthy shared Homebase stale most of the day. The generous window only exists to skip hosts that are clearly gone; the in-call fallback (next section) absorbs everything subtler. A joined record must also actually be in the joined state — an invitation you have not accepted never routes — and advertise at least one model.

Freshness gates are a bet, and bets can be lost in both directions: a Homebase can die seconds after advertising, and a "stale" one might actually be reachable. That is fine, because freshness is only the outer of two safety layers.

6. Two layers of fallback

Here is the deep reason the router is shaped the way it is: a call site cannot re-route mid-call. By the time a view model holds a provider and has started a completion, there is no seam to say "actually, that desktop is dead, please redo this elsewhere." So failure handling cannot live at the call site. It lives in two layers below it.

Layer one — routing time. Everything this article has described so far: resolve() and provider(model:) check each arm's prerequisites — key on the ring, verified endpoint, fresh advert, joined state, valid selection — and any miss returns the on-device provider instead of the fancy one. There is even a re-check between the two functions: resolve() may say .external, and in the instant before provider(model:) finishes, the runtime could be forgotten, the advert could expire, the Homebase grant could be revoked. Each of those races lands on-device, not on a trap.

Layer two — inside the call. The two relay-routed providers are constructed with an injected on-device fallback provider baked in. RemoteInferenceProvider arms a watchdog ladder per request — no accept within 10 s, no first output within 90 s, a 30 s mid-stream stall — and every failure of any kind (timeout, model unavailable, host error, an interrupted stream) is answered by silently running the same request on the fallback. The user sees an answer; at most they notice it reads like the smaller model. CrossAccountInferenceProvider does the same, plus one protocol-specific retry: if the host's conversation cache cannot serve the incremental request, it retries once with the full conversation inlined before falling back.

One more rule rides layer two: tool-using calls never cross the wire. If a request carries tools, the relay-routed providers hand it straight to the on-device fallback. The agent loop — tool execution, memory, files — stays on the handset by design (see Mechanism: CallTool — One Tool to Bind Them), and shipping a tool round-trip across the relay is future work, not a silent half-feature.

We could have surfaced these failures instead — an alert, a retry button, a "your Homebase is unreachable" sheet. Plenty of software does, and for a file sync that honesty is right. We decided an agent is different: it is a conversation, and a conversation partner who sometimes goes silent because of your network topology is one you stop talking to. Degrading to a smaller brain is visible in the answer's quality and correctable in Settings; failing to answer teaches the user the feature is flaky. The one thing Pixie never silently does is the reverse — quietly upgrading you to a cloud you did not choose. Fallback always falls toward the device in your hand, never toward someone else's computer.

7. The floor under everything

Follow any arm of the tree to its end and you reach AppleFoundationProvider — and even that has a fallback inside it: if your chosen community MLX model is not downloaded or not ready, it runs the built-in Apple system model, which is always present on a supported device (that last step belongs to Mechanism: On-Device Curation). The recursion bottoms out at a brain that cannot be missing.

That is why LLMRouter has no error path, no "unconfigured" state, and why needsConfiguration drives only a banner. The full picture, in one diagram:

model-source radio
│
├─ Apple Intelligence / Community ─────────────────► on-device
│
└─ External ─ dropdown
     ├─ OpenAI ── key? ──yes─► OpenAI API      ─no─► on-device
     ├─ Claude ── key? ──yes─► Anthropic API   ─no─► on-device
     └─ Ollama ── model selected?  ─no──────────────► on-device
          ├─ joined Homebase, fresh ─► cross-account ─fail─► on-device
          ├─ verified local endpoint ─► Ollama (loopback)
          ├─ fresh sibling advert ───► relay inference ─fail─► on-device
          └─ otherwise ─────────────────────────────► on-device

The moral of this mechanism is the same one that shapes the whole harness: routing is a promise about the worst case, not the best one. The best case was never hard — any demo can call a big model when everything is configured and online. The engineering is in guaranteeing that every misconfiguration, every expired advert, every sleeping desktop, and every mid-stream stall resolves to the same place: a model that is physically in your hand, answering.

References

Next: Mechanism: CallTool — One Tool to Bind Them.