Agent
Agent
16 min read
Your phone holds the most complete record of your relationships that has ever existed — every message, every call, every plan, every "we should really catch up." Pixie ships a personal agent that reads that record and helps you actually keep the people in it close. This is the story of how you build that agent so the record never has to leave your hands.
This article is the entry point to a series, and the series is the third pillar of the system. The other two pillars are about moving bytes privately: Contact Discovery is how friends find each other without the server learning who knows whom, and The Relay is how messages travel without the server learning who is talking. This series is about the cognition that rides on top of them — the agent. Each hard sub-problem gets its own deep dive, and this article links out to them as we go.
One scoping note before we start. This series is about the agent harness — how models are chosen, bound, decomposed, extended, and shared. It is not about machine learning. We will treat the language model the way you treat a database in a systems course: a component with an API, a latency, a context budget, and failure modes. If you are comfortable with programs, APIs, and a bit of systems thinking, you have everything you need; you never need to know what happens inside the model.
1. The problem
Pixie is a relationship-care app. Its one job is helping you stay close to people — the friend whose exam was yesterday, the group trip that never gets scheduled, the parent you keep meaning to call. Social software has historically attacked this job with feeds ranked for engagement, and the result is well documented: more scrolling, not more closeness. Pixie's bet is different: put a personal agent on your side of the glass. It reads what is already there — your conversations, your calls, your posts, your calendar — and surfaces the small, timely thing worth acting on: call Priya; text the gym group back about Saturday; it's Rohan's birthday and you haven't talked in a month.
What do we mean by "agent"? Strip the buzzword and it is a simple loop:
while task not done:
model reads context (the conversation so far + tool results)
model either answers, or asks to call a tool
harness runs the tool, appends the result, loops
A language model that can request actions — search my memory, read that thread, check the calendar — and a harness that executes those requests and feeds results back. The model supplies judgment; the harness supplies memory, tools, permissions, and the rules of the game. Everything interesting in this series lives in the harness.
Two facts make this particular agent hard to build honestly:
- The agent is only useful if it can read your life. An assistant that cannot see
your conversations is reduced to guessing. This is why Pixie ships its own E2EE
messaging and calling at all: Apple does not expose iMessage threads or phone-call
audio to third-party apps, so conversations you choose to have inside Pixie are the
ones your agent can learn from (calls are transcribed on the device by Apple's
SpeechTranscriber; audio and transcripts never leave it). - "Your life" is precisely the data you must never ship to a cloud. The corpus that makes the agent smart — who you love, what you promised, what you worry about — is the most sensitive dataset you own.
Almost every AI assistant on the market resolves this tension in the same way: upload the data, run the model in the cloud, promise to behave. We resolved it the other way.
2. The trust decision: the agent comes to the data
The naive architecture is a thin client and a smart server: your phone streams context to a hosted model, the platform's servers do the thinking. It is the easy road — big models, no on-device constraints, one place to iterate. It also means the platform holds a live feed of your relationships, and your "personal" agent is a rented one: cancel the subscription and your assistant — with its memory of you — evaporates.
We know the pull of that road because we walked partway down it. Phase 2 shipped a
bundled-inference tier (llm-svc — a thin proxy that forwarded a device's LLM calls
under the platform's account). We eliminated it. Not deprioritized: eliminated — no
tier of the product routes cognition through our servers, and there is no agent runtime
on the server side anywhere in the system. The rule that replaced it is the trust
decision this whole series unpacks:
Local-first cognition: the agent comes to the data, not the reverse. Your memory, your relationship map, and the reasoning over them live on hardware you own. What leaves the device leaves sealed.
Saying "local-first" is cheap; the architecture only becomes trustworthy when the expensive consequences are accepted. There are three, and they shaped everything:
- Small models must be enough. On-device models are a fraction of the size of hosted ones, and a small model asked to plan, follow instructions, and juggle a dozen tool schemas at once simply fails. The answer is not a bigger model but a better harness — decomposing the work so each small context does one job. That is The L1–L2–L3 Harness.
- Memory must survive a model swap. If retrieval depended on vector embeddings, every model change would orphan the index (embeddings from different models are mutually gibberish). So Pixie has no vector store at all: memory is an append-only event log with a plain-text mirror on disk, and retrieval is an agentic grep loop — the agent searches with regexes, reads what it finds, refines, repeats. Swap the model and the memory doesn't even notice. That design is Mechanism: Memory & the Notebook.
- Ownership must be structural, not contractual. Because the agent, its memory, and your contact graph are on your device, they survive subscription cancellation as a matter of physics. Cancelling Pixie feels like leaving a club, not digital death — the club's coordination perks stop; you remain intact.
We want to be honest about the price. A hosted frontier model is smarter than anything that fits in a phone, and a large part of this series is the engineering spent closing that gap — decomposition, routing, and an escape hatch that lets you borrow big-model power from hardware you own (your Mac) without breaking the rule. The bet is that for a personal agent, custody beats raw IQ: a modest intelligence that knows your whole context and answers to no one else is worth more than a brilliant one on someone else's computer.
3. The three-layer picture
The whole system is a clean horizontal cut, and the cut is the product thesis:
┌─────────────────────────────────────────────────────────────────┐
│ Layer 1 — Personal Cognitive Layer (this series) │
│ the agent: memory log · grep retrieval · planner · tools · │
│ skills · Notebook. Lives on your devices. Yours permanently. │
├─────────────────────────────────────────────────────────────────┤
│ Layer 2 — Social Relay Layer (Relay + Contact Discovery) │
│ sealed-sender transport: the server routes envelopes it │
│ cannot read, from senders it cannot identify. │
├─────────────────────────────────────────────────────────────────┤
│ Layer 3 — Model Execution Layer (article 01 below) │
│ the swappable model bench: Apple's system model · local MLX │
│ models · your API keys · your Mac. Never platform-owned. │
└─────────────────────────────────────────────────────────────────┘
The design invariant is that layers above can swap layers below without losing identity. Change the model (Layer 3) and your agent is still your agent — same memory, same relationships, same behavior contract. That invariant is why the no-embeddings rule exists, and it is what makes "the platform never owns the cognition substrate" a structural claim instead of a promise.
Layer 1 is the agent proper, and it is what articles 02, 03, and 05 dissect: the append-only memory log, the read-only tool registry, the planner, the user-extensible skills, and the Notebook — the agent's distilled, mostly user-legible memory of you and your people.
Layer 2 is the transport the agent rides, and the two sibling series own it. The one
term you need from them here is sealed sender: every envelope on the wire hides its
sender even from the relay — and the recipient is hidden too. A 1:1 message is not
addressed to anyone's public key; it rides an opaque rotating pair subject, a value
both friends derive from their shared channel key that rotates every hour and that no
pubkey, phone number, or account id ever feeds into. The relay sees that opaque subject
and a little coarse delivery metadata — never who is talking, and never to whom. Two properties of that
layer matter constantly in this series. First, anything the agent sends anywhere —
including to your own other devices — goes as a sealed envelope, so building a new agent
feature never means giving the server a new window. Second, your devices form a fleet:
an iPhone, an iPad, a Mac, all holding the same memory, kept coherent by a sealed
same-account sibling channel called sync_delta (the wire kind that carries notebook
and memory deltas, state digests, and — as we will see — even inference traffic between
your own devices). A seat-token election called the driver's seat — the mechanism we call
GroupThink — ensures exactly one device runs autonomous agent work at a time, so your
agent never shows up twice to the same conversation.
Layer 3 is the model bench, and it is deliberately a bench, not a model: a set of
interchangeable providers behind one Swift protocol (LLMProvider), with one routing
authority deciding which arm thinks for any given task. That is article 01.
4. The harness in five questions
Everything below Layer 1's surface falls out of five questions. Each is an article; the deeper "Mechanism" pieces under them are the engine room.
| Question | Deep dive |
|---|---|
| Which model does the thinking — and who controls that choice? | Choosing Brains: the Model Bench |
| How does a small on-device model do reliable multi-step work? | The L1–L2–L3 Harness |
| How do you extend and steer the agent without retraining anything? | Skills, Tools, and the @-Namespace |
| How do you borrow big-model power without renting a cloud? | The Homebase |
| Where does all of this actually meet your day? | The Agent in Your Day |
A quick tour, so the map has terrain.
Choosing brains. Every model call in the app funnels through LLMRouter, the one
component allowed to decide which provider thinks. The bench it routes across: Apple's
built-in system model (the floor — always present on a supported device), a curated
catalog of open-weights models from the mlx-community hub running locally via MLX, your
own OpenAI or Claude API keys as an explicit opt-in, and external models served from your
own Mac. The user-facing truth is a single model-source radio in Settings — three
positions, and the radio is the only routing authority, so picking a model in some
sub-pane can never silently reroute your cognition. Every arm degrades to on-device
instead of failing: a dead Mac or an expired key costs you quality, never availability.
How the catalog decides which community models qualify (tool-calling and vision gates,
quantization tiers, device capability) is Mechanism: On-Device
Curation; the external arms and their fallback
ladder are Mechanism: External Routing & Fallback.
The L1–L2–L3 split. Ask one small model to plan a task, follow the plan, and wield a
toolbox of full JSON schemas simultaneously, and it drops threads — we measured it, and
the monolithic executor lost badly. So the runtime splits the cognition three ways: an
L1 planner writes a short free-text plan; an L2 skill-follower executes it while
seeing only one-line signatures of the tools (not their schemas) plus a single
meta-tool; L3 tool-agents are fresh, disposable sessions each bound to exactly one
real tool's full schema for a single call. No context ever carries more than one schema,
which is what keeps the harness reliable today and is the scaling hedge for a many-tool
future. The single meta-tool trick — CallTool, the one door through which every L3
dispatch passes — is Mechanism: CallTool — One Tool to Bind
Them.
Skills and the @-namespace. A skill is a user-ownable bundle of instructions —
"how I like my weekly review done" — that the planner can pick up by name. Skills and
tools live in one shared @-namespace (@search-info, @check-schedule, …), so a
skill can reference tools and other skills by handle, renames cascade instead of
dangling, and the planner's menu of @-entries doubles as the user's alignment surface:
what the agent can reach for is exactly what you can see listed. Skills travel, too —
you can send one to a friend as a sealed message, closure and all, to be reviewed before
import. That path (the transitive @-closure, collision re-slugging, review-before-import)
is Mechanism: Skill Bundles & Sharing.
The Homebase. The rule was never "small models only" — it was "hardware you own."
Your Mac runs the same app as a full peer (same keys, same memory, same agent) and
becomes the Pixie Homebase: it hosts big open models via Ollama and serves your
handset over the sealed relay — an infer_request / accept / chunk / done round
riding sync_delta, with timeout watchdogs at every stage and a silent on-device
fallback wired inside the provider, so a sleeping Mac is a quality downgrade, not an
error (Mechanism: Inference over the Relay). It can
lend that power to people you trust under revocable grants whose usage metrics count
bytes and tokens but structurally cannot contain content (Mechanism: Sharing a
Homebase). And it opens the one governed door for
other agents — Claude Desktop, a local model, anything speaking MCP (the Model
Context Protocol, the emerging standard for exposing tools to agents) — to drive Pixie: a
loopback-only server whose read tools project only the unsealed zones, and whose three
write tools suspend on a per-call human approval sheet showing the exact draft, where
dismissing means deny (Mechanism: MCP — Letting Other Agents
In).
The agent in your day. Where it all cashes out: the chat tab; Summon — the @pixie inline
tag — mention the agent inside a human conversation and it answers only on your
screen, the peer never sees the summon (a single property-tested chokepoint refuses
summon-shaped bodies at every peer-send path); proactive nudges that arrive in the right
conversation at the right moment ("how was your day?" fires when you actually get home,
not at 6 PM sharp); huddles, where the agents of a mutually-trusting friend group
talk to each other in structured rounds over group E2EE and come back with at most a
nudge — no human ever reads the transcript, and there isn't one; plans that turn a huddle
into a calendar entry; and the Notebook, the agent's curated memory — a clear zone
you can read and edit (your edits are ground truth) and a sealed zone for what the
agent learned in confidence from other people's agents, encrypted under a key you don't
hold, with the leak direction structurally forbidden. That surface — and the memory
machinery under it — is article 05 and Mechanism: Memory & the
Notebook.
5. One request, end to end
To make the layers concrete, here is a real question moving through the harness. You ask the chat: "What did I promise the gym group last month?"
you ──▶ chat surface
│
▼
LLMRouter reads the model-source radio → binds a provider
│ (say: an mlx-community model, running locally)
▼
L1 planner writes a free-text plan:
│ "check the notebook and search memory for
│ gym-group commitments; summarize."
▼
L2 skill-follower holds the plan + one-line tool signatures
│ + ONE meta-tool: call_tool{tool, instruction}
│
├─ call_tool("grep_memory", "gym group commitments, last month")
│ └─▶ L3 session, bound to grep_memory's full schema,
│ regex-greps the on-disk text mirror of the event log
│
├─ call_tool("check_notebook", "notes about the gym group")
│ └─▶ L3 session, bound to check_notebook's full schema,
│ searches the Notebook's clear zone, locally
▼
answer, assembled from the gathered results — with sources
Count the network calls in that trace: zero. The model is local, the memory is local, the tools read local stores. If the radio had pointed at your Homebase, the prompt would have crossed to your Mac as a sealed envelope between your own devices — still nothing a server could read. The shape of this trace — why L2 never sees a schema, why L3 is disposable, why the answer is assembled from tool results rather than the model's paraphrase of them — is the substance of article 02.
6. What the agent may not do
A harness is defined as much by its refusals as its abilities, and Pixie's sharpest design choices are refusals:
- The device agent's tool registry is read-only.
grep_memory,read_conversation,read_post_thread,read_call,check_notebook, file and calendar lookups — the agent reads and plans. It never sends a message, never posts, never mutates your relationship data. One-tap call/text actions are plain app UI; what surfaces in your feed is decided by deterministic pipelines, and every surfacing decision is logged with a reason — "why did my agent tell me this" always has an answer. - The model never touches relationship state. Your circles are set by you; the agent's private warm/cold sense of each relationship is derived by code outside the tool loop. No LLM output writes either.
- The sealed zone stays sealed. What the agent learned from other agents in a huddle can never be rendered, never be moved to the clear zone, and never enter the chat agent's context at all — it cannot leak what it cannot see.
- What you tell it not to share, it cannot share. The Notebook's behavioral-guidance section ("don't bring up my job search") is enforced by an on-device gate before any huddle turn leaves the store — a filter in code, not a plea in a prompt.
- Outside agents get a door, not a hallway. MCP writes suspend on a human approval showing the exact draft; silence is denial.
Notice the pattern: every boundary is structural — a missing tool, a missing key, a chokepoint in code — rather than an instruction the model is trusted to follow. Prompts are how we ask the agent to behave; architecture is how we guarantee it.
7. The map of the series
00 Agent (you are here)
│
├── 01 Choosing Brains: the Model Bench
│ ├── 06 Mechanism: On-Device Curation
│ └── 07 Mechanism: External Routing & Fallback
│
├── 02 The L1–L2–L3 Harness
│ └── 08 Mechanism: CallTool — One Tool to Bind Them
│
├── 03 Skills, Tools, and the @-Namespace
│ └── 09 Mechanism: Skill Bundles & Sharing
│
├── 04 The Homebase
│ ├── 10 Mechanism: Inference over the Relay
│ ├── 11 Mechanism: Sharing a Homebase
│ └── 12 Mechanism: MCP — Letting Other Agents In
│
└── 05 The Agent in Your Day
└── 13 Mechanism: Memory & the Notebook
Read top-down for the story, bottom-up for the mechanisms:
- Start with Choosing Brains: the Model Bench — the bench and its one routing truth are assumed everywhere else.
- The L1–L2–L3 Harness and Skills, Tools, and the @-Namespace are the heart of the cognition design; read them in order.
- The Homebase is where the local-first rule meets big-model appetite; its three Mechanism leaves are self-contained.
- The Agent in Your Day closes the loop from machinery back to the product's actual goal, and Mechanism: Memory & the Notebook beneath it is the deepest dive in the series.
- If a Mechanism article cites transport guarantees, the sibling series hold the proofs: The Relay for sealed delivery, Contact Discovery for private matching.
A recurring theme, and really the moral of the pillar: the hard part of a personal agent is not intelligence — it is custody. Models improve on their own schedule, and a swappable bench inherits every improvement the day it ships. What no model release can retrofit is where the thinking happens and who owns the memory it thinks with. Get the harness right and the agent gets smarter every year without your life going anywhere at all.
References
- Apple, Foundation Models framework — the on-device system model and the provider seam Pixie's bench is built against.
- MLX and the mlx-community hub — the local open-weights arm of the bench.
- Anthropic, Model Context Protocol — the standard behind the Homebase's governed door for external agents.
- Signal, "Sealed Sender" — the transport idea (hiding the sender from the server) whose post-quantum descendant carries every envelope in this series; see The Relay.
- Wikipedia: Software agent, Local-first software, End-to-end encryption.