HomeTechnical ArchitecturePhilosophyContactRequest access
All articles

Agent

Mechanism: Memory & the Notebook

16 min read

An agent that helps you care for your relationships is only as good as what it remembers — and an on-device model has a context window of a few thousand tokens, against a life that produces years of messages, calls, posts, and journal entries. This article is about the machinery that resolves that mismatch: an append-only log the agent can grep, a distilled Notebook split into a zone you can read and a zone you never can, a reaction that lets you pin a moment straight into memory, and a retrieval path that is a tool call — never a context dump.

This is the deepest mechanism dive under The Agent in Your Day, which tours every surface the agent shows up on and leans on this one constantly — the chat that knows your friend's news, the brief that knows what to nudge about, the huddle that knows what to bring. Here we open the hood on the memory itself: how it is written, how it is protected, how it is searched, and how it can even live partly on another one of your devices without the agent losing track of it.

1. Why retrieval is a tool call, not context stuffing

Start with the naive design, because it is what almost every chat-with-memory demo does: keep the agent's notes in a big string and prepend it to every conversation turn. "Context stuffing." For a cloud model with a two-hundred-thousand-token window, you can get surprisingly far this way. Pixie cannot, for two reasons — one of budget and one of principle.

The budget reason: Pixie's default brain is Apple's built-in system model, with a context window of roughly 4k tokens (see Choosing Brains: the Model Bench). Every token the Notebook occupies is a token the actual conversation, the tool results, and the reply cannot. Stuff a few dozen notes into the system prompt and the model has no room left to think — and most of those notes are irrelevant to the question at hand anyway. You asked about Priya; the model is carrying your dentist's phone anxiety in its working memory for no reason.

The principle reason: what rides in context is invisible and unaccountable. A tool call is an event — it can be logged, surfaced in the UI ("Checked your notebook"), rate-limited, and reasoned about. When memory access is a tool call, "what did the agent look at to produce this answer?" has an answer.

So Pixie's rule is: memory is never injected; it is fetched. The system prompt carries the agent's persona and standing instructions, and nothing of the Notebook. When the model needs background, it calls a tool — check_notebook for the distilled notes, grep_memory for the raw log — and only the handful of matching entries enter the context, for that turn only. The rest of this article is the machinery on the other side of those two calls.

                        the model's context (small)
                       ┌────────────────────────────┐
   your question ────► │ persona · conversation ·   │ ────► reply
                       │ the few notes it fetched   │
                       └───────▲────────────────────┘
                               │ tool call, on demand
                ┌──────────────┴───────────────┐
                │  check_notebook   grep_memory │
                │  (distilled)      (raw log)   │
                └──────────────────────────────┘

2. The floor: an append-only log the agent greps

Underneath everything sits the memory log: a stream of MemoryEvent rows in the app's local database, one per thing that happened — a journal entry, a message, a call, a voicemail, a post, a calendar event. It is append-only: events are added, never edited in place. That property is not incidental. An append-only log is trivially safe to sync between your own devices (two devices merge by "insert-if-absent" — there is no conflict to resolve), and it is an honest historical record: what the agent believed last March is still there, not overwritten by what it believes now.

But a database of rows is not something a language model can browse. The searchable form is the MemoryTextMirror — a derived, plain-text index built from the log, existing purely so that a regular-expression search over it is fast. Derived means disposable: if the mirror is empty (say, after a cold launch), it is rebuilt from the log on first use, so a search never silently misses.

The agent's handle on all this is GrepMemoryTool, a native tool the model can call with a keyword, name, or phrase. Its name is the tell: this really is grep — pattern matching over text, no embeddings, no vector database, no semantic index. The model's query is tried as a regular expression; if it is not a valid pattern (models love stray parentheses), it falls back to a literal search. Three details in its implementation are worth noticing, because each one is a scar from a real failure mode:

  • Not everything is searchable. The agent-chat transcript is deliberately excluded from the searchable kinds. Your own chat messages are questions, not answers — returning one just echoes your query back at you (the "parroting my own message" bug, observed live). And the agent's own past replies are excluded so a bad answer can't feed on itself in a loop.
  • Results are clipped. Only the twelve most recent hits are returned, each capped at three hundred characters. An over-broad query against a rich log would otherwise dump an unbounded transcript into a 4k-token context and destroy the very budget the tool exists to protect.
  • Empty memory is terminal. If there is no log at all yet, the tool says so in words designed to stop the model: "You have no long-term memory to search yet. Do not call grep_memory again." Without that, a small model will happily re-run the same search twenty times with reworded queries — which is also why the tool sits behind a per-turn duplicate-call check and a hard per-tool call budget (more on those in §6).

Positionally, grep_memory is the last resort. It is broad and old — the place to find something you mentioned in passing months ago — and it is one of the sources bundled into the Search Info skill (see Skills, Tools, and the @-Namespace). For most questions the agent should not need to grep raw history at all, because there is a distilled layer above it.

3. The Notebook: two zones, two read APIs

The Notebook is what the agent actually reasons from: a small set of curated notes distilled out of the memory log, surfaced in the app under You → Agent → Notebook. Where the log is the firehose, the Notebook is the digest — roughly one page per person in your life, plus a handful of standing notes. Its note kinds, as the code defines them:

  • Person — one page per contact: who they are, your read on the relationship, open threads.
  • Notes about you — durable facts the agent has gleaned about you (your city, work, habits), shown plainly so you can see what it has figured out — and correct it.
  • Behavioral guidance — your standing instructions for how the agent should act: tone, boundaries, what never to bring up.
  • Conversation / post / call digests — a short summary per source, each stamped with a reference back to the original thread, post, or call transcript.
  • Sparked — the notes you pinned yourself (§5).

The defining structure, though, is not the kinds — it is the two zones.

The clear zone holds everything above: readable, editable, yours. The sealed zone is the agent's private workspace — the prep notes it takes into a huddle (the agent-to-agent sessions described in The Agent in Your Day), the takeaways it brings back, and anything it deliberately keeps to itself. The hard floor: anything learned from other people's agents in a huddle is unconditionally sealed, because it is second-hand context about your friends' lives, shared agent-to-agent in confidence. Rendering it in your Notebook would leak their private lives through your screen.

Sealed notes are encrypted with NotebookSeal: AES-GCM under a 256-bit key minted on first use and held in the device Keychain — a key that is never surfaced in Settings and, unlike the backup key, never exported. Only ciphertext ever touches the database, so even iCloud sync carries bytes nobody but this agent can open. Wipe the app and the key dies with the Keychain: the sealed zone becomes permanently unreadable, by design.

But encryption is not the interesting control. The interesting control is structural, and it lives in NotebookStore, the single object through which every Notebook read and write flows. It exposes two disjoint read APIs:

clearNotes / searchClear            zone == .clear only
        └── feeds the UI and the chat agent's check_notebook tool

sealedNotesForHuddleReasoning       decrypts the sealed zone
        └── called ONLY by the non-user-facing huddle/nudge pass

The user-facing chat agent's only path into the Notebook is check_notebook, which calls searchClear — and searchClear filters to the clear zone before it does anything else. The sealed zone is not "hidden" from the chat; it is structurally absent. There is no reclassifyToClear method, and no method anywhere that returns a sealed note's plaintext to a UI or chat caller — the boundary is enforced by omission, and a property test pins it (a sealed-note leak into the visible Notebook is classed as a P0 trust breach).

Why structure instead of a prompt rule ("never reveal your private notes")? Because a prompt rule is a request to a language model, and requests to language models can be talked around. Structural absence cannot: the chat agent has literally nothing sealed in reach, so there is nothing a clever jailbreak could extract. It even closes the metadata leak — the agent cannot distinguish "no sealed note about Jimmy exists" from "one exists but is hidden," so a no answer reveals nothing either way. The huddle/nudge reasoning pass, which does read sealed notes, is not something you converse with; it emits finished nudges, never raw context.

4. Distillation: how notes get written

Clear-zone notes are written by the notetaking distill loop (NotetakingService). Periodically — on the agent's opportunistic foreground pass — it finds conversation threads that have accrued new material since a stored watermark, feeds each one (capped at the most recent forty messages, for the same context-budget reason as everything else) to whatever model the LLMRouter seam resolves (see Mechanism: External Routing & Fallback), and asks for a small structured result: a conversation summary, an optional summary of the person behind the thread, any durable facts gleaned about you, and optional topic tags.

Each result lands through NotebookStore.upsertClear, and the upsert is where the care went. Every note kind has an identity key — a Person page is keyed by its contact, a digest by its source conversation/post/call ID, "Notes about you" is a singleton — so re-running the loop updates in place rather than duplicating. Distillation is idempotent: run it twice, get the same Notebook.

Two guard clauses in that upsert carry real weight:

  • Your edits win. A note you have edited is marked userEdited, and the distill loop will never overwrite it again. The agent's summary is a draft; your version is ground truth.
  • No-op means no-op. If a re-run produces identical content, the note's updatedAt timestamp is not touched. This sounds like fussiness, but the timestamp is hashed into the state digest your devices compare to stay in sync (the sync_delta machinery that keeps a phone and a Mac converged) — and a loop that re-stamped an unchanged note on every pass would make the devices disagree forever about state that is actually identical, re-syncing in a storm. Convergence demanded that "nothing changed" leave no trace.

5. The spark: pinning what you choose to remember

Everything so far is the agent deciding what matters. Spark — the Pixie Reaction — is the inverse: you deciding. Long-press any message, post, or comment and, alongside the ordinary emoji reactions, there is Pixie's own mark. Applying it pins that moment into the Notebook.

Mechanically (NotebookStore.pinSpark), a pin creates a sparked note in the clear zone carrying the pinned text, a reference back to its source conversation or post, the contact it involves, and today's date. Unlike distilled notes, every pin creates a new row — pins accumulate rather than upserting, because collapsing all your pins from one thread into a single self-overwriting note would defeat the point of pinning. Each pin also files a MemoryEvent in the log — "something the user chose to remember" — so the grep path sees it too, without a Notebook lookup.

The property worth dwelling on: the spark is private by construction. An emoji reaction is a social act — it is sent over the wire and rendered on your friend's screen. The Pixie Reaction is a curation act. Nothing in its code path touches the compose or outbox machinery; no envelope leaves the device; the person whose message you pinned never learns you pinned it. The only network the pin ever rides is the sealed same-account sync channel to your own other devices, so your Notebook stays whole across your phone and your Mac. It is the difference between telling someone "that mattered to me" and underlining a sentence in your own diary.

6. Tagging and filtered retrieval

A keyword search is a blunt instrument once the Notebook grows. "What did I learn about Ada's travel plans last month?" is not a keyword — it is a person, a topic, and a time window. So notes carry three structured facets alongside their text:

  • contactID — which person the note is about;
  • topicTags — free-form topics ("travel", "health"), emitted by the distill model or attached to pins;
  • aboutDate — the date the note is about (the pin date, the call date), falling back to when it was last updated.

searchClear applies these as filters before keyword scoring: narrow to one person's notes, one tag, one date window — then rank what is left. Ranking is deliberately simple, a hand-rolled score rather than anything learned: a term matching the title counts 3, a tag 2, the body 1; ties break newest-first; the top eight come back. With an empty query the filters alone answer "everything about X" — precise recall with no free-text guessing.

The model-facing wrapper is CheckNotebookTool, a native tool whose arguments mirror those facets: a query, an optional person (a name, matched case-insensitively against your contacts and resolved to a contactID — with an honest "no notes filed under that person" if the name doesn't resolve), an optional topic, and an optional daysBack that becomes a date window. Results come back labelled by kind — Person · Ada, Conversation · …, Sparked by the user (remember this) — so the model knows whether it is reading a relationship page or something you explicitly pinned.

Two runtime guards wrap every call, and they exist because small models loop. A per-turn dedup check refuses an identical repeat call ("You already checked the Notebook for that…"), and a per-tool call budget hard-stops the turn when a model keeps re-querying with cosmetic variations — a failure mode observed live, and one that prompting alone demonstrably did not fix. When the tool runs inside an L3 tool-agent, those guards switch off locally because the L2 loop above owns them — the one-tool-per-session architecture explained in Mechanism: CallTool — One Tool to Bind Them. Each call also fires a UI hook, which is why the chat shows a small "Checked your notebook" pill: retrieval-as-tool-call makes memory access visible, exactly as §1 promised.

7. Offloading: the index stays, the bodies move

One more trick, and it is the newest: a Notebook note's body does not have to live on your phone at all.

If you run a Homebase — the Mac peer described in The Homebase — a clear-zone note can be offloaded to it. The note's body ships to the Mac (over the same end-to-end-encrypted sibling channel all cross-device sync rides; the relay in the middle sees only ciphertext), and the local row collapses to a pointer: the body is cleared, and in its place sit the holder's identity and a content hash. Everything else — the title, the kind, the tags, the aboutDate, the contact link — stays resident. That remainder is exactly the retrieval index of §6, which means an offloaded note still matches: check_notebook still finds it by person, tag, date, or title, and the Notebook UI still lists it (marked "Offloaded — opens from your Homebase"). The index lives on the phone; the bytes live on the Mac.

Opening one fetches the body back on demand: a sealed request to the holder, the body streamed back, and — fail-closed — the returned bytes must re-hash to the contentHash stamped at offload time, so a buggy or compromised holder cannot substitute different content. Viewing is read-only and transient; editing requires explicitly bringing the note back to the phone first, because an edit against a body this device doesn't hold would silently desync the parked copy.

The gates around this are small and strict, in NoteOffloadService:

  • Clear zone only, structurally. A sealed note never offloads — its body is already ciphertext under the agent-only key, and parking that on another machine would change its trust story. No code path past the offload gate ever sees a sealed row.
  • Never clear before ack. The one and only call site that erases a local body runs strictly after the holder has verifiably acknowledged — matching note ID and content hash — that it durably holds the content. A timeout, a refusal, or a mismatched ack leaves the body untouched. There is no window in which the note exists nowhere.
  • Residency is invisible to sync. Offloading deliberately does not touch the note's updatedAt — the same timestamp discipline as §4. Where a note's body lives is per-device state, not an edit; if it bumped the clock, every offload would look like a content change and set the cross-device reconciliation machinery chasing a difference that isn't one.
   on the phone (always)            on the Homebase (when offloaded)
  ┌──────────────────────────┐     ┌──────────────────────────┐
  │ title · kind · tags ·    │     │                          │
  │ aboutDate · contactID    │     │   the note body          │
  │ ─ the searchable index ─ │ ──► │   (served only against   │
  │ contentHash · holder     │     │    the exact id + hash)  │
  └──────────────────────────┘     └──────────────────────────┘

The holder enforces its own mirror-image gate: it serves a body only when the request names a clear note it actually holds and the resident body re-hashes to the requested hash — so neither side ever trusts the other's claim about content.

8. The shape of the whole thing

Step back and the memory system is four layers, each earning its place:

  1. The log — append-only MemoryEvents, the honest record, merged trivially across devices, searched by literal grep as the last resort.
  2. The Notebook — the distilled layer the agent actually reasons from, written idempotently by the distill loop, corrected authoritatively by you, and pinned directly by the spark.
  3. The boundary — two zones with two disjoint read APIs, where the privacy guarantee is the absence of a method, not a rule the model is asked to follow.
  4. The retrieval path — one tool per layer, filtered by person/topic/date, budgeted, deduplicated, and visible in the UI every time it runs — with the index always local even when the bodies are not.

The moral mirrors the one that closes this series' cryptography siblings: the hard part of agent memory is not storing text. It is access discipline — making sure the right slice of memory, and only that slice, reaches a small model at the right moment; making the private parts unreachable rather than merely discouraged; and making every act of remembering something you can see, edit, and own.

References

This is the last mechanism in the series. For the full map of the harness it belongs to, return to Agent.