Memory Is Easy.
Retrieval Is Hard.

When we first built Rheo, our internal AI assistant at Novadiem, we assumed memory would be the difficult part.

It wasn't.

The difficult part was getting Claude to reliably use the memory we had already stored.

The first version of Rheo forgot things constantly. A user would send a message, then follow up ten minutes later:

We had conversation history. We had search tools. We had a memory API. None of it mattered.

The bottleneck was retrieval, not storage.

The architecture that finally worked has three memory layers: an append-only conversation ledger, compressed episodic summaries, and durable memory items. A fourth piece, boot-time context injection, ties them together.

01The setup

Rheo runs as a headless assistant on a server. A Flask webhook receives Telegram messages and shells out to claude -p. One process per incoming message. The response returns to Telegram.

Without external memory, every message is a cold start. Claude knows nothing about earlier sessions.

The obvious solution (stuffing raw history into every prompt) works at first, then falls apart as history grows. The context gets noisy, the token bill climbs, responses slow down.

02The architecture

The final design uses three memory layers:

03Layer 0: The conversation ledger

Every turn is stored permanently in an immutable table. This is the canonical, append-only record: no edits, no summarization, no cleanup. An FTS5 index enables fast keyword search via a chat_search tool, and sessions are detected server-side: a gap greater than two hours from the same chat_id opens a new one.

CREATE TABLE conversation (
  id            INTEGER PRIMARY KEY AUTOINCREMENT,
  chat_id       TEXT NOT NULL,
  role          TEXT NOT NULL,  -- 'user' | 'rheo'
  content       TEXT NOT NULL,
  ts            TEXT NOT NULL,
  session_id    TEXT
);

04Layer 1: Session digests

Raw history is expensive. At session close, a background task creates a compressed summary:

1. Fetch all turns for the completed session.
2. Send to claude -p (summarization-only, no tools).
3. Parse structured JSON output.
4. Store the digest.

Each digest includes a short narrative, candidate facts, procedural notes, and key decisions. At the start of a new conversation, the last five digests are injected, providing recent history without burning thousands of tokens on raw turns.

=== EPISODIC MEMORY ===

[2026-06-16 21:00]
Discussed Nutrifax invoice reconciliation.
Decided to postpone Stripe migration until July.

[2026-06-15 14:22]
Reviewed CryptoWatchTools redesign.
Preferred dashboard layout B.

05Layer 2: Memory items

Digests answer "what happened recently?" Memory items answer "what should never be forgotten?" Examples: user preferences, deadlines, relationships, project details.

Claude writes these mid-conversation via a dedicated write_memory MCP tool. The server enforces ownership (derived from source_turn_id) and makes all writes append-only.

{
  "type": "fact",
  "content": {
    "label": "Arowyn school",
    "properties": { "school": "Crawford Bay School", "grade": 9 }
  },
  "confidence": 0.9,
  "reason": "Explicit user statement",
  "source_turn_id": 12345
}

Why conflicts are preserved

Most memory systems overwrite old facts. We don't. When a new fact contradicts an old one, both records are kept. The older one gets a superseded_by pointer. High-confidence conflicts (both ≥ 0.8) raise a flag for review. So you get a history you can trace, instead of facts quietly overwriting each other.

06The critical mistake: assuming the model would retrieve

The plan was simple: store everything, give Claude retrieval tools (chat_recent, chat_search), and let it call them when needed. It failed in production.

When a follow-up message arrived without visible history, Claude would conclude it was a fresh session and respond with "I don't have context" instead of going looking for it.

Why?

A prompt with no recent history looks almost identical to a brand-new conversation, so the model treats it like one: it answers, it doesn't go hunting. We'd been treating "the tool exists" as "the tool gets used."

07The fix: boot-time injection

We stopped relying on the model to retrieve and started injecting context before Claude sees the user message. Every prompt is now assembled from four sources, and the most recent session turns are always in the prompt, so continuity works because the model never has to go find them.

=== OPEN M.O.T. TICKETS ===
... current tickets ...

=== EPISODIC MEMORY ===
... recent session digests ...

=== CURRENT MEMORY ===
... active durable facts ...

=== CURRENT SESSION ===
... last ~12 turns of active session ...

---
<current user message>

Example in action

A user says: "My daughter Arowyn starts Grade 9 this fall." → a durable memory item is created. Two weeks later: "What do I still need to prepare before school starts?" The fact is already in context under === CURRENT MEMORY ===. No search required. No risk of forgetting.

08What the design review missed

Our internal Challenger (fresh-context critic) caught many structural issues but missed this behavioral one. We'd designed for a competent autonomous agent. What we actually had was a reactive pattern-matcher that works off whatever happens to be in the prompt.

A perfectly designed retrieval system is useless if the model never reaches for it. And it won't, just because the tool is there.

The design was sound on paper. Whether the model would use it in practice was a behavioral question, and those only get answered in production.

09What's next

The current system handles within-session context and remembers across sessions. Next: entity-centric retrieval for questions like "What did we discuss about Nutrifax last month?" This will require FTS5 over memory items, entity graphs, and traversal tools, all gated on digest quality.

10What this costs

Boot-time injection isn't free. Each request constructs a prompt from four sources (tickets, digests, memory items, and recent turns) before Claude sees the user message. In practice this adds a few hundred milliseconds and roughly 1,000–2,000 tokens per turn depending on session length.

The compression layers keep this manageable: digests replace hundreds of raw turns with a few sentences, and memory items are capped. But there's a ceiling. Very long sessions or large memory sets will eventually push against it. We'd rather hit that ceiling and have a real problem to solve than stay stateless to avoid it.