Memory

Storage surfaces

Memory in Ultron lives in three places. Two are persistent stores in Postgres backed by pgvector. One is a runtime compression cascade applied to the conversation buffer before every turn. Together they give the chat loop continuity across turns, across sessions, and across days.

Updated today

Overview

At a glance

Persistent stores: 2 (agent_memories, brain_files + file_embeddings)
Runtime surface: 1 (conversation cascade)
Embedding model: 1536-dim via OpenAI text-embedding-3-small
Vector index: pgvector HNSW with cosine distance
Write entry points: save_memory tool, brain file upload, automatic post-skill summaries
Read entry points: Hybrid retrieval before every turn, in-tool searches via search_memory, file lookups via brain_search_files

The memory layer is structured so that the model is never asked to remember things on its own. Anything important is persisted into one of the surfaces by the engine, the user, or a skill, and pulled back at the start of the next turn that needs it. The model does not store state between calls; the surfaces do.

Each surface answers a different question. agent_memories answers what does the system know about this user. brain_files answers what documents has the user uploaded. The conversation cascade answers what just happened on this turn that the model needs to keep in its head.

Three surfaces

The full map at a glance.

Surface	Backing store	Shape	Read path	Write path
`agent_memories`	Postgres + pgvector	Short text + tags + 1536-dim embedding	Hybrid retrieval (judge + cosine)	save_memory tool, post-skill auto write
`brain_files` + `file_embeddings`	Postgres + Storage	Uploaded document chunked into embedded passages	brain_search_files (vector RPC)	Upload flow, async chunk + embed
Conversation cascade	Runtime, in-memory	Compressed view of the message buffer for the current turn	Always read at turn start	Generated on every turn, not persisted

agent_memories

The keystone surface. Anything worth remembering about a user ends up here.

supabase/migrations/202602080002_agent_memories.sqlsql

6create table agent_memories (

7 id uuid primary key default gen_random_uuid(),

8 agent text not null, -- skill or persona id that wrote the row

9 text text not null, -- the remembered statement

10 embedding vector(1536), -- OpenAI text-embedding-3-small

11 tags text[] not null default '{}',

12 metadata jsonb not null default '{}',

13 user_id uuid references auth.users(id) on delete cascade,

14 created_at timestamptz not null default now()

15);

17create index agent_memories_embedding_hnsw_idx

18 on agent_memories using hnsw (embedding vector_cosine_ops);

20create index agent_memories_user_tags_idx

21 on agent_memories using gin (user_id, tags);

Every row is one short statement. The text is the canonical form; the embedding is the lookup key for semantic retrieval. Tags exist for explicit filtering (the note tag carves notes out of the same table, the persona:cortex tag scopes writes to one persona, and so on). Metadata is jsonb for whatever the writer wants to keep alongside the text without polluting it.

Note

The agent field is the id of the skill or persona that wrote the row. It is not the user. Filtering by agent retrieves entries created by a specific skill, which is useful when one skill wants to read its own prior output without seeing every other skill's notes.

brain_files + file_embeddings

The two-table pair behind uploaded documents.

supabase/migrations/20260414_brain_files_table.sqlsql

1create table brain_files (

2 id uuid primary key default gen_random_uuid(),

3 user_id uuid not null references auth.users(id) on delete cascade,

4 file_name text not null,

5 file_size bigint,

6 content_hash text, -- dedupe key

7 storage_path text not null, -- Supabase Storage

8 status text not null, -- 'processing' | 'ready' | 'failed'

9 chunks_count int,

10 created_at timestamptz not null default now()

11);

13create table file_embeddings (

14 id uuid primary key default gen_random_uuid(),

15 user_id uuid not null references auth.users(id) on delete cascade,

16 file_name text not null,

17 chunk_index int not null,

18 content text not null,

19 embedding vector(1536),

20 metadata jsonb,

21 created_at timestamptz not null default now()

22);

brain_files tracks the file itself: name, size, content hash, storage path, status, chunk count. file_embeddings tracks the parsed chunks: one row per passage, each with the chunk text and its embedding. Searches happen over file_embeddings; the file row exists so the UI can list uploads, show progress, and dedupe by hash.

Tip

A file in status processing is uploaded but not yet chunked or embedded. The chunker runs asynchronously after upload. Searches against file_embeddings only return passages for files in status ready.

Conversation cascade

Not a table. The runtime compression that produces what the model sees on this turn.

On every turn the chat handler loads the recent message buffer, applies a four-layer compression cascade, and gives the model the compressed version. The cascade only fires the layers it needs to fire to fit the token budget. The full architecture, with the rules each layer applies and the order in which they cascade, lives on the Conversation compression page.

How memory enters and exits

A grounded sketch of write and read paths from the perspective of one turn.

Event	Surface	Path
User mentions a fact worth remembering	agent_memories	Model calls save_memory tool; addMemory writes a row with embedding
Skill finishes a mission	agent_memories	Engine auto-writes a summary entry tagged with the skill id
User uploads a PDF	brain_files + file_embeddings	POST /api/brain/files creates a file row, async chunker fills file_embeddings
User creates a note in Brain	agent_memories	POST /api/brain/notes inserts a row tagged 'note'
Turn starts	All three	Hybrid retrieval reads agent_memories; brain_search_files reads file_embeddings on demand; cascade prepares the message buffer
User edits a memory	agent_memories	PATCH on agent_memories with re-embed
Conversation deleted	agent_memories	Cascade delete does not touch agent_memories; memories persist by design

Tenant and persona isolation

Every read is scoped. Every write carries the user id.

Both persistent tables include a user_id column with a foreign key to auth.users and a cascade-delete relationship. Row-level security ensures that a given user can never read or write another user's rows. Skills that filter further (by persona, by project, by tag) do so on top of the user-level isolation, never as a replacement for it.

Persona scoping is opt-in. A skill that wants to keep its memories separate from neighbours writes with a persona tag and reads with the same tag filter. Skills that want the full picture omit the filter and see everything the user has accumulated.

TTL and pruning

Memory is intentionally durable. The engine does not garbage-collect entries.

Neither agent_memories nor brain_files has a TTL. Entries persist until the user removes them or the account is deleted. This is deliberate: a missing memory is a regression the user notices, while a stale memory the user can edit or delete from the Brain UI. The cost of keeping rows is small; the cost of dropping a useful one is high.

Conversation-level state is the exception. Cascade compression discards content on every turn that exceeds the budget. The discard is local to the turn buffer; it does not delete anything from the persistent tables.

File map

src/lib

agentMemory.tsaddMemory, searchMemories, recallMemories, deleteMemory

brain-tools.tsbrain_search_files executor

skills

memory.tscascadeCompress + compressConversation

src/app/api/brain

notes/route.tsGET/POST/DELETE on tagged agent_memories rows

files/route.tsUpload + chunk + embed + list

supabase/migrations

202602080002_agent_memories.sql

20260414_brain_files_table.sql

20260414_file_embeddings.sql

Glossary

Surface: One of the three places memory lives. Two persistent (agent_memories, brain_files + file_embeddings) and one runtime (cascade).
Embedding: A 1536-dimensional vector representing the semantic content of a text. Stored alongside the text and indexed with HNSW for cosine search.
Tag: A short string attached to an agent_memories row. Used for filtering (e.g. note, persona:cortex, project:acme).
Metadata: A jsonb column on agent_memories for arbitrary structured data the writer wants kept alongside the text.
Chunk: One row in file_embeddings. A passage extracted from an uploaded file, with its own embedding.
Cascade: The four-layer compression that runs on every turn to shrink the message buffer. Not a table, not persisted.