AI Agent Memory Systems: Short-Term, Long-Term, and Vector Memory

AI agent memory systems store and retrieve information across interactions using three tiers: short-term memory (the current context window, 128K-2M tokens), long-term memory (persistent facts in a database), and vector memory (semantic retrieval from embeddings). Production agents combine all three — conversation in context, user facts in Postgres, and knowledge in pgvector — with retrieval accuracy above 90% when each tier handles what it's built for.

The Three-Tier Memory Architecture

The tiers answer different questions: short-term is "what are we talking about right now," long-term is "what do I know about this user," vector is "what relevant knowledge exists."

Short-Term Memory: Managing the Context Window

The context window is your fastest and most expensive memory. Three management patterns:

Sliding window

def build_context(history, max_turns=20):
    return history[-max_turns:]  # simplest; loses old context

Summarization + recent turns

def smart_context(history, llm):
    if len(history) <= 12:
        return history
    summary = llm.chat(
        model="gpt-5-mini",  # cheap summarization
        messages=[{"role":"user","content":
          f"Summarize key facts/decisions: {history[:-8]}"}]
    )
    return [{"role":"system","content":f"Context: {summary}"}] + history[-8:]

Token-budget packing

def pack_context(history, budget=32000, tokenizer=None):
    packed, total = [], 0
    for msg in reversed(history):
        cost = len(tokenizer.encode(msg["content"]))
        if total + cost > budget:
            break
        packed.insert(0, msg)
        total += cost
    return packed

Summarization+recent wins for chat; token packing wins for agentic loops where every instruction matters.

Long-Term Memory: The User Fact Store

Structured facts — name, preferences, project details, past decisions — belong in SQL, not embeddings. Exact recall, instant updates, no similarity noise:

CREATE TABLE user_memory (
  id BIGSERIAL PRIMARY KEY,
  user_id UUID NOT NULL,
  memory_type TEXT NOT NULL,   -- 'preference' | 'fact' | 'decision'
  key TEXT NOT NULL,           -- 'timezone'
  value TEXT NOT NULL,         -- 'Asia/Shanghai'
  confidence FLOAT DEFAULT 1.0,
  created_at TIMESTAMPTZ DEFAULT now(),
  updated_at TIMESTAMPTZ DEFAULT now(),
  UNIQUE(user_id, key)
);

-- Write path: extract facts after each session
async def extract_facts(session_text, user_id):
    facts = await llm_json(model="gpt-5-mini", prompt=f"""
      Extract durable user facts from this conversation.
      Return JSON list of {{type, key, value}}.
      Only facts that persist beyond this session.
      Conversation: {session_text[-4000:]}
    """)
    for f in facts:
        await db.upsert_memory(user_id, **f)  # ON CONFLICT update

# Read path: inject into every future session
system_prompt += f"\nKnown user facts: {format_facts(memories)}"

The write path runs on cheap models (gpt-5-mini at $0.15/1M handles extraction well); the read path is a single indexed query. This is also how you implement "the AI remembers me" without re-reading every past conversation.

Vector Memory: Semantic Knowledge Retrieval

When knowledge exceeds what fits in context — documents, codebases, support history — vector retrieval finds the relevant slice:

-- pgvector: same Postgres, no new infrastructure
CREATE EXTENSION vector;
CREATE TABLE knowledge (
  id BIGSERIAL PRIMARY KEY,
  content TEXT NOT NULL,
  embedding vector(1536),  -- text-embedding-3-small
  metadata JSONB
);
CREATE INDEX ON knowledge 
  USING hnsw (embedding vector_cosine_ops);

-- Retrieval: top-5 relevant chunks
SELECT content, 1 - (embedding <=> $1) AS similarity
FROM knowledge
ORDER BY embedding <=> $1
LIMIT 5;

Chunking strategy matters more than model choice: 256-512 token chunks with 15% overlap, split on semantic boundaries (paragraphs, function definitions), consistently beat naive fixed-window splitting on retrieval accuracy.

Putting It Together: The Memory Manager

class AgentMemory:
    def __init__(self, user_id):
        self.user_id = user_id
        self.short_term = []       # this session's messages

    async def context_for(self, new_input):
        # 1. Long-term facts: exact, always included
        facts = await self.load_facts(self.user_id)
        
        # 2. Vector retrieval: only if input needs knowledge
        knowledge = await self.retrieve(new_input, top_k=5) \
            if self.neits_knowledge(new_input) else []
        
        # 3. Short-term: summarize if over budget
        history = self.pack(self.short_term, budget=24000)
        
        return compose(facts, knowledge, history, new_input)

    async def commit_session(self):
        # Extract durable facts → long-term store
        await self.extract_facts(self.short_term)
        self.short_term = []

Memory Failure Modes and Fixes

Cost Profile of Memory Operations

Memory adds both latency and tokens. Budget roughly:

• Fact injection: 200-800 tokens per request (trivial cost, huge coherence win)
• Vector retrieval: 1-3K tokens of retrieved context when triggered; the pgvector query itself is 5-20ms
• Session summarization: one mini-model call (~$0.0003) per session end
• Fact extraction: one mini-model call per session (~$0.0005)

Full memory stack adds under $0.001 per session while making agents feel continuity-aware. DrAI's gateway handles the model calls (routing extraction to cheap models automatically), and the built-in knowledge base implements this exact pgvector pattern — try it free at ai.dr-ai.top/signin. For deeper retrieval design, see the RAG implementation guide and vector database comparison.

Production memory at scale — compare per-model pricing and pick the cheapest capable models on the DrAI pricing page.