AI Agent Memory Planner

Agent memory refers to the mechanisms that allow AI agents to retain and access information across interactions. Types include: in-context memory (tokens in the active window), external memory (vector databases, key-value stores), episodic memory (past conversation logs), and semantic memory (distilled knowledge). Each type has different cost and latency characteristics.

The context window is the maximum tokens an LLM can process at once. Larger windows allow more in-context memory but increase cost quadratically for attention computation. For agents that need long-term memory, RAG (retrieval-augmented generation) is often more cost-effective than simply expanding context.

RAG (Retrieval-Augmented Generation) stores information in a vector database and retrieves only the most relevant chunks at inference time. Instead of keeping all knowledge in the context window, you retrieve top-K chunks (typically 3-10). This reduces token costs dramatically while maintaining access to large knowledge bases.

Most managed vector databases (Pinecone, Weaviate Cloud, Qdrant Cloud) charge per vector stored and per query. A typical 1536-dimension OpenAI embedding costs ~$0.0001/vector to store monthly on Pinecone. 1M documents might cost $50-100/month in storage plus query costs.

Chunk size depends on your content and model. Common sizes: 256 tokens (precise retrieval), 512 tokens (balanced), 1024 tokens (more context per chunk). Smaller chunks = more precise but may miss context. Overlap of 10-20% between chunks helps maintain coherence.