The problem
Large language models are stateless: every conversation starts from zero, and anything you taught the assistant yesterday is gone today. Existing memory tools tend to be bolted onto a single app for a single user — the knowledge can't follow you across assistants, and it can't be shared safely between people.
The approach
Recall is a standalone memory service that speaks the Model Context Protocol (MCP), so any MCP-compatible client — Claude, IDE agents, custom tools — can store and retrieve knowledge through the same server. Three design decisions shaped it:
- Hybrid retrieval, not just vectors. Memories are embedded with Voyage AI and stored in Postgres via pgvector, but semantic similarity alone misses exact identifiers and names. Recall runs vector search and Postgres full-text search side by side, then fuses the ranked lists with Reciprocal Rank Fusion (RRF) so both kinds of relevance count. Notes are chunked before embedding, so retrieval can match a specific passage while still returning the parent note for context.
- Multi-user from day one. OAuth sign-in with GitHub and Google via Auth.js, with sessions carried by EdDSA-signed JWTs, so every memory is scoped to its owner rather than to one global store — and per-user isolation is enforced at the API layer, not just in the UI.
- Reachable from anywhere an agent runs. The knowledge base is exposed to LLM clients such as Claude Desktop and Cursor through a Streamable HTTP MCP server, and to the terminal through a Typer CLI. Both authenticate with personal access tokens, so the same store is available from an editor, a chat client, or a shell script.
- Real infrastructure, not a demo. The service is deployed the way a production API would be — containerized, load balanced, cached, and shipped by CI/CD with no long-lived cloud credentials.
Architecture
- API: async FastAPI — async SQLAlchemy over asyncpg — containerized and running on AWS ECS Fargate (ARM64/Graviton for better price/performance), behind an Application Load Balancer with HTTPS via ACM.
- Network: a custom VPC, with secrets held in AWS Secrets Manager and logs and metrics in CloudWatch.
- Storage: RDS Postgres with the pgvector extension — one database for both structured data and vector search.
- Cache: a self-hosted Valkey (Redis-compatible) sidecar for hot paths and session data.
- Interfaces: a Streamable HTTP MCP server for agents, a Typer CLI for the terminal, and a Next.js dashboard and landing page deployed on Vercel.
- CI/CD: GitHub Actions authenticating to AWS with OIDC — deploys use short-lived federated credentials, so no AWS secrets live in the repo.
Running it cheaply
Production-shaped infrastructure is easy to make expensive. Two choices cut roughly $35/month out of the bill without giving up the architecture: NAT-free networking, which removes the standing hourly cost of a NAT gateway, and a self-hosted Valkey sidecar in place of managed ElastiCache. ARM64 Graviton tasks on Fargate do the rest.
Highlights
- Hybrid semantic + keyword search fused with Reciprocal Rank Fusion.
- Multi-user OAuth (GitHub & Google) with EdDSA-signed JWT sessions and per-user isolation at the API layer.
- Streamable HTTP MCP server plus a Typer CLI, both authenticated with personal access tokens.
- Chunked embeddings with parent-note retrieval for granular matches with full context.
- Keyless deployments: GitHub Actions → AWS via OIDC federation.
- ARM64 (Graviton) containers on Fargate, ~$35/month saved through NAT-free networking and a self-hosted cache.
- Works with any MCP-compatible assistant, not one specific app.
Stack
- Python
- FastAPI
- async SQLAlchemy + asyncpg
- PostgreSQL + pgvector
- Voyage AI embeddings
- Valkey (Redis)
- Docker
- AWS ECS Fargate (ARM64)
- RDS
- ALB / ACM HTTPS
- Secrets Manager
- CloudWatch
- GitHub Actions + OIDC
- Next.js
- Vercel
- Auth.js (OAuth)
- MCP
- Typer CLI