Long-term memory grows unbounded and nothing evicts it

The symptom

Your long-term memory SQLite file keeps growing. A few hundred KB after a week, a few MB after a month, tens of MB after a quarter. Search quality degrades in a subtle way — older entries that aren’t relevant anymore keep surfacing, and you can’t figure out why the crew keeps “remembering” facts from six weeks ago that have since been superseded.

Why this happens

CrewAI’s long-term memory is designed to be durable — that’s the whole point of it — so by default nothing gets evicted. Every observation, every entity update, every memory write from every run is appended and kept. There is no TTL, no LRU eviction, no archival step. It’s a journal that only grows.

For a single-user demo this is fine. For a production system running thousands of crews a week, it is an unbounded resource leak with correctness consequences:

1. Storage grows linearly forever. Eventually you notice your SQLite file is hundreds of megabytes and copying it across deploys gets slow.
2. Search quality degrades. With LIKE-based retrieval and no relevance ranking, older stale entries can outrank newer relevant ones just by being more numerous.
3. Stale facts haunt you. An entity that had property X six months ago but has property Y today will keep surfacing X, because the memory contains both and nothing says “X is stale”.

Why this persists upstream

Retention policies are opinionated. Any default CrewAI ships will be wrong for someone — a research team might want infinite retention, a customer-facing product might want strict 30-day deletion for privacy, a compliance-sensitive workload might want immutable append-only. The upstream-safe answer is to ship nothing and let teams implement their own. That is what CrewAI has done, and it is defensible — but the practical effect is that most teams don’t implement a policy at all and get bitten later.

How Fast-CrewAI helps — and where you still need to do work

Fast-CrewAI does not ship a retention policy. That’s deliberate — see above — but we do make it much easier to implement one:

• The FTS5-backed memory index we install supports efficient date-range queries, which is what retention policies need. Pruning entries older than 90 days becomes a single DELETE instead of a scan.
• Connection pooling via r2d2 lets a background pruning task run in parallel with normal agent queries without contending for the main connection.
• BM25 ranking gives newer, more relevant entries a better chance of outranking stale ones even before pruning runs, so the degradation curve is gentler.

For the retention policy itself, you still write it. Ours looks roughly like:

import sqlite3
from datetime import datetime, timedelta

def prune_long_term_memory(db_path: str, max_age_days: int = 90):
    cutoff = (datetime.utcnow() - timedelta(days=max_age_days)).isoformat()
    with sqlite3.connect(db_path) as conn:
        conn.execute("DELETE FROM long_term_memory WHERE created_at < ?", (cutoff,))
        # FTS5 triggers will propagate the delete to the index automatically
        conn.commit()

Run it nightly on a cron. Adjust max_age_days to your workload. If you need per-tenant retention, scope the delete with a tenant_id filter.

Workaround you can ship today

Even without Fast-CrewAI, you can:

1. Add a created_at column to your long-term memory table if one isn’t there.
2. Write a pruning function similar to the one above.
3. Run it on a schedule.
4. Monitor the SQLite file size as a metric. Alert if it grows faster than expected.

The upstream fix is harder because it requires taking a stance on defaults. Individual teams can just pick one.

When it matters

Any CrewAI deployment that runs for more than a few weeks. If you’re past prototype stage and you haven’t audited your long-term memory, do it now. We include memory retention as part of every performance audit — it’s one of the most common silent failures.