Building an AI Memory That Outlives Any Single Agent

TL;DR: Every hour you spend teaching an agent your codebase, your preferences, your project history is an investment — and right now, most people are making that investment inside one vendor’s proprietary memory format. Nobody knows who wins the model race, and the leaderboard has already changed hands more than once this year. The durable move is to keep your knowledge as plain, git-versioned files that live outside any single tool, and let the agent’s own config be a thin pointer into that vault instead of the vault itself. Any agent can pick it up cold. Zero ramp-up, regardless of who’s winning when you switch.

Nobody knows who wins, and that’s the point

I’m not going to pretend I know whether Anthropic, OpenAI, Google, or someone that hasn’t been founded yet ends up with the durable lead in agentic AI. Nobody credible is claiming that with a straight face right now. The frontier leaderboard has changed hands repeatedly over the last eighteen months — Stanford’s 2026 AI Index puts it plainly: “U.S. and Chinese models have traded places at the top of performance rankings multiple times since early 2025” (Stanford HAI, 2026 AI Index Report). That’s the churn at the level tracked across countries — inside just the U.S. labs, the same reshuffling happens lab-by-lab, with different vendors leading different benchmarks in the same quarter, let alone the same year.

That instability isn’t really the risk — it’s normal at this stage of the race. The risk is what most people building agent workflows haven’t priced in: while the leaderboard churns, you’ve been quietly accumulating months of project context, coding preferences, and institutional history inside one vendor’s tool. Every session you run teaches an agent something about how you work. That accumulated context has real value — and if it’s trapped in a proprietary chat history or a tool-specific config format, its value is hostage to that vendor’s roadmap, not yours.

This isn’t hypothetical vendor-switching anxiety. It already happened, on a real timeline, to a lot of people. OpenAI retired GPT-4o, GPT-4.1, GPT-4.1 mini, and o4-mini from ChatGPT this February, following a sunset schedule the company set on its own clock, not the user’s (OpenAI Help Center, “Retiring GPT-4o and other ChatGPT models”). If your accumulated memory lived inside conversations pinned to one of those models, or inside a tool whose whole interaction model assumed that model would always be there, you didn’t choose when to migrate. The vendor did.

What you’ve actually built, and where it’s sitting

Think about what a working agent relationship accumulates after a few months of real use. It’s not just “the AI knows my codebase.” It’s:

• Project-specific operational trivia — the port conflict, the flaky test, the deploy quirk — that took real sessions to surface and correct.
• Your standing preferences: how you like code reviewed, what you never want auto-committed, which shortcuts you’ve explicitly banned after they bit you once.
• A running history of decisions and their reasoning, so the agent stops re-litigating settled questions.
• The connective tissue between your projects — which pattern you use where, and why one repo does something differently than another for a documented reason.

I’ve written before about the first two layers of this — a CLAUDE.md as the onboarding doc you never wrote for humans, and the larger structured knowledge base behind it, the three-layer wiki architecture of immutable sources, model-written synthesis, and an index. Both of those posts are about the shape of good agent memory. This one is about a question neither of them answers: where does that memory actually live, and what happens to it the day you switch tools?

For a lot of setups right now, the honest answer is: inside the tool. A vendor’s chat history that isn’t exportable in a form another agent can consume. A config format specific to one CLI. A “memory” feature marketed as magic that’s really a database row you can’t git clone. None of that is a bug in any one product — it’s just what happens by default when a tool’s job is to keep you inside it.

The failure mode: your memory, hostage to someone else’s roadmap

Here’s the scenario that should worry you more than it currently does. You’ve spent six months building up real context inside an agent tool — codebase quirks, project history, your accumulated preferences. Then one of three things happens, on the vendor’s timeline, not yours:

1. Pricing changes. The tier you’re on gets more expensive, or the usage model shifts in a way that makes your workflow uneconomical.
2. The model gets deprecated or the product gets sunset. Not hypothetical — see above. OpenAI’s own 2026 deprecation wave touched everything from legacy GPT models to the Assistants API, on schedules the company set unilaterally.
3. Someone else ships a genuinely better agent, and staying put starts costing you the productivity gap between what you have and what’s now available.

None of those are hypotheticals — vendors make calls like this constantly, and each one is a completely reasonable business decision for them to make. The problem is entirely on your side of the relationship: if your accumulated context only exists as that vendor’s proprietary artifact, “switch to the better tool” now means “rebuild months of context from scratch,” and that switching cost is exactly the thing that keeps people locked into a worse tool past the point it makes sense. That’s not a hypothetical about AI — it’s the same lock-in economics that’s played out with every walled-garden platform before this one, just compressed into a market that changes leaders every few months instead of every few years.

The alternative: a memory layer that lives outside any vendor’s walls

The fix isn’t complicated, and it isn’t new — it’s the same discipline that’s always protected people from platform risk: keep the asset in a format you own, and treat any single tool as a client of that asset, not its owner.

Concretely, that means:

• Plain, structured files — markdown and JSON, nothing proprietary, nothing that needs a specific app to open. Any agent that can read a filesystem can read your memory.
• An index file that catalogs what’s there, so any agent — not just the one you trained — can orient itself in one read instead of scanning everything.
• Git for versioning, because you want history, diffs, and the ability to roll back a bad edit, and because git is itself a portable, vendor-agnostic standard that predates every AI tool by two decades and will outlive most of them.
• The whole thing living outside any single vendor’s application — not inside a chat history, not inside a tool-specific database, not inside a format only one CLI parses.

The agent’s own config file — a CLAUDE.md, an AGENTS.md, a system prompt, whatever your particular tool calls the thing it reads on session start — stops being the store of record and becomes a thin adapter. Its job shrinks to: here’s who I am, here’s where the real memory lives, go read it. That’s a much smaller surface to lose if you switch tools, because the valuable part was never in there to begin with.

This is the same instinct that shows up anywhere lock-in risk actually gets managed well — a thin integration layer over a third-party dependency, so the dependency can be swapped without touching everything built on top of it. Same principle, applied to your own accumulated knowledge instead of a vendor API.

Why the layered structure matters

A single flat folder of notes rots. That’s true whether the reader is human or an agent, and I’ve gone deep on the mechanics of a healthy layered wiki elsewhere — the three layers, the ingest/query/lint loop, the rules that keep it from turning into a junk drawer. For the portability question specifically, the layering matters for a slightly different reason: different kinds of memory have different shelf lives, and mixing them is what makes a vault stop being trustworthy.

Roughly, the useful split is:

• Durable facts — things true regardless of which project you’re on or which tool is reading: your standing engineering preferences, security defaults you never compromise on, how you like work reviewed. These barely change and should be readable by literally anything.
• Project-specific state — the operational trivia and history for this codebase or this engagement. Changes often, needs a clear “last updated” signal, and needs to be scoped so it doesn’t bleed into unrelated projects.
• Pointers to external systems — a link into a ticket tracker, a reference to a running service, a note that says “the real detail is over there.” These are explicitly not duplicated into the vault; the vault just knows where to look.

Collapse these into one undifferentiated pile and you get exactly the failure mode that kills most personal knowledge bases: nobody can tell what’s still true, what’s stale, and what was only ever relevant to a project that ended eight months ago. Keep them separated and dated, and the vault stays something you can actually trust years in, not just the week you built it.

Rehydration: any agent, same knowledge, zero ramp-up

The last piece is the mechanism that makes portability actually feel portable instead of theoretical: something that automatically loads the right slice of the vault at the start of a session, regardless of which agent is running.

In practice this is usually a session-start hook, or the equivalent your tool supports — a small script that runs before the agent’s first real turn and pulls in the durable facts plus whatever project-specific state is relevant to the current directory or task. The mechanism is deliberately boring: read a few files, inject them into context, done. The interesting part is what it implies. Because the hook is reading plain files from a vault you own, it doesn’t care whether today’s session is running one vendor’s CLI or a competitor’s — the vault doesn’t know or care which agent showed up to read it, and neither should you.

That’s the actual test of whether you’ve built this correctly: hand the same vault to a different agent tomorrow and see how long it takes to get productive. If the answer is “immediately, because it just reads the same files,” you’ve built a memory layer. If the answer is “I’d have to re-explain everything,” you’ve built a chat history, and it’s only ever going to be as durable as the app it lives in.

This is insurance, not paranoia

None of this requires you to predict the outcome of the model race, hedge across three vendors simultaneously, or treat every tool with suspicion. That’s not the ask, and it’s not how I run my own setup either. Use whichever agent is best for the job today — the field reshuffles too often to do otherwise.

The only discipline this actually requires is a habit, not a hedge: when an agent learns something worth keeping, it goes into files you own, not into a store only that tool can read. That’s a small tax paid continuously, and it buys you the thing that actually matters — the freedom to move to whatever’s better next quarter without paying for it twice. You don’t need to guess who wins. You need your knowledge base to not care who does.

If you’re standing up agentic workflows across a team and want a second opinion on where the real risk sits — memory portability, credential handling, or the security posture underneath the whole thing — that’s the kind of engagement I run.