Agent Memory Should Start as Files, Not a Database

---

title: "Agent Memory Should Start as Files Your Team Can Inspect" slug: agent-memory-start-with-files status: draft-pending-founder-publish primary_keyword: AI agent memory secondary_keywords:

• agent knowledge management
• AI memory for small business
• files vs vector database for agents
• human in the loop agent memory seo_title: "Agent Memory Should Start as Files, Not a Database" seo_description: "Build agent memory as inspectable files first. Separate what happened from what is currently true, keep sources attached, and only store knowledge that is cheaper to remember than to look up." excerpt: "Most agent memory projects start with a database. Start with files your team can read, keep evidence separate from current belief, and treat extra knowledge as a cache with a maintenance cost." source_url: https://x.com/devstein64/status/2094440843595706863 source_author: Devin Stein source_article: "How to Build Agent Memory: Where Does Knowledge Live?" cover_direction: "Premium light-background consultant whiteboard. Left column labeled Artifacts (Slack, tickets, transcripts). Right column labeled Current knowledge (short markdown files in Git). Arrows from artifacts into knowledge, each with a source link. A cache note: remember only if cheaper than looking it up. Sparse charcoal lines, bronze checkpoints, one vermilion stale-memory warning. Subtle Pratap icon only; no company-name text." cover_fallback: "https://www.pratap.ai/what-we-do-1.png" image_prompt: Marketing/Blog/Images/2026-09-01-agent-memory-start-with-files-image-prompt.md cta: 20-minute AI Opportunity Call company: Pratap AI Innovations created: 2026-09-01

---

Quick answer

Agent memory is two decisions, not one. First, decide the shape of the knowledge: a short note, a fact list, a folder of topics, or a graph of relationships. Second, decide where it lives. Storage is easy to change later. The shape you choose is what the agent, and your team, will actually use.

If you are building this for a founder-led operation, start with markdown files in Git. Agents already know how to search and read them. People can open the same files and see what the agent believes. Add databases, embeddings, and graphs only after a simple file layout fails in a specific way.

This reading is adapted from Devin Stein's X article How to Build Agent Memory: Where Does Knowledge Live?, then applied to how Pratap designs operator workflows.

Representation first, storage second

A memory can be a paragraph, a bullet list of facts, a topic tree, or a set of linked entities. Any of those can sit in a folder, Postgres, or a vector store. If you pick the store first, you usually lock the team into a shape they cannot inspect.

Files work early because coding agents already use directories, rg, and parallel reads. A WhatsApp-to-CRM workflow does not need a graph database on day one. It needs a small, named record that a person can open: need, constraint, confirmed facts, open question, owner. See customer context handoffs.

Keep artifacts and knowledge apart

Artifacts are records of what happened: a Slack thread, a ticket, a meeting transcript, a session log. They are high fidelity. They are not always true today. A design note can describe an architecture that was never shipped. A chat from last year can explain a process that has already changed.

Knowledge is what the agent should believe now. Treat it as a short, current view over the artifacts, with links back to the originals. Summaries are lossy. That is useful when it reduces noise. It is dangerous when the source is thrown away.

Practical rule for operators:

• Confirmed facts stay labeled as confirmed.
• Open questions stay labeled as open.
• If a summary disagrees with the original customer message, the original wins and a person reviews it.

That is the same discipline as a human-in-the-loop workflow: automation can prepare the card. It should not silently promote a guess into policy.

Current knowledge goes stale

Artifacts are history. Once a message was sent, it was sent. Knowledge is a claim about the present. If memory says "we confirm visits by WhatsApp only" and the team moved to a booking link last week, the memory is not slightly outdated. It is wrong.

Git is a useful model here. You can see the current file, the diff, and when the belief changed. Feature work also needs a temporary overlay: an agent implementing a new process should not rewrite company knowledge until that process is actually live.

For a small team, this can be as simple as:

1. Shared folder of current operating notes.
2. Task-specific notes that die when the task ends.
3. A person who merges a note into the shared folder only after the change is real.

Knowledge is a cache with a bill

Remembering everything is a poor goal. Artifacts are cheap to keep. Canonical explanations are not. The moment you write "this is how billing works," someone has to keep that sentence true.

Store knowledge when at least one of these is true:

• It is irreplaceable. A founder said a key customer depends on a quirk that exists nowhere else.
• It is expensive to rediscover. An agent spent twenty minutes tracing a process across tools and chats.
• It is used constantly, so looking it up every time wastes the day.

Everything else can be looked up from the live system: the CRM stage, the invoice status, the calendar slot, the config file. Memory should help with context. Systems of record should own facts.

A starting layout you can actually run

Create four folders and nothing more:

• artifacts/ original threads, tickets, transcripts, with dates.
• knowledge/ short current notes, each linking back to artifacts.
• tasks/ temporary overlays for work in progress.
• decisions/ the few rules a person approved, with an owner.

Do not embed the whole company on week one. Watch where agents fail:

• They cannot find the note: improve names and retrieval.
• They need relationships: add a little structure, not a new platform.
• They repeat the same expensive research: materialize that one result.
• Notes go stale faster than people can edit them: then design maintenance.

Complexity should be earned. Markdown in Git is not the final form of agent memory. It is the first form that both the agent and the founder can inspect.

Common questions

Should we start agent memory in a vector database?

No, not as the first store. Start with files your team can open. Add embeddings when search across a large set of notes is the actual failure.

What is the difference between artifacts and knowledge?

Artifacts record what happened. Knowledge is the current belief you are willing to act on. Keep the link between them so the agent can show why it believes something.

When should an agent be allowed to update company knowledge?

After the change is real, and after a person accepts the note. Task-specific memory can live on a branch or in a task folder until then.

What should never live only in agent memory?

Money, legal commitments, inventory, appointment times, and identity or payment details. Those belong in the system of record and still need a human boundary when the action is hard to reverse.

Start with one inspectable note

Pick one recurring process: lead follow-up, visit confirmation, or order exceptions. Write a one-page current-state note. Link the last three real conversations under it. Put the file where both a person and the agent can read it. Then run one week and see what was wrong or stale.

Pratap AI can map that first memory layout against your actual tools, and mark where Voice Calling Automation or WhatsApp Business Automation should write a card versus where a person still decides. Start with workflow automation or a 20-minute AI Opportunity Call via contact.

Credit: argument adapted from Devin Stein, How to Build Agent Memory: Where Does Knowledge Live?.