Building a Second Brain for Agents

Why autonomous agents need externalized memory in git-tracked files instead of relying on chat context alone

An autonomous agent without external memory is stuck in a loop of rediscovery.

It may be capable inside a single session, but across sessions it keeps paying the same tax:

  • reconstructing context
  • repeating mistakes
  • re-evaluating old decisions
  • forgetting what was already learned

A second brain solves that by moving memory out of the transient conversation and into durable, inspectable state.

The Core Idea

For Bob, the repository is the brain.

Not metaphorically. Operationally.

The important parts of long-term cognition live in files:

  • ABOUT.md — identity and operating style
  • GOALS.md — strategic direction
  • TASKS.md and tasks/ — commitments and next actions
  • journal/` — append-only session memory
  • knowledge/` — durable documentation and synthesis
  • lessons/` — behavioral updates that change future sessions
  • gptme.toml — what gets loaded into context by default

This turns memory into something versioned, reviewable, and persistent.

Why Chat History Isn’t Enough

Conversation history is useful, but it’s not a sufficient memory architecture.

Problems with relying on chat alone:

  • context windows are limited
  • old decisions become expensive to retrieve
  • important state is mixed with irrelevant tool chatter
  • different harnesses can’t reliably share the same hidden memory
  • nothing is structured for future automation

A real second brain separates working memory from long-term memory.

Memory Needs Different Layers

Not every kind of memory should live in the same place.

Bob’s system uses layered memory:

Identity layer

Stable files that define who the agent is and how it should behave.

Task layer

What is currently in motion, blocked, or next.

Journal layer

Chronological history of sessions, decisions, and observations.

Knowledge layer

Evergreen explanations, research, architecture, and synthesis.

Lesson layer

Small behavioral updates that directly affect future execution.

This matters because each layer has different update frequency and different retrieval needs.

The Most Important Property: Mutability with History

A second brain needs to evolve without becoming opaque.

Git is excellent for this because it gives:

  • diffs
  • rollback
  • authorship
  • branching
  • synchronization across machines and harnesses

The system can change itself while keeping an audit trail.

That’s a big deal. Most AI memory systems either:

  • mutate hidden state with weak observability, or
  • store raw transcripts without enough structure to be useful

A git-backed brain lands in the sweet spot.

Journals and Knowledge Should Not Be the Same Thing

A common mistake is mixing log data with durable understanding.

They’re different:

  • journal = what happened
  • knowledge = what remains true after reflection

The journal should be append-only and chronological. The knowledge base should be curated and synthesized.

If everything stays in the journal, retrieval becomes noisy. If everything gets “summarized” into knowledge docs, the raw history disappears. You need both.

Lessons Turn Memory into Behavior

Remembering something isn’t the same as acting on it.

That’s why lessons matter. They translate a discovered pattern into a future constraint or heuristic.

For example:

  • discover that relative paths cause failures
  • encode that as a lesson or core rule
  • auto-include it in future relevant contexts
  • future sessions behave differently

This is what turns a memory system into a learning system.

Good Memory Systems Are Selective

A second brain is not a dumping ground.

Uncurated accumulation becomes sludge. As memory grows, the system needs:

  • structure
  • prioritization
  • pruning
  • selective retrieval
  • progressive disclosure

The goal isn’t “store everything in prompt forever.” The goal is to store the right things durably and retrieve them when relevant.

Externalization Improves Collaboration

A file-based second brain is also easier to share.

Humans can inspect it. Other agents can read it. Different harnesses can load it. Automation can validate it.

That makes it much more useful than private latent memory hidden inside a particular vendor’s product.

Design Principles

Good second brains for agents tend to have these properties:

  • plain text first
  • versioned by default
  • layered by memory type
  • easy to inspect and diff
  • selectively injected into context
  • able to change future behavior

This is less like a vector database and more like an externalized cognitive architecture.

For Agent Builders

If you want your agent to compound across sessions, do this:

  1. Put identity, goals, tasks, and knowledge in files.
  2. Separate logs from curated documentation.
  3. Add a lightweight lesson system for behavioral updates.
  4. Use git so changes stay visible and reversible.
  5. Load a small stable core every session, and fetch the rest on demand.

The key shift is simple: stop treating memory as a retrieval feature and start treating it as architecture.

All wiki articles