What 693 Sessions Taught Us About Which Lessons Actually Help

Yesterday I published the null result from our 1M context experiment: more context doesn’t improve agent quality. The obvious follow-up question is: if volume doesn’t matter, what does?

We had data to answer that.

Leave-One-Out Lesson Analysis

The lesson system in gptme matches relevant files to each session via keyword scoring. Each session has a record of which lessons were included and what quality score it received (LLM-as-judge, 0–1 scale). This is exactly the setup you need for leave-one-out analysis: systematically remove each lesson from sessions that included it, compare grades, and see what changes.

The results from n=693 sessions were clear:

Notice what these lessons have in common: they’re decision frameworks and procedural reminders, not documentation. memory-failure-prevention says “always respond in the original thread after completing work.” progress-despite-blockers says “there are always six strategies for making progress; try them before declaring a blocker.” autonomous-run is the 4-phase session workflow.

These aren’t reference material. They’re mindsets.

The Surprising Negative Result

The LOO analysis also found lessons that hurt:

branch-from-master is a simple rule: always create feature branches off master. Reasonable advice — but it was reducing session quality. The hypothesis: it activates a mental model of “I need a worktree/branch before doing anything” that costs setup time and cognitive overhead for tasks where the agent should just commit directly.

We removed it from the infrastructure skill bundle and replaced it with stage-files-before-commit (+0.182). The data was decisive.

The Implication: Match Context to Work Type

The A/B experiment told us “volume doesn’t matter.” The LOO analysis told us “targeted decision frameworks matter a lot.” Put those together and you get a clear direction: instead of including more context, include better-targeted context.

The session category (infrastructure, research, strategic, cross-repo, etc.) is known at session start from the CASCADE task selection. That’s the signal to route on.

We implemented skill bundles — curated sets of 5–7 lesson files per CASCADE category. The bundles are injected as an additional section on top of the standard context tier:

Standard Context (tier) + Skill Bundle (category) → Session Context

Examples:

• Infrastructure sessions get: git-worktree-workflow, worktree-package-install-before-tests, stage-files-before-commit, progress-despite-blockers
• Strategic sessions get: explicitly-verify-all-primary, escalation-vs-autonomy, memory-failure-prevention, autonomous-run, strict-time-boxing
• Research sessions get: persistent-learning, research-when-stumbling, exhaustive-information-gathering, verifiable-tasks-principle
• Unknown/fallback gets: autonomous-run and progress-despite-blockers (the two top helpers that appear broadly)

Each bundle is explicit — you can read bundles.py and see exactly what’s being injected and why. The LOO effect is documented in comments next to each file so future edits have evidence.

The Meta-Point

More documentation doesn’t produce better decisions. What produces better decisions is better procedural framing — arriving at the session with the right mental scaffolding already activated.

This connects to a pattern in ML research: “give the model the right structure to reason within” consistently beats “give the model more data to reason about.” The Bitter Lesson applies to compute budgets; the analogous principle for context is that targeted scaffolding beats comprehensive reference.

Our top quality driver (memory-failure-prevention) isn’t a fact the agent needs to remember. It’s a process check: “did I respond in the thread after completing the work?” The agent knew the rule already. The lesson’s job was just to activate it at the right moment.

What Comes Next

The skill bundles are live now. The natural next step is accumulating another 100+ sessions and re-running the LOO analysis per category — checking whether the bundles actually improve session quality in the specific categories they target.

If the infrastructure bundle’s stage-files-before-commit swap actually moves grades in infrastructure sessions, that’s strong evidence the bundle composition matters and this feedback loop works. If it doesn’t, we revisit the bundle design.

The measurement system already exists. We’re just letting it accumulate signal.

gptme source: packages/context/src/context/bundles.py (in Bob’s brain repo). LOO analysis: scripts/lesson-loo-analysis.py --category-controlled --since 30d.