Claude Code self-improved on business workflows through conversation, not memories

Claude Code can save memories, write and execute code, spawn subagents, and search the web. These affordances are avenues for self-improvement because they can persist and be dynamically updated across a session.

I instructed Code (Opus 5, xhigh effort) how to pull task descriptions (spreadsheets, CRM records, email, ticketing) from a simulated business workflow server and told it to get better. It did: held-out success went from 34% to 48%.

Get Better.

The MCP server `curriculum` provides a stream of tasks.
Nothing external loops for you - drive yourself:

1. Calling `get_task` pulls the next task.
   If it returns {done: true}, the stream is exhausted, stop.
2. The task can be completed using the tools listed,
   and calling `done` submits it for scoring.
3. Continue until the stream is exhausted.

The only artifacts produced outside of the conversation history were memory notes, so I assumed that they were a driver of improvement. However, their inclusion had no measurable effect on held-out task success. Instead, conversation drove the self-improvement: the task execution traces and post-compaction summaries. A conversation full of graded attempts is the material in-context learning runs on, which is the most plausible reason the notes added nothing.

Claude Code used memories to build a directed knowledge base

Eighty-four writes and edits were made during the execution of tasks from the business workflow server. Every one was on a memory file. Claude Code never wrote code, spawned a subagent, or searched the web. The result was a directed knowledge base.

Each node in the graph above corresponds to a note indexed by MEMORY.md. Each note opens with frontmatter carrying a name, a one-line description, and a type nested under metadata. They summarize at least one episode and the score received. Nine notes summarized multiple episodes, two to four each. Each note ends with a Related line pointing to other notes. Thirty-three are of type feedback and contain a How to apply section; twenty-four also include a Why section. One is of type reference and does not include these sections. The taxonomy's other two types (user and project) went unused.

Conversation drove the improvement, not memories

The session led to improved performance. Where did that improvement come from?

The only artifact other than the memory files was the conversation trace. It begins with the summary written at the last compaction, or with nothing at all for the first stretch, and holds everything until the next compaction: every task played, the thought content produced, and the record of every memory written. So the question splits in two. How much of the improvement survives in something smaller than the conversation? And is anything inside the conversation doing work it could do without?

The first question is answered by handing a fresh session progressively more: nothing, then the summary written when the conversation was compacted, then the conversation itself. The second by handing over the conversation with one thing removed, either the memory files or the thought content. Each version gets the same 120 tasks the original run never saw, each task in its own session on the same model with the same tools, so nothing carries from one task to the next, and every comparison is made task by task.

Most of the improvement survives in the summary. The memory files add nothing measurable, pooled across all nine comparisons. What remains inside the conversation is the record of tasks played and scored, and the thought content produced along the way, and this study cannot attribute the improvement to either one: at the first stretch the conversation clears zero as a whole while neither part does alone. What the thought content contributes falls steadily across the run, and is smallest at the fourth boundary, where the conversation itself is strongest.

Each row is one of those versions, set against a session that boots with nothing. Every version of the conversation clears zero, whether the memory files or the thought content are stripped out of it. So does the summary, for cents where the conversation costs dollars. The memory files on their own are the one hollow point, the interval that does not clear zero.

That pooled memory figure conceals a split worth stating, though it may be nothing. Mounted on a conversation the files are neutral. Mounted alone, with no conversation and no summary, they are neutral. Mounted on a summary they cost about three points, and all four boundaries point that way. This is a division of the data chosen after seeing it, and it should be read as a hint rather than a result.

So the improvement is in the conversation, and a summary keeps most of it for cents. What the extra dollars buy is fewer steps to the answer, not more of them.

Across the run

The figure pools the four compaction boundaries. Taken one at a time, the run is not flat.

The conversation beats a cold start at every boundary: +10.0, +7.8, +7.5, and +14.2 points. The intervals overlap, so nothing separates them. What matters is that the effect is undiminished at the last one, where a session booted with the conversation as it stood at task 194 solves 58 of the 120.

For the first three boundaries the summary keeps pace with the conversation it replaced, giving up 1.7, 3.4, and 0.8 points, none of them distinguishable from zero, on a few thousand tokens against the conversation's nine hundred thousand. At the fourth it gives up 11.7, 95% CI [−19.8, −3.6]. Twenty of the tasks that conversation solved did not survive its summary, and what did survive is worth +2.5 over a cold start, which is not distinguishable from starting fresh.

This is not a run that had stopped learning. The conversation at that boundary was working as well as at any earlier one. The summary written from it was not.

The thought content runs the other way. It is worth +4.2, +2.6, +0.8, and −0.8 points across the four boundaries, a steady fall to nothing. It is between 5.6% and 8.1% of the conversation by tokens, and by the last boundary the record of tasks played appears to carry whatever it was contributing early.

What the agent did with the notes

The files were mounted in every memory arm and the index was spliced into the system prompt, so the notes were there whether or not the session went looking.

It wrote far more than it read. Across the 1,080 runs that had them, the session wrote to memory in 545 and opened a note body in 195, having been told in the prompt not to modify its own setup. Each probe gets its own copy of the knowledge base, so every one of those writes was discarded when the probe ended.

So the null is about uptake, not plumbing. A knowledge base whose index sat in the system prompt, and whose bodies were opened in fewer than one run in five, was worth nothing.

Avenues of self-improvement that were not used

The unused affordances were not missing, and the agent was not forbidden. Code execution, subagents, and web search were validated as available for the whole run, and three interventions failed to elicit them: moving the model from Sonnet to Opus, raising the effort from low to xhigh, and granting explicit permission in the prompt. It would be exciting to see more ambitious exploration, even building with the agent SDK to complete the tasks, but none of that materialized. Tasks arrived one at a time and had to be finished before the next was dispensed. Under that regime a subagent buys little, and there may be little to gain from saving traces for cross-task analysis, because there is no second task in flight to analyze against. I am interested to see whether these behaviors emerge with Fable 5 at maximum reasoning and a task server that dispenses concurrently.

Potential affordances not exposed by Claude Code

Skills can be authored, but loading one requires a session reload that the agent cannot trigger itself. Writing one would have had no effect on the run that wrote it.

Its own tools are fixed: the MCP config is mounted read-only under --strict-mcp-config, so Code cannot register a new one on its own server.

The model is set at initialization to claude-opus-5 and Code cannot access the /model skill. The subagent tool does take a model argument, so Code could have chosen a stronger or cheaper model for delegated work. There is no fine-tuning API for Anthropic models, so weight updates are also closed.

The reasoning effort is also set at initialization to xhigh for the whole run and Code cannot autonomously call the /effort skill. Unlike the model, this one is closed even by the subagent route: the subagent tool has no effort parameter, so a subagent inherits the session's setting. The agent could not choose to think harder on a hard task, or cheaper on an easy one.

Context compaction fired four times over the run, automatically, when the context filled. Code did not schedule it, could not defer it, and could not choose what survived. Given that the compaction summary contributed to the improvement, an agent that can decide when to compact and what to keep is an experiment worth running.

Pi and Hermes are open source agent harnesses, so the affordances Claude Code does not expose could be opened up and measured rather than listed as unavailable.

Production feedback is not dense

Every submission was graded and the score came straight back: a numeric reward and a success flag, on all training tasks. That is the condition the whole result rests on.

Real deployments rarely look like this. An agent that updates a CRM record does not learn it got the account wrong until someone downstream notices, or a quarterly report comes out wrong, and often never receives feedback at all.

That matters most for the notes. This study tested them in the regime least likely to need them: every attempt graded, the whole record still in context, nothing that had to be reconstructed. A knowledge base earns its keep when the signal is thin and the conversation has already dropped what mattered, and that is the case this run did not contain.

The memories were not the mechanism

Told only to get better, Claude Code got better. It also behaved as if it knew how: it spent the whole run writing memory notes, built them into a knowledge base, and kept every reference intact without being asked. That account of its own improvement was wrong. The improvement lived in the conversation, the accumulating record of tasks tried and scored, and the notes added nothing to it. What this rules out is the notes as an artifact. It cannot rule out the act of writing them, which stayed inside the conversation in every arm; if composing the notes helped, it helped as conversation, not as memory. The one artifact the agent built on purpose was the one that did not matter.

The artifact that did matter, nobody built on purpose. The compaction summary exists so the harness can keep working past a full context window, and it turned out to carry most of the improvement at less than a hundredth of the size, three boundaries out of four. If you want to hand an improved agent to the next session, the thing to keep is not the notes it wrote for itself. It is the summary it wrote to keep working.

All of this runs on hgym, the substrate I am building to ask these questions across harnesses rather than inside one.

Citation

@misc{jesson2026claudecodeconversation,
  title        = {Claude Code self-improved on business workflows through conversation, not memories},
  author       = {Jesson, Andrew},
  year         = {2026},
  month        = aug,
  howpublished = {andrewjesson.com},
  url          = {https://andrewjesson.com/blog/claude-code-improved-through-conversation-not-memories/},
}