>Proactive Recall for Coding Agents: Why Memory Belongs in the Harness
Agent Harness Engineering, Part 1 — a Droidworks project
Have you tried external memory systems for your coding agents? I tried a lot of them. And I felt they didn’t deliver on their promise. A session gets summarized. Facts go into an index. A CLI search returns a relevant memory.
It looked right on paper, but I never saw real value in my work with agents. I kept seeing the same pattern. The agent had worked in the repository before. It had already learned a useful lesson. When a new task arrived, it opened the same files. It followed the same wrong path.
The agent had access to the memory tool. It was primed to use it, but it never chose to. This is the problem I wanted to tackle. Storage is one problem. Recall policy is another.
Tool availability is not tool usage
I tried several memory systems while working on this: mem0, Claude Mem, Cognee, and smaller projects exploring similar ideas. Many of them have strong storage and retrieval pipelines. They extract facts from sessions, save them, and provide an interface for retrieving them quickly. There are multiple approaches: BM25, graph memory systems, and vector memory systems. Some memory tools unify them all.
But I still experienced this weak point. The agent never decided to search memory on its own.
In my experience, agents start with whatever sits in front of them. Reading another file is an obvious next action. So the agent reads files. It never tries to reach for the experience captured during previous sessions.
The same agent. A different recall policy.
The write path and the read path
I decided to split agent memory into two paths. The write path is solved. Today agents are effective at recording useful experience:
- what changed
- what failed
- what fixed it
- which local conventions mattered
- what the next agent should know
The write path is easier to make reliable. A simple instruction in AGENTS.md changed my agents’ behavior:
Before the session ends, record what was learned.
The read path retrieves that experience so it can affect the next task. The read path is harder. The agent cannot see the context it is missing. Initially, I tried the same trick, priming it with a direct rule: “Search memory before using grep.” In my sessions, that changed very little.
So the next step was to improve the read path. The agent harness already knows when a session starts, when a prompt arrives, and when planning begins. I thought those events could trigger recall directly.
Proactive recall
Proactive recall means the workflow runs the memory search without the agent having to ask for it. And timing matters. A useful memory should arrive before the agent stumbles onto a path that failed before. Proactive recall should run as early as possible, when earlier memories have the highest chance of changing the path it takes.
This is the distinction I use between a memory system and a memory harness.
- A memory system extracts, stores, indexes, and retrieves information.
- A memory harness decides when recall runs, what query it uses, and how much context it injects.
The recall step moves into the harness, before the agent commits to a plan.
What Intellex does
Intellex is my take on this memory harness layer. Yes, I started with a full database architecture and my own opinionated storage layer. But it was overkill. So I tried to make it as small as possible. The current version is:
- a Bun CLI exposed as
intellexandix - project state stored in
.intellex/(how many dotted folders do you have in the repo at this point? :) ) - Claude Code integration through lifecycle hooks
- mem0-backed extraction and search
- session backfill for turning previous work into memory
Side note. Session backfill is the only thing that survived from v0.1, and it isn’t directly needed for proactive recall. I haven’t seen any agentic memory system take advantage of sessions you have already completed with your agent. There are tons of lessons there that you can use right away.
The setup is simple:
ix init
ix setup claude
ix backfill
After setup, the hooks own the basic workflow.
At the start of an agent session, Intellex primes the agent and processes recent sessions in the background without blocking the main flow. When the user submits a prompt, Intellex searches memory with that prompt. It injects the top results before the agent starts solving your problem.
The flow looks like this:
prompt arrives
↓
search previous experience
↓
inject relevant memory
↓
agent plans and works
↓
on new session: clean all finished sessions
↓
store new lessons
I love the simplicity of this loop. Nothing here is magic. I expected agents to do it themselves, but moving everything into the harness made it automatic. It just works.
Right now, I have adapters for Claude Code and mem0. But the design is intentionally abstract. The same design can support other agent runtimes and memory backends.
How I tested it
I didn’t want Intellex to be another extension someone created overnight and then claimed it just worked. So I dug into memory benchmarking. Most memory benchmarks test whether a system can retrieve a fact from a long conversation.
But I think a coding benchmark needs to answer a different question:
Does experience from earlier tasks improve later tasks?
I settled on SWE-bench-CL. It has the right shape for my problem. It organizes real issues chronologically within the same repositories. Earlier tasks can produce experience that may help with later tasks.
I ran one chronological campaign for multiple memory settings to understand how my system actually affects real-life scenarios. Claude Code used Haiku as the coding agent. Intellex used mem0 with gpt-5.4-mini for memory extraction.
The subset I used from SWE-bench-CL contained 141 tasks across the Django, Sympy, pytest, and xarray lanes. Each repository started with empty memory. After every task, Intellex stored a cleaned record of the work. Later tasks could receive retrieved memories before planning and execution, depending on the benchmark setting.
The top-level results were:
| Setting | Passed | Rate |
| Baseline (no ix) | 90 / 141 | 63.8% |
| 3 memory blocks | 109 / 141 | 77.3% |
| 5 memory blocks | 106 / 141 | 75.2% |
| 10 memory blocks | 111 / 141 | 78.7% |
Proactive memory improved the overall result. Hooray! It works!
Interestingly, the result was not monotonic. Five memory blocks performed worse than three. Ten performed best overall. Ideally, I would run the same test five times and average the results. But I decided to invest that time in digging deeper into these directionally promising runs.
Different repositories reacted differently. Sympy showed the biggest win. It moved from 24/50 to 40/50 in the best Intellex runs. Django improved slightly. Pytest improved at higher memory depth. Xarray stayed close to baseline even with active recall enabled. This proved that a Haiku-class model is not the best at coding hard tasks (like you didn’t know that already). But Haiku was cheap and fast for testing and running these benchmarks.
Exploring the benchmark run logs, I saw that throughout all runs, the agent never initiated an Intellex lookup itself. The agent was primed with the tool’s availability but never used it. Deterministic memory-harness hooks were the only path for injecting memories into the agent’s context, producing the measured improvement.
Limitations
This was one exploratory campaign per setting rather than a statistically powered experiment. Agent runs are stochastic, so repeated campaigns and confidence intervals are required before treating the exact deltas as stable. The benchmark also measures issue resolution in selected open-source repositories; it does not measure long-running proprietary projects, memory quality over months, or the operational cost of incorrect recall. These results support further testing of proactive recall, while the per-repository variation warns against assuming a universal gain.
The dogfooding moment
A small and inspiring example happened while I was working on Intellex itself.
I started a Claude Code session to test my recent changes. We were debugging Intellex hooks, and I saw that, for some reason, the agent had updated a blog post record we had worked on a few days earlier. The prompt-submit hook found an existing memory about the post and added it to the context. Claude used it naturally during the session.
Intellex surfaced an earlier project commitment without being asked.
That was the first moment Intellex felt like a coherent memory-harness workflow rather than a database search tool.
The relevant project state arrived on its own. It was magical.
Where proactive recall can fail
Automatic recall has real costs. It should keep improving to keep up with models:
- irrelevant memories can suggest the wrong analogy
- stale memories can preserve old decisions
- injected context consumes tokens
Recalling the wrong memory with confidence can be especially bad. The agent may reuse an old pattern as the current truth.
A memory harness needs limits and knobs. It should control memory depth, favor recent and relevant evidence, track context cost, and make recalled blocks visible for inspection.
Conclusion
A coding agent can have access to a capable memory system and still behave as if every task is its first. The missing layer is recall policy: when retrieval runs, how results are selected, and where they enter the model's context.
Moving recall into deterministic lifecycle hooks made that policy explicit and testable. The benchmark results are preliminary, yet they show why this architectural distinction deserves attention. Memory should enter the workflow before the agent has to remember that memory exists.
Agent Harness Engineering
This post is Part 1 of Agent Harness Engineering, a series inside Droidworks. Droidworks is my informal lab for the systems around AI agents: memory, context, orchestration, tools, and evaluation. I want to explore the frontier of AI engineering and push it further.
The next posts will explore:
- observational memory for long-running agent sessions
- Mixture of Agents inside the harness loop
- practical benchmark design for agent harnesses