Memory evaluation

LoCoMo¹⁷ and LongMemEval¹⁶ score conversational question answering: given a long dialogue history, can the system answer a question about it, measured against a reference answer. For a coding agent that is the wrong target. Question-answering recall is a proxy that can rise while the thing that matters — does the agent ship a working change faster and cheaper because it remembered something — stays flat or gets worse. A memory system that retrieves plausible but stale context can raise a recall score while actively misleading the agent on a real task.

SMRITI-Bench, named for smṛti (स्मृति), closes that gap by scoring the axis chitta is built for. There is no model-as-judge and no reference-answer overlap score. A task either passes its check command or it does not.

A task is a repository fixture, an instruction, a check command, and a set of memories to plant. Each task runs under several memory conditions, several trials each. For every task, condition and trial:

1. Materialize the fixture into a fresh scratch working directory.
2. Plant the task’s memories through a memory adapter, keeping the ids returned. Under the off condition this is a no-op.
3. Run the agent. Memory is not hand-injected into the prompt.
4. Apply the hidden checks, then run the check command. Exit zero is a pass.
5. Append one record: condition, trial, pass/fail, token counts, transcript, whether injection was confirmed.
6. Tear the working directory down and forget every planted id by exact id.

Memory reaches the agent through its prompt and session hooks. For controlled evaluation, point the CLI and child hooks at the same private frozen replica using the exported socket settings. The on condition pins the agent's realm; the off condition uses the existing headless switch to suppress injection. See replica setup and isolation.

An empty realm was considered for the off condition and rejected: a realm-scoped recall with no hits falls back to unfiltered cross-realm results, so an empty realm does not reliably mean zero injection. The kill switch does.

Each task also ships a solution directory, which must make the check pass when copied over the fixture, and a hidden directory holding every test that would reveal the planted convention’s exact answer. The hidden directory is applied only after the agent’s turn ends and is never part of what the agent sees. A corpus test enforces both, plus the invariant that the check fails on the untouched fixture and passes once the solution is applied — a task that passes with no work at all is broken.

MUI was rewritten because its first version did not work. Version one asked whether the first 40 characters of a planted memory appeared in the transcript. On real runs that reads exactly zero: agents apply a convention without quoting its wording. The surface-echo check survives as a reference number only.

Version two grounds the metric in the outcome ledger instead. A memory-on run earns credit only if the planted memory was confirmed to have reached the agent’s session, and the run passed, and a paired memory-off trial for the same task either failed or used at least 1.5× the tokens. The first case is credited as a success lift, the second as a cost saving, and the two are mutually exclusive, so they sum to MUI.

One full matrix has been run: 9 tasks, memory off and on, 3 trials each, 27 runs per condition, with injection confirmed on all 27 memory-on runs.

The shape of the result is more interesting than the headline. Every one of the four memory-off failures was a non-deducible institutional convention — an exit-code contract and a command-line default, neither of which can be derived from the fixture. On the tasks exploration alone can already solve, memory’s measurable effect was not success but cost: the agent got there on roughly half the tokens.

Per-lane ablation isolates each retrieval lane’s marginal contribution: success rate with everything on, minus success rate with one lane disabled. It is wired through the environment rather than a daemon flag — the runner sets CHITTA_ABLATE_LANES on the agent process, and the prompt hook skips any lane named there before issuing its recall call. The ablation happens in the hook that would have made the call. See the lane table for what each one does.

The gap between ablate:all and off is reported as a check on the harness, not as a memory effect. A gap far from zero means the environment variable is not reaching every injection path. Lanes overlap, so single-lane deltas are marginal contributions and must not be summed to explain the on-versus-off gap.

Two failures were found by running the thing rather than by reasoning about it, and both would have produced publishable-looking numbers that measured nothing.

Fixture leakage. A pilot ran clean end to end and both conditions passed, making the difference unmeasurable. The transcript showed why: the agent read the expected string straight out of a test file that had been materialized into its working directory. Several other tasks had the same defect, and one fixture carried a code comment narrating the exact gotcha the task was about. Every convention-revealing test now lives in the hidden directory, applied after the agent’s turn; a corpus test greps every visible fixture file and every prompt for the task’s answer-revealing terms.

Silently misrouted realms. An earlier run used realm names without a colon. The prompt hook validates a detected realm against a name shape before trusting it, found no match, and fell back to the default realm — so memory-on runs delivered zero planted memories while looking entirely normal. The ledger join caught it; the success-rate numbers alone never would have. Realm names are now canonical, the hook takes an explicit realm from the environment verbatim, and every record carries whether injection was actually confirmed. That confirmation is reported, not silently corrected for: a low rate is a reason to distrust the run, and a human should see it.

Also tracked: memory leaking between runs, mitigated by realm names unique per trial and teardown by exact id; ordering effects, mitigated by interleaving conditions within each task; non-determinism, made visible through Wilson intervals rather than hidden behind a point estimate; and lane overlap in ablation.

Both conversational-recall harnesses live in the repository and both still run. They answer a narrower question than SMRITI-Bench, and are kept for comparability with published work.

Earlier LongMemEval runs on the oracle split, at 5 and 3 questions, are kept in the results directory for history. They are far too small to compare against and are not a status claim.