Open call: test your agent memory layer on an adversarial coding benchmark

🤗Datasets
1

I built AMB, an open and preregistered benchmark for memory layers in coding agents.

Most memory benchmarks test whether a system retrieves a relevant chunk. AMB tests whether that memory actually helps an agent complete real coding work, or causes a worse change.

Claude Code runs inside a sandboxed repository, and executable tests grade the resulting artifact. The benchmark is adversarially constructed with absent, stale, superseded, contradictory, adjacent, and irrelevant memories.

The current public feed contains 195 pre-authored session transcripts. A separate hard corpus record scales to 4,900 documents, including 196 real documents and 4,704 synthetic distractors, with roughly 143,000 chunks.

I am inviting small memory vendors to test their systems. There is no participation fee. Vendors can review the adapter contract, run with their own credentials, and publish their configuration and results. Negative results are welcome.

The current suite mainly measures the read path. It does not yet measure whether a memory layer learns from the agent’s own work across sessions, and no multi product ranking has been published.

Repository:

What would you want this benchmark to measure before trusting its results?

2

The execution-based evaluation sounds much more useful than measuring retrieval alone. I’d also include a no-memory baseline and a small leakage check, then report stale, contradictory, and irrelevant memory failures separately. That would make the results easier to interpret.

3

Thanks, I changed the benchmark in that direction. It now evaluates whether Claude completes the coding task and passes an executable checker, rather than scoring retrieval alone. I added a bare arm with no memory layer, plus a protocol arm with the shared instructions but no memory surface, so I can separate memory from generic prompting.

I also added leakage audits and sandbox isolation checks. The benchmark now reports absent, stale, contradictory, irrelevant, and present memory conditions separately.

4

I just shipped a causal memory layer with a 246-transcript benchmark called Silent Rotation. Memoryless agents converged on a planted decoy 21/25 times. Our system: 20/23 correct. The mechanism is retroactive salience backfill from a 2024 Nature paper, it reaches backward from failures to find causes that similarity search misses. Happy to test against AMB.

Repo: GitHub - samvallad33/vestige: Vestige enhances agents by deterministic root-cause retrieval that reaches backward through time to find the quiet change, decision, or service that caused today’s failure, not the lookalike. · GitHub

5

Once the read path is stable, I would add sequence-level scoring: memory should improve outcomes across several linked tasks without accumulating false beliefs. Measure task success, harmful-memory rate, useful-memory precision, and total memory/token overhead. A strong layer should not only rescue failures; it should know when not to write or retrieve. Freezing a held-out corpus before vendor runs would also reduce tuning leakage.

6

I think this is the right next layer. Single task success can show that memory sometimes rescues a failure, but it cannot show whether the system improves over a sequence, without accumulating stale or false beliefs.
I would evaluate chain success, harmful memory events, useful memory precision, selective writing and retrieval, and total token and latency overhead separately.
The existing longitudinal design is a good foundation, but the held out corpus should be frozen, hashed, and preregistered before vendor runs begin. That would make the result measure generalization and memory discipline, rather than adaptation to the evaluation set.