Agent memory benchmarks you can check yourself. This repository holds the harnesses behind the results on past.dev/benchmarks, and every answer, piece of evidence and judge verdict from the published runs. They use only past.dev's public API, the same calls you would make to add memory to your own product.
| Folder | Benchmark |
|---|---|
beam/ |
BEAM, long-conversation memory at 100K, 500K, 1M and 10M tokens |
dolphin/ |
DolphinBench, Mem0's agent-memory benchmark: 600 tests over three users' histories, graded on the agent's app calls |
verify.py recomputes every BEAM score and the summary from the saved judge verdicts. It calls no
memory API and no model, so it needs no key.
git clone https://github.com/pastdotdev/benchmarks.git
cd benchmarks/beam
pip install -r requirements.txt
python verify.py results/100k # or results/500k, results/1m, results/10m
python verify.py --prompts # our prompts match ExaBase's published fileThe first command downloads the public BEAM dataset once, to confirm that every question has one record.
To run a benchmark again, you need a past.dev account (sign up), its organization key, and an OpenRouter key. Each conversation gets its own project, and a new account allows fewer projects than a full split needs; the folder's README says how to run within that.
The BEAM harness reaches past.dev only through beam/past_api.py. To run it on
another memory system, replace that class with one that sets up a store for each conversation,
ingests the turns and answers a recall. The dataset, the answer and judge prompts and the scoring
stay as they are, so the scores compare. If you publish a port and its results, open an issue and we
will link it here.
The saved baseline covers all 100 conversations and 2,000 questions. The combined score, weighted by question count, is 90.02%.
| Split | Conversations | Questions | Score | Files |
|---|---|---|---|---|
| 100K | 20 | 400 | 92.08% | Summary · Results |
| 500K | 35 | 700 | 89.63% | Summary · Results |
| 1M | 35 | 700 | 90.65% | Summary · Results |
| 10M | 10 | 200 | 85.03% | Summary · Results |
See the BEAM README for the category breakdown and instructions for verifying the saved results or running the benchmark.
Hermes Agent with GPT 5.6 Luna and past.dev as its memory passed 490 of 600 DolphinBench tests. In DolphinBench's official runs of the same agent and model, the other memories pass 355 to 424.
| Memory | Passed | Morgan | Alex | Riley | Cost | Median latency | Files |
|---|---|---|---|---|---|---|---|
| past.dev | 490 / 600 | 173 | 148 | 169 | $196.72 | 51.6 s | Run |
| Mem0 | 424 / 600 | 156 | 132 | 136 | $96.21 | 37.7 s | Official |
| Hindsight | 417 / 600 | 144 | 127 | 146 | $84.65 | 55.3 s | Official |
| Honcho | 411 / 600 | 144 | 120 | 147 | $142.85 | 44.7 s | Official |
| Built-in | 394 / 600 | 131 | 111 | 152 | $61.48 | 44.4 s | Official |
| Supermemory | 355 / 600 | 138 | 98 | 119 | $345.64 | 52.7 s | Official |
past.dev's own minimal agent, a separate configuration with the same model, passed 520 of 600 for $15.77. See the DolphinBench README for the split by task kind, every way the run differs from the official ones, and how to check or rerun it.
The code is MIT licensed (LICENSE). Benchmarks, datasets and third-party prompts keep
their own terms, credited in each folder's NOTICE.md.