We Built a Benchmark That Didn't Exist Yet. Here's What It Found
We tested whether one AI agent can use knowledge only a teammate learned, and compared Litica against mem0 and plain RAG. Full results, including where we lost.
The question nobody was measuring
Picture a small development team. A researcher, Aria, a strategist, Blake, and an engineer, Casey, working together on a project. Aria hears something important in week one. Three weeks later, Blake needs that exact fact to make a call. The catch: Blake was never in the room when it was said.
Even for humans, this is a bottleneck. Maybe someone repeats it in a later meeting, or it's written down in a shared doc, or it comes up over coffee. Just as often, it doesn't reach Blake at all until the wrong moment. Teams lose information this way constantly, and it's a known cost of working with other people.
Now replace those three people with three AI agents. The problem gets worse, not better. AI agents don't have coffee breaks or hallway conversations. If Agent A learns something and Agent B never sees it directly, Agent B has no way to know it happened, unless the memory system they share is built to carry that knowledge across.
That's the question this benchmark tests: can one agent use knowledge that only a teammate learned, when the two never communicated directly?
The answer is harder to get than one might think. Most "AI memory" benchmarks test something narrower. Does the assistant remember what you personally told it? That's one agent, one conversation, recalling its own history. Nobody was testing the team version where knowledge actually crosses the line between agents. So we built a benchmark to test that directly, ran it against two other approaches people use for AI memory today, and are publishing the full results, including the parts where we didn't win.
The setup, in plain terms
We recreated the human team as an agentic one, with Aria (research), Blake (strategy), and Casey (engineering) working on one project. Over the course of the project, they collectively pick up 30 individual facts: things they learned, decided, or were told.
Then we asked five questions a manager might ask. Things like "what are our current priorities" or "what's blocking us right now." Most of the five questions touch knowledge from more than one agent. Two of them are built so the only way to answer well is to combine what multiple agents know, and those two are where we focus below. No single agent's memory alone contains the full answer.
We ran this setup through three different systems:
- Litica, our system
- mem0, a widely used open-source AI memory tool
- RAG, a standard keyword/similarity search, the default approach most people reach for first
Each system got graded the same way. For each question, did the top 3 things it retrieved actually contain the right answer?
We decided not to include LLM Wiki in this analysis because it sits on a different axis. LLM Wiki isn't comparable to a retrieval system, since it requires an agent to do the heavy lifting. It is something we will eventually evaluate against.
Why not just give one agent everything?
A fair question. Why not hand one agent all 30 facts, or set up a supervisor agent that every teammate reports to?
Because that doesn't remove the memory problem. It hands it to an agent with the exact same memory problem. The supervisor still has to store everything and find the right facts weeks later, and now it's a single point of failure where 30 facts become 30,000. Worse, agents can't know in week one which fact will matter in week four, so they either drop things at the source or drown the supervisor in everything, paying tokens on every handoff.
That's why Litica is a tool call, not an agent. Every agent gets two abilities, remember and recall, with no middleman to route through. Aria stores a fact the moment she learns it, and Blake pulls it out three weeks later without either knowing the other was involved. You connect your agents once and you're done, whether the team is three agents or forty. The test that matters isn't whether a model can answer when handed everything. It's whether the right facts come back out of everything the team has ever learned, and that's what this benchmark measures.
What makes these numbers real?
Here's exactly what we did to keep our benchmarks fair and honest:
- We wrote the answer key before the evaluation. The correct answers for each question were written down by a human before looking at what any system retrieved.
- Every AI-driven system ran on the same models. This included Gemini for LLM inference and all-MiniLM-L6-v2 for local embeddings.
- Litica ran on our production instance. This tests what users use today rather than a dev setup.
- We ran the nondeterministic parts more than once to compute averages. RAG is deterministic, so a single run is sufficient.
The primary metric is P@3, precision at 3, which means: of the top 3 results returned, what fraction were actually relevant?
The secondary metric is MRR, mean reciprocal rank, which measures how close to the top the right answer landed. It matters because it shows near-misses. Even when the right answer isn't first, a high MRR means it's close, and ranking improvements can convert those near-misses into wins.
The results
| System | Runs | P@3 (mean ± sd) | MRR (mean ± sd) |
|---|---|---|---|
| Litica | 3 | 0.36 ± 0.03 | 0.62 ± 0.03 |
| RAG | 1* | 0.27 | 0.60 |
| mem0 | 3 | 0.17 ± 0.00 | 0.23 ± 0.00 |
*Single run. RAG is deterministic, so there is no variance to report.
In plain terms, when asked to find the right answer among its top 3 guesses, Litica got there 36% of the time. That's a third better than plain keyword search at 27%, and more than double a leading memory product's 17%.
Two honest notes on these numbers. First, absolute scores are low for every system. That's by design. These questions are hard, and several require combining facts no single agent holds. Second, overall ranking quality (MRR) is close to plain search. The gap shows up specifically on the cross-agent questions, which is exactly what this benchmark exists to measure.
Where the win actually comes from
The interesting part isn't the overall number. It's which questions Litica wins on. The two questions built purely around cross-agent knowledge show why.
| Question | Plain RAG | mem0 | Litica |
|---|---|---|---|
| Sprint commitments, spanning strategy and engineering | 33% | 0% | 67% |
| Failed attempts, spanning all three agents | 0% | 33% | 33% |
This is the whole point of the test. When memory is organized around what information means instead of just matching similar words, one agent's knowledge becomes something a teammate can actually build on. Plain keyword search can't do that at all. It scored zero on the hardest question.
Where it doesn't win
To be transparent, we didn't win at everything. Here's where we came up short:
- On one question, plain search actually did better: 33% versus Litica's 11%. This touches a known weak spot in how Litica handles rare, sparsely mentioned facts.
- One cross-agent question went to mem0. Litica and plain search both scored zero, and mem0 scored 50%. The answer key expected two specific commitments, and our retrieval ranked broader priority statements above them. Ranking specific commitments higher when the question asks for focus is a known gap, and it's on the roadmap.
- This is a small test. Five questions and a few runs each is enough to show the gap between Litica and plain search is real and not just noise. But there is much to work on, and results for longer, less-trivial runs will be benchmarked as well.
What this benchmark shows, and what it doesn't
This round tested whether knowledge can move between AI agents working on the same team, not just get recalled by the agent that stored it in the first place. It doesn't tell you how these systems hold up over months of real use, how they perform at large scale, or what happens when new information contradicts something learned earlier. That last one is a related problem we're working on separately. We kept the scope narrow on purpose, so this result means exactly what it says and nothing more.
Why we're publishing this at all
We're building Litica because AI agents should remember the way people do. A human brain doesn't log information in separate silos and hope to find it later. Millions of cortical columns share the work of remembering, and knowledge learned in one place is available everywhere it's needed. In Litica's world, those columns are agents. Cross-agent memory hadn't really been measured before, so we built the benchmark. This is the first result, and there will be more.
If you want to talk or learn more about these results, feel free to reach out to me on LinkedIn.
More Research