metor memory takes #1 on EnterpriseRAG-Bench

We have news we are proud of: metor memory holds the top spot on the official EnterpriseRAG-Bench leaderboard, with an overall score of 80.34. The maintainers of the benchmark verified our submission, rescored it themselves, and published it. Single pass: one agent, one attempt per question, no retries, no cherry-picking.

EnterpriseRAG-Bench is, as far as we know, the hardest public benchmark for question answering over company knowledge. Below is what it measures, why we think it is the benchmark to watch, and where our lead actually comes from.

The standings

Official leaderboard standings, 28 August 2026. Further down the board: RAGFlow, Amazon Q (Kendra), Azure AI Search, hRAG.

Two things stand out. First, it is close at the top. CDL sits 1.4 points behind us, and Troml answers slightly more questions correctly than we do. Second, look at where the lead comes from. Answer completeness: metor memory has the highest value on the board. Invalid extra documents: among the lowest. The engine returns whole answers with little noise, and the leaderboard score rewards exactly that. A question only scores if the answer is correct, and then it scores its completeness. Confident half-answers count for little; confident nonsense counts for nothing.

Why this benchmark is different

Most retrieval benchmarks are built on public text: Wikipedia, web pages, forums, scientific articles. Company knowledge looks nothing like that. It lives in chat threads, email chains, tickets, meeting transcripts and half-maintained wikis, full of duplicates, contradictions and things filed in the wrong place. The team at Onyx built EnterpriseRAG-Bench to close that gap, and they describe it in their paper, EnterpriseRAG-Bench: A RAG Benchmark for Company Internal Knowledge (Sun, Rahmfeld et al., May 2026). Four design choices make it meaningful.

A corpus that looks like a company

The corpus simulates one technology company, about 512,000 documents across nine source systems, with volumes that follow real life rather than convenience: roughly 275,000 Slack messages, 120,000 emails, 35,000 Linear tickets, 25,000 Drive documents, 15,000 CRM records, 10,000 meeting transcripts, 8,000 pull requests, 6,000 support tickets and 5,000 wiki pages. Chat and email dominate by an order of magnitude, exactly as in a real organisation where the informal channels produce far more than the curated ones.

The documents are not generated in isolation. Everything grows out of shared scaffolding: a company overview, a set of initiatives, an employee directory, and project manifests. Documents reference the same decisions, meetings and tickets, so a question can depend on several of them, and internal codenames and jargon mean nothing outside the company.

Noise on purpose

Real archives are not clean, so the benchmark adds the mess deliberately. Eight percent of the documents were moved to plausible but wrong locations, some by random walk, some by a model that picks a nearby folder a tired colleague might have picked. Near-duplicates were created with specific facts changed, sometimes across source types, so that a Slack message can contradict a CRM record. Off-topic directories full of memes and hackathon notes sit next to the real material. A system that answers from the first document that looks right will answer wrongly.

Ten kinds of questions

The 500 questions span ten categories, each targeting a different capability: 175 basic lookups, 125 semantic questions with deliberately low keyword overlap, 40 that need distant sections of one long document, 40 that aggregate a project across documents, 30 where qualifiers narrow several candidates down to one, 20 with contradicting sources, 20 that demand an exhaustive list, 20 that hide in informal corners, 10 that need a synthesis across the corpus, and 20 whose only correct answer is that the information is not there. That last category is the one most systems fail: it rewards saying “no” instead of producing something plausible.

Scoring that punishes confident nonsense

Each answer is judged on four dimensions. Correctness is a binary judgement against the gold answer. Completeness is the fraction of the gold answer’s atomic facts that the answer supports, each fact checked on its own. Document recall counts the gold documents that were retrieved, and invalid extra documents counts the noise that was retrieved with them. The leaderboard uses one aggregate: per question, the completeness if the answer is correct, otherwise zero, averaged over all 500 questions.

The gold set is not treated as gospel either. When a system retrieves documents outside the annotated set, three independent judges classify every candidate as required, valid or invalid, and only a majority can change the gold labels. All leaderboard entries are scored against the same fixed gold set, so the numbers are comparable regardless of when a system was evaluated.

Harder than it looks

The authors also show why this corpus is hard. In embedding space, each document’s ten nearest neighbours are as close as in Onyx’s own real company data, and much closer than in open-web benchmarks: distractors are everywhere, and a retriever cannot rely on a wide margin between the right document and the rest. Their baselines make the point in numbers. Plain BM25 keyword search reaches 68.8 percent correctness and 56.0 percent completeness. Vector search with a strong embedding model reaches only 51.4 and 42.9, even on the semantic questions it was supposed to win. An agent with shell tools that explores the corpus for up to ten minutes per question reaches 60.6 and 61.1. Performance on public benchmarks, the paper concludes, does not transfer to enterprise retrieval.

What we take from it

The score lives in the memory, not the model. We ran the identical memory under two harnesses and four frontier models. The quality lives in the engine: how it counts, disambiguates, decides when semantic matching helps, and makes the agent read before it claims. For our own runs we settled on Claude Code with Opus 5, a choice, not a dependency; metor memory speaks MCP and ships a CLI, so the harness and the model remain yours to pick.

Benchmarks with teeth are a gift. Before publishing, the maintainers had us answer a fresh, blind validation set and rescored our entire submission themselves. Our number went up under their scoring. That is how trust in a leaderboard is built, and we are grateful to Joachim Rahmfeld and the Onyx team for running it this way.

A benchmark is a benchmark. The corpus is synthetic, one company at one stage of its life, and the authors say so plainly in their limitations. It is the best public proxy we know of for the real thing. The real thing is your knowledge base, and that is where we would like to run next.

Interested in the detailed results, a demo, or metor memory on your own data? About metor memory, or write to get-in-touch [at] metor.com.