Internal knowledge assistants in 2026: what the benchmarks show and what has to be true first

In the most realistic public benchmark of enterprise retrieval, plain keyword search beat vector search by 17 points on correctness, and multi-document completeness scored 40%. Here is what that means for anyone about to buy one.

Before buying an internal knowledge assistant, know this: in the most realistic public benchmark of enterprise retrieval, plain keyword search beat vector search by more than 17 points on correctness. The standard pitch for semantic search does not survive contact with a real corpus.

The problem is older than the technology

The figure everyone cites for time lost to internal search comes from McKinsey Global Institute, and it is from 2012. Interaction workers spend nearly 20% of the working week looking for internal information or tracking down colleagues who can help. The versions you see expressed as 1.8 hours a day or 9.3 hours a week are someone else's arithmetic on that 20%, not McKinsey figures.

We cite the 2012 date deliberately. This is not a problem AI created an opportunity to solve. It is a problem organisations have been failing to solve for fourteen years, which should inform how confident anyone is that a new tool fixes it.

What a realistic benchmark found

Keyword search beat vector search on a realistic enterprise corpus

Roughly 500,000 documents across nine real platforms, seeded with misfiled documents, near-duplicates and conflicts.

• BM25 keyword, correctness
• Bash agent, correctness
• Vector search, correctness
• BM25 keyword, document recall
• Vector search, document recall

Source: Sun et al., EnterpriseRAG-Bench, arXiv:2605.05253, May 2026. 500 evaluation questions.

A 2026 benchmark built a corpus of roughly 500,000 documents across nine real enterprise platforms, including Slack, Gmail, Linear, Google Drive, HubSpot, GitHub, Jira and Confluence, and deliberately seeded it with misfiled documents, near-duplicates and conflicting information. Then it asked 500 evaluation questions.

The headline result is uncomfortable for anyone selling embeddings. BM25, a keyword ranking algorithm that predates the current wave by decades, reached 68.8% correctness and 68.4% document recall. Vector search reached 51.4% and 46.0%. The authors attribute the gap to embedding models being trained on public corpora with limited exposure to enterprise-specific vocabulary, which is a polite way of saying your internal shorthand is not in the training data.

Where it actually breaks

Where internal retrieval works, and where it collapses

Accuracy by question type on the same benchmark. Design around the difference.

• Correctly reporting information not found
• Resolving conflicting documents
• Constrained, single-answer questions
• Semantic matching
• Completeness, find everything about X

Source: Sun et al., EnterpriseRAG-Bench, arXiv:2605.05253, May 2026.

Split by question type and the picture becomes actionable. Single-fact lookup and constrained questions do well. Conflict resolution reaches 90%. Correctly saying "this is not in the corpus" reaches 100%, which is more valuable than it sounds.

Completeness collapses to 40%. "Find me every policy that touches contractor expenses" is the question people actually ask, and it is the one the technology is worst at.

Design around that. An assistant scoped to answer specific questions well, and to say clearly when it cannot, is deliverable today. An assistant promised as a complete answer to any question about the corpus is not.

The part that is not a technology problem

What has to exist before the assistant does

The retrieval technology is rarely the binding constraint.

• A named owner for which version of a document is current. Three versions of the expenses policy across Drive, Confluence and an email attachment is the normal state, not a failure.
• A retirement process for superseded content. A perfect retrieval system will confidently return a policy that was replaced in 2023.
• The fifty questions your team actually asks. Answered by hand first, with the time recorded. This becomes your evaluation set.
• An agreed scope for what the assistant will not answer. Saying nothing is in the corpus scored 100% in benchmark. Use that.
• A measure that is not usage. Track the questions it could not answer and what happened next. That list is the product.

McKinsey Global Institute estimated in 2012 that interaction workers spend nearly 20% of the week looking for internal information. The problem predates the technology by well over a decade.

Source: McKinsey Global Institute, The Social Economy, July 2012; EnterpriseRAG-Bench, May 2026; Thinklytics practice.

If three versions of the expenses policy exist across Drive, Confluence and someone's email attachment, a perfect retrieval system will confidently return one of them, and it has no way to know which is current. Nobody has to have made a mistake for this to happen. It is the natural state of any organisation that has existed for more than a few years.

Ownership of which document is current is a governance job. It has to exist before the assistant does, and it is the work most programmes skip because it is unglamorous and it names people.

How to measure it

Not usage. Usage rises with novelty and falls after, and tells you nothing.

Measure the questions it could not answer, and what happened to them. That list is the actual product of the first six months: it tells you which content is missing, which is stale and which is contradictory. A rising answer rate against a shrinking set of repeat questions is the signal that something is working.

What we would do first

Take the fifty questions your team actually asks most, answer them by hand, and record how long each took and how many places you had to look. That baseline costs a week and it does two things: it tells you whether the 20% figure is true for you, and it becomes the evaluation set for anything you buy.

Our RAG consulting practice builds these on a governed content layer rather than on top of whatever is in the drive, and data governance consulting covers the ownership question underneath it. For the broader pattern, see what production AI automation requires from data.

Frequently asked questions

How much time do staff lose looking for internal information?

The most cited figure is real but almost always misquoted. McKinsey Global Institute estimated in 2012 that interaction workers spend nearly 20% of the working week looking for internal information or tracking down colleagues who can help. The widely repeated conversions of that into 1.8 hours a day or 9.3 hours a week are third-party arithmetic, not McKinsey figures. The 2012 date matters and is worth stating: this problem long predates the AI wave.

Does semantic search beat keyword search for internal documents?

Not reliably, which contradicts the standard vendor pitch. A 2026 benchmark built a corpus of roughly 500,000 documents across nine real enterprise platforms including Slack, Gmail, Google Drive, Confluence and Jira, seeded with misfiled documents, near-duplicates and conflicting information. Plain BM25 keyword search scored 68.8% correctness against 51.4% for vector search, a gap of more than 17 points. The authors attribute it to embedding models having limited exposure to enterprise-specific vocabulary.

Where do internal knowledge assistants actually break?

Multi-document completeness. In the same benchmark, single-fact and constrained questions scored well, conflict resolution reached 90%, and correctly saying information was not found reached 100%. Completeness questions, meaning find me everything that touches this, scored 40%. Looking one thing up is close to solved. Assembling a complete answer across documents is not.

Why does vendor demo performance not survive contact with our content?

Because the demo ran on public-style content and your corpus is not that. The benchmark authors' own conclusion was that public benchmark performance does not necessarily transfer to enterprise retrieval. Enterprise corpora contain misfiled documents, near-duplicates, superseded versions and outright contradictions, and those are the conditions that separate systems rather than the ones vendors demonstrate against.

What has to be true before an internal assistant works?

Someone has to own which version of a document is current. The retrieval technology is not usually the binding constraint. If three versions of the expenses policy exist across Drive, Confluence and an email attachment, a perfect retrieval system will confidently return the wrong one. Ownership of currency is a governance job and it has to exist before the assistant does.

Should we buy or build an internal knowledge assistant?

Evidence on this is thin and the one available data point is weak, so weigh it lightly. MIT NANDA reported that pilots built with external partners reached deployment around 67% of the time against 33% for internally built tools. That report has documented methodological problems. The more reliable guide is that the hard part is content governance rather than the assistant, and that part cannot be bought.

How should we measure whether it is working?

Not by usage. Measure the questions it could not answer and what happened next, because that list is the actual product. A rising answer rate on a shrinking set of repeat questions is the signal. Usage alone rises during novelty and falls after, and tells you nothing about whether anyone got time back.