Is Keyword Search Dragging Your Hybrid Relevance Down?

Whether you are working on a RAG pipeline or a standalone retrieval system, the ability of your hybrid search to surface relevant documents may mean the difference between your agent performing a meaningful next step or sending your system (or user) down a rabbit hole, wasting tokens and time. For the end-user, the difference between a helpful response and a frustrating experience may be one relevant document showing up at the top of the results.

When search behavior is suboptimal, the tendency is to look for, train, or tune a better model. But with hybrid search it takes two to dance. Keyword search and semantic search must work together to surface the best results. Keyword search is often treated as a solved problem, and in many ways it may be. But no two keyword search solutions are the same. Even while implementing similar tokenization, the ranking and fusion math could be drastically different. Leaving your keyword signals unexamined may cause weeks of work focused on the wrong problem.

The Question

The common wisdom is that using vector similarity search produces better results than the keyword search alone, but to get the best results with one-shot retrieval (no re-ranking), one must use a hybrid search (combine the keyword and semantic signals). Does this common understanding hold true under scrutiny? Is semantic search always better than keyword search? Is hybrid search always better than semantic?

Given the same ML model and the same dataset, the semantic search alone should produce about the same results on any system (accounting for the differences in vector search parameters). So the only differentiators in the hybrid search results would be the quality of keyword search and the fusion math.

Most approaches default to RRF (Reciprocal Ranked Fusion) as their (often only) fusion algorithm, the question then becomes, "how does the relevance of the keyword search affect the relevance of the hybrid search?"

Luckily, search relevance can be measured. This post follows this rabbit hole.

About nDCG@10

Throughout this document we will use nDCG@10 as the metric for search relevance. nDCG@k (normalized Discounted Cumulative Gain at k) is a metric that measures the quality of ranking results. It considers both the relevance of retrieved documents and their position in the result list. A perfect ranking would have an nDCG of 1.0. Higher values indicate better search effectiveness. nDCG@10 measures the quality of top 10 results.

Is Semantic Always Better than Keyword Search?

Before we get into the nitty gritty of the hybrid search, let us answer the first question from our introduction: no, semantic search is not always better.

It depends on your data. Semantic search is usually better on Natural Language (NL) datasets, but as we've seen in our benchmarks¹ of the PC Parts dataset, models can struggle with identifier-heavy data (part numbers, SKUs, special chars, etc.) - there is simply not enough semantic meaning in those datasets for a model to produce meaningful inferences about relevance. Keyword searches alone do significantly better with this sort of data. Consider these numbers from the above benchmark:

Metric	Full Text	Model
nDCG@10	0.8216	0.5494

Dense model alone scored significantly worse than Full Text keyword search. It found relevant documents only about half of the time. Almost 30% of relevance was left on the table compared to Full Text search.

Is Hybrid Search Always Better than Semantic?

Hypothesis

Whether hybrid search is better than semantic search alone depends on the quality/relevance of the keyword search.

How to Test the Hypothesis

In order to test our hypothesis we need to compare the relevances of the hybrid search results between systems with significantly different keyword search quality. Fortunately for us, such systems exist.

In another set of benchmarks² we measured keyword-only search relevance of Typesense, Meilisearch, Elasticsearch, and Amgix, on a number of BEIR datasets. Here is the summary of the nDCG@10 results for the tested datasets:

BEIR Dataset	Docs	BM25 Baseline	Typesense	Meilisearch	Elasticsearch	Amgix
SciFact	5K	0.6650	0.3386	0.3616	0.6953	0.6637
Quora	523K	0.7890	0.2521	0.2547	0.8058	0.8018
NQ	2.6M	0.3290	0.0364	0.0490	0.3116	0.3121

BM25 baseline numbers are from Pyserini BEIR Regressions³

As you can see, there are some significant disparities in search relevance between two groups of engines: Typesense and Meilisearch on one end of the spectrum, Elasticsearch and Amgix on the other. While Elasticsearch and Amgix consistently score in the ballpark of BM25 baseline, Typesense and Meilisearch results are about half as relevant on the small SciFact dataset and get progressively worse on bigger datasets. By the time we get to the NQ dataset with 2.6M documents, Typesense and Meilisearch relevance numbers suggest that they return almost no relevant results.

Test Setup

To test our hypothesis that keyword search relevance is a significant variable in hybrid search quality, we will compare the hybrid search results between Meilisearch and Amgix on the small SciFact dataset. The SciFact dataset is small (saving us indexing times) and Meilisearch keyword search was strongest on it.

Both systems will use the same ML model: BAAI/bge-small-en-v1.5. Documents will contain the same flat representation of the text: {{doc.title}}\n{{doc.text}}. Meilisearch uses all defaults for keyword and model inference. Amgix uses Full Text (full_text) tokenizer with all defaults. We'll measure nDCG@10 at multiple settings of, what Meilisearch calls "semantic ratio" (weight of semantic search results in the fusion logic): from 0 (pure keyword search) to 1 (pure semantic search). In Amgix we will also compare two fusion modes: RRF (default) and Linear fusion.

Results

Semantic Ratio	Search Type	BM25 Baseline	Meilisearch	Amgix (RRF)	Amgix (Linear)
0.00	Keywords Only	0.6790	0.3767	0.6918	0.6918
0.25	Hybrid		0.5880	0.7080	0.7258
0.50	Hybrid		0.7171	0.7365	0.7437
0.75	Hybrid		0.7190	0.7342	0.7340
1.00	Model Only		0.7190	0.7200	0.7200

Bold numbers represent the best score for given engine/configuration

We excluded Amgix (RRF) numbers from the chart for visual clarity - Linear and RRF numbers are very close.

Discussion

Things to note in the results:

The keyword-only results are familiar to us from the previous benchmarks. Amgix scores a little above the baseline, while Meilisearch produces about half the relevance. Nothing new here.
Model Only (semantic) search numbers are also predictable: the difference between 0.7190 and 0.7200 is insignificant and probably due to the vector search tuning parameters in the systems. The same model produces the same, basically, relevance numbers. That's what we would expect.
Hybrid space is where things get interesting:
- In all configurations tested, hybrid search improved relevance compared to the pure keyword search results. The jump from 0.3767 to 0.5880, and then to 0.7171, for Meilisearch is a huge improvement over their keyword alone numbers. Amgix numbers improved too, but not as dramatically, since the keyword baseline was already fairly high.
- But let's now look at how the hybrid numbers compare to the pure semantic (model only) numbers. Meilisearch hybrid relevance never exceeded the relevance of the model alone. In fact, looking at these numbers, one can make an argument that introducing their keyword search into the mix (at least with this one dataset tested) doesn't make search results better.
- The Amgix story is different, in 5 out of 6 hybrid measurements, hybrid configuration outperformed pure semantic search. The only exception is RRF config with semantic ratio of 0.25 where the system scored 0.7080 (probably due to the outweighed significance given to the keyword signal).
- Amgix hybrid scores of 0.7365 (RRF) and 0.7437 (Linear) are meaningful improvements (+2.2%, 3.2%) over pure semantic search relevance.

Conclusion

In this test, with this dataset, with these configuration settings, it seems like our hypothesis is correct. The quality of your keyword search may make the difference between the hybrid search relevance being an improvement over pure semantic search or not.

The devil is always in the details, but if your keyword search is broken, including it in a hybrid search may not give you the improvements you are looking for.