Search without an index: the model walks the document corpus itself

The RAG bottleneck

Language models have grown a lot in recent years, while retrieval in most RAG systems still relies on word overlap or similarity in a fixed embedding space. The result is an imbalance: the generator is smart, but it only sees what a simple search engine brought it.

The authors give an example. Asked about the main character of Christopher Nolan's 2000 film, word-based search drifts toward the "Gladiator" page — it matches on the year. A system that first decides which parts of the corpus are relevant at all keeps the "Nolan" and "Memento" branches and finds the right paragraph.

How DecisionRAG works

• A hierarchy instead of an index. The corpus is split into levels: topics, documents, paragraphs. The system descends from the top, like a person in a library, and at each level decides which branches are worth opening.
• Decisions without generation. A "decider" model scores all candidates of a level in one pass and outputs a relevance probability for each — without generating a single token.
• Shared context. Candidates are scored together, so their scores are comparable. A separate test showed that scoring paragraphs jointly adds 7.7 points of R@1 compared with scoring one page at a time.

Results

With no task-specific fine-tuning and no first-stage retrieval, DecisionRAG beats dense retrieval with the Qwen3-Reranker-8B reranker. On Natural Questions the search runs over 2.68 million Wikipedia paragraphs and takes 0.7 to 1.8 seconds per query.

The authors also compared it with generative navigation: the same hierarchy, but GPT-5 makes the decisions. On the first 100 Natural Questions it found the right passage first in 59% of cases against 65% for DecisionRAG, but spent 250.8 seconds and 57.2 cents per query against 1.73 seconds and 2.33 cents.

Limitations

• Pruning is irreversible. If the right paragraph sits in a discarded branch, this path will never find it. Too tight a traversal budget loses the correct cluster.
• Hierarchy quality decides. A poor structure or empty section descriptions make retrieval worse. "Without an index" does not mean without preparation: for Natural Questions, 108 thousand pages were grouped by clustering and described by a model.
• Not a win everywhere. On some metrics the navigation variant loses to a simple rerank of dense candidates. The authors advise keeping dense retrieval as a safety net.

What it means in practice

Corporate knowledge bases are already hierarchical: regulations, sections, document versions. On such corpora, navigating the structure is natural and transparent — you can see why the system chose this particular section. A sensible production setup combines it with ordinary vector search, so that a pruned branch does not become a lost answer.