Don't Pick a Vector Database First
Agents don't need relevant text; they need the right bundle in the right shape. Define the data contract before you choose a retrieval vendor.
Even a vector-database company is now admitting that vector search alone isn't enough. The reason is visible to anyone building agents: they burn the context window rediscovering what they should already know — re-reading documents, re-asking answered questions, reassembling the same context every run. One estimate puts that rediscovery at up to 85% of agent compute.
Classic RAG was built for a chatbot
Retrieval that finds three semantically similar chunks works when the answer lives in a paragraph and a human is asking. An agent doesn't ask and stop; it runs a task, and it needs a bundle — the customer record plus the policy plus the entitlement plus the prior tickets, assembled into the right shape. Leave it to rebuild that from raw search results every run and the rediscovery tax kicks in.
Match the retrieval unit to the work
Different knowledge has different shapes, and picking the wrong one forces the model to compensate at real token cost:
- Prose → vector search.
- Long structured documents (filings, contracts) → document trees, because the structure carries the meaning that chunking throws away.
- Governed business data → a semantic layer and table-native reasoning, not a spreadsheet flattened into text.
- Relationships → a graph.
Bigger context windows don't rescue you here; they invite context rot, where performance degrades as the window fills with undifferentiated material. The goal is appropriate context, not maximum context.
Start with the contract, not the vendor
Most agent projects go wrong by choosing the database first and then constraining the agent to whatever it's good at. Reverse it. Write down exactly what the agent must receive, in what form, from where, at what freshness, and who is allowed to see it. Then choose the primitives that deliver that bundle. The vendor debate dissolves into an engineering decision.