Most RAG pipelines are surprisingly dumb.
User types a question. Embed it. Cosine similarity. Top 5 chunks. Stuff into prompt. Done.
One shot. One fetch. The LLM never gets to say "actually, I need more context" or "that search was too broad, let me try a different angle."
We've been experimenting with a different approach: give the LLM tools and let it decide what to fetch.
The Standard Pipeline Is a Dead End
Here's the conventional RAG flow that most teams ship:
It works. For simple questions with well-structured knowledge bases, it's fine. But the moment your query requires nuance — combining information from multiple sources, resolving ambiguity, or handling broad topics — the pipeline falls apart.
The retrieval step has no feedback loop. It doesn't know if the chunks it returned are relevant. It can't rephrase. It can't dig deeper. It's a one-way street.
The Agentic Alternative
Instead of a fixed retrieval step, you run an agent loop. The model gets access to tools — web search, file system, database queries, API calls — and it decides:
- What to search for — the model formulates its own queries
- Whether the results are good enough — it evaluates before proceeding
- If it needs the full page behind a search result snippet
- If it should try a completely different search angle
- When it has enough context to produce the final output
It's the same pattern that makes tools like Claude Code effective. You define tools with typed schemas, hand them to an agent, and it figures out the rest in a loop.
What We Found
After running this on real content tasks, the differences are stark:
| Metric | Fixed Pipeline | Agentic RAG |
|---|---|---|
| Tool calls per run | 1-2 | 15-25 |
| Self-correction | None | Automatic rephrasing on bad results |
| Output variance | Low (same chunks every time) | High (different retrieval paths) |
| Latency | 2-5 seconds | 1-2 minutes |
| Output quality | Acceptable | Significantly better |
Three things stood out:
1. The agent self-corrects. Bad search? It rephrases. Missing detail? It fetches the source page. It doesn't just accept whatever the first retrieval returns.
2. Different runs produce genuinely different outputs. The non-deterministic retrieval path means the model finds different context each time. This is a feature, not a bug — it surfaces information that a fixed pipeline would never reach.
3. 15-25 tool calls is the sweet spot. That's an order of magnitude more retrieval than a fixed pipeline, and each call is guided by what the model has already learned.
The Tool Design Problem
The trick is figuring out what tools to give it. This is where most teams get it wrong.
Too few tools and the agent can't find what it needs. It loops, retries the same search with slight variations, and eventually produces a mediocre result.
Too many tools and it wastes time exploring irrelevant paths. We've seen agents spend 30+ calls wandering through tool options that aren't useful for the task.
The right starting point:
- Web search — broad context gathering
- File/document access — deep dives into specific sources
- One domain-specific data source — the thing that makes your use case unique
Then expand based on what the agent actually tries to call. If it's consistently trying to do something it can't, that's your signal to add a tool.
The Tradeoff Is Real
Let's be honest about the cost. A fixed pipeline takes seconds. An agent loop takes 1-2 minutes. You're burning more tokens, more API calls, and more wall-clock time.
For a chatbot answering simple questions, that tradeoff doesn't make sense. Stick with the pipeline.
But for content generation, research synthesis, deep analysis — anywhere output quality matters more than latency — the agentic approach wins by a wide margin. The output quality isn't even comparable.
Where This Is Going
The gap between fixed pipelines and agentic retrieval will only widen. As models get better at tool use and reasoning, the agent loop becomes more efficient. The 15-25 calls today might be 8-12 tomorrow with better planning.
If you're still doing embed → retrieve → generate in 2026, you're leaving a lot of quality on the table. The models are capable of much more — you just have to let them drive.
