Project·Reference architecture

Graph-Augmented Agentic RAG

Similarity and relationship are different retrieval problems.

Documented May 6, 2025

Why I made it

I sketched this architecture to make the retrieval choice explicit instead of treating graph search as a decorative upgrade to vector RAG.

Standard RAG retrieves semantically similar chunks. Graph RAG traverses entity relationships. Agentic RAG iterates and self-corrects. This project combines all three: an MCP server exposes the knowledge graph, a planning agent decides whether to retrieve by similarity or traverse by relationship, and a critic agent evaluates faithfulness before returning. The synthesis handles multi-hop questions that defeat flat vector search.

Tools: LangGraph · Neo4j · MCP · RAG · GPT-4o

There's a category error in how people talk about GraphRAG, as a "better RAG": more powerful, more sophisticated, the upgrade. That framing leads teams to bolt a knowledge graph onto everything and then wonder why the expensive thing didn't outperform the simple thing.

The problem

Vector RAG asks: what passages are similar in meaning to this query? GraphRAG asks: what entities are connected to this query, and what do their relationships reveal? These sound like variations on a theme; they're not.

Consider a corpus of organizational documents: Alice is mentioned in a project proposal, her team in a budget document, that team's budget cut in a board memo. The question "how did the budget decision affect Alice's project?" spans three documents. No single chunk contains the answer; the answer is in the structure connecting the chunks, not their content. A vector index will retrieve chunks about Alice, or about budgets, or about proposals, but it can't tell you how they chain together.

This project is built around the position that vector search and graph traversal solve different problems, and a serious RAG system needs both, plus a way to decide which to use, and a way to check the answer before returning it.The routing decision is the hard part, and it is easy to over-engineer. A classifier that gets it right most of the time, with vector search as the fallback, beats a clever router that fails in ways nobody can debug.

How it works

The diagram above shows the request path. Three pieces:

1. A planning agent looks at the incoming query and decides whether it needs similarity search, graph traversal, or both. Content-lookup and paraphrase-style queries route to vector search. Multi-hop, entity-centric, and "how does X relate to Y" queries route to the graph.
2. An MCP server exposes the knowledge graph (Neo4j) as a set of tools the agent can call (entity lookup, relationship traversal, neighborhood expansion) rather than requiring the agent to write Cypher directly.
3. A critic agent evaluates the retrieved evidence for faithfulness before generation: does the retrieved context actually support an answer to this query, or does the agent need to retrieve again with a refined query?

Signal	Route to vector search	Route to graph traversal
Query type	Content lookup, paraphrase matching	Multi-hop relational, entity-centric, global summary
Answer location	Contained in a single chunk	Spans relationships between multiple chunks/entities
Explainability need	Medium — source chunks are traceable	High — graph path doubles as a reasoning trace

When a query has both a content component and a relational component, the planner runs both paths and merges results with Reciprocal Rank Fusion before handing evidence to the generator.

Key design decisions

• Routing instead of always-graph. Entity extraction at index time is expensive: building and maintaining the graph costs far more per document than chunking for vector search. Routing means the graph is queried (and kept warm) only for the slice of traffic that actually benefits from it.
• MCP as the graph interface, not raw Cypher generation. Letting an LLM write arbitrary Cypher against a production graph is the same failure mode as letting it write arbitrary ALTER TABLE statements (see DiffDDL); one bad traversal query can be a denial-of-service against the database. Exposing a fixed set of MCP tools (lookup, traverse, expand) bounds what the agent can do.
• A critic agent as a faithfulness gate, not just a re-ranker. The critic's job is narrower than "is this a good answer": it specifically checks whether the retrieved evidence supports the claim the generator is about to make. If not, it triggers one re-retrieval pass with a refined query before giving up and saying so explicitly.

Status

This is a working prototype exercised against a synthetic multi-hop QA corpus (entities, teams, and events with several hops of relational depth). The planner/critic loop and the MCP graph tools are implemented end-to-end; what's still open is hardening the re-retrieval loop so the critic can't send the agent into an unbounded retry cycle on genuinely unanswerable queries. No code is public yet.

Limitations

• Graph maintenance is the real cost. When a source document changes, every entity and relationship it contributed has to be re-resolved; this is the ongoing tax that makes "route only when needed" the right default rather than an optimization.
• The critic adds latency. A faithfulness check (and possible re-retrieval) on every query is a real cost; for latency-sensitive applications, the critic should probably be sampled rather than run on 100% of traffic.

For the chunking decisions that feed both retrieval paths, see Chunking is choosing the unit of meaning. For how the system reasons about what it actually "knows" versus what it retrieves, see LLM knowledge has three shapes

---

Related notes

• Chunking is choosing the unit of meaning
• LLM knowledge has three shapes