Observability is the primitive, not a feature

Every article in this series ends with some version of the same advice: instrument this, measure it, you can't improve what you can't see. That advice is correct, but it treats observability as a feature: something you layer on top of a working system as a finishing touch.

The correct model is the opposite. Observability is the substrate. An LLM application without traces and scores is not a system with missing monitoring. It is a system you cannot engineer. The difference is fundamental.

What you cannot do without it

You cannot debug a hallucination without knowing what was retrieved. You cannot catch a prompt regression without a score that was higher before the change. You cannot tune chunk size without measuring whether the new size actually improved retrieval hit-rate. You cannot justify a fine-tuning decision without a baseline to compare against.

Every intervention described in this series (every fix for hallucination, every chunking improvement, every chain-of-thought decision, every fine-tuning choice) requires a before and an after. Observability is what makes the "before" exist. Without it, you are not iterating. You are guessing, deploying, and hoping.

The teams that build reliable LLM systems are not smarter or luckier. They built the instrumentation first, and everything else follows from that.

Time to diagnose a production failure: instrumented vs. uninstrumentedWithout traces and scores, debugging an LLM failure requires reconstructing context from user reports. Instrumented systems reduce diagnosis time by orders of magnitude.

Loading chart...

What a trace actually is

A trace is a record of one user request as it moves through your system. It contains spans, individual units of work, each with its input, output, timing, and cost. An LLM call is a span. A retrieval step is a span. A tool invocation is a span. The trace is the tree of all these spans, linked under a single request identifier.

This is not exotic infrastructure. It is the same distributed tracing pattern that has existed in backend engineering for fifteen years, adapted for the non-deterministic, generative nature of LLM systems. The only new element is that LLM spans include things like token counts, prompt versions, retrieved chunk IDs, and faithfulness scores, because those are the dimensions along which LLM systems fail.

A real request, slow and expensive: find out why

This request took 1,440ms. Click a span to see what it actually spent time on.

Embed query

40ms

Retrieve (vector search)

120ms

Rerank (cross-encoder)

480ms

LLM generation

620ms

Tool call (order lookup)

180ms

Rerank (cross-encoder): 50 candidates → top-8 · cross-encoder/ms-marco-base · this is the bottleneck

Without this breakdown, the natural guess is "the LLM call is slow", it's the biggest, most visible cost. The actual bottleneck is the reranker, taking nearly as long as generation for a step nobody budgets time for. You only know that because the span exists.

DORA's own research on incident response draws a specific line here: reaching Elite-tier mean time to recovery (under one hour) is, in DORA's own findings, contingent on having metrics, logs, and traces aggregated and centralized so incidents are detected automatically rather than reported by users. Teams stuck in the Medium tier lose most of their time in the detection and diagnosis phases, not the fix itself, which is exactly the phase a trace collapses from hours to seconds.

The four questions traces answer

Is the system slower than it was yesterday? Latency per span, aggregated over time. A sudden increase in retrieval latency is a different problem than a sudden increase in generation latency. Traces separate them.

Is it getting the right context? Retrieval hit-rate, measured by whether the chunks that appeared in the trace were the right ones for the question. Without logging which chunks were retrieved, this question is unanswerable.

Is it telling the truth? Faithfulness score, measured by whether the answer's claims were supported by the retrieved context. This requires both the retrieved context and the answer, both of which are in the trace.

Why did it cost twice as much this week? Token counts per span, aggregated. Usually it is because a prompt changed and is now injecting larger contexts. Sometimes it is because a loop is triggering more tool calls than expected. The trace shows exactly which span is responsible.

Four production questions, and where their answers liveEvery production LLM question has a specific metric in a specific span. Observability connects the question to the data.

Break down by span type: retrieval latency and generation latency respond to different fixes.

The minimal viable setup

You do not need a sophisticated observability platform to start. You need to log four things for every request: the prompt and retrieved context (inputs), the generated answer (output), the span timing (latency), and the token count (cost). Put those in a table. Query it when something goes wrong.

Everything else (dashboards, alerts, eval pipelines, A/B experiments on prompt versions) is built on top of that foundation. But it requires that foundation. You cannot build an alert on a trend you have no record of. You cannot compare prompt versions you haven't been tracking.

What to log at each layerMinimal logging that makes every major failure category diagnosable.

Layer	Log this	Enables
Retrieval	Query embedding, chunk IDs, similarity scores, chunk text	Hit-rate measurement, drift detection, chunk quality analysis
LLM generation	Full prompt (prompt version tag), completion, token counts, model name	Cost attribution, prompt regression detection, hallucination scoring
Tool invocations	Tool name, input arguments, output, duration	Tool misuse detection, latency profiling, error root-cause
User interaction	Session ID, optional user feedback (thumbs/edit/abandon)	Ground truth signal for quality; the cheapest eval you have
Output	Final response delivered to user, any citations shown	Faithfulness audit; what the user actually saw

Observability is not expensive. An async logging call and a database table cost almost nothing. What is expensive is not having it, and then spending four hours per incident reconstructing from user reports what a trace would have shown in thirty seconds.

This is the same argument observability practitioners keep making in production:

Aurimas Griciunas on why tracing is core infrastructure for non-deterministic GenAI systems.

References: Langfuse, Arize Phoenix, LangSmith