Observability for LLM Apps
Because once the model calls tools, console.log becomes a sad little flashlight.
Observability means you can inspect what happened after a user says "the AI gave a bad answer." In normal apps, you check logs, metrics, and traces. In LLM apps, you also need the prompt, retrieved context, model output, tool calls, costs, and sometimes the entire conversation path.
What to capture
Inputs: user message, system instructions, retrieved documents, tool schemas, and relevant metadata.
Outputs: model response, structured fields, refusal reasons, and parser errors.
Tool calls: tool name, arguments, result, latency, and failure.
Performance: model, token counts, cost, latency, retries, and cache hits.
Feedback: thumbs, labels, edits, support tickets, and human review notes.
Why it matters
Most LLM failures are system failures. Bad retrieval. Ambiguous instructions. Missing validation. Wrong model. Tool schema mismatch. Without traces, every failure looks like "the model is dumb."
With traces, you can see where the system lost the plot.