Follow-up to the RAG Failure Mode Checklist docs update: a visual debug companion

[onestardao]

Hi folks, quick follow-up for LlamaIndex builders who are already running RAG in real projects.

I keep seeing the same pattern:

Your LlamaIndex app runs.
Indexing succeeds.
Retrieval returns something.
The query engine completes.
But the final answer is still off-topic, unstable across runs, or clearly wrong in production.

Recently I submitted a docs PR that extends the existing RAG Failure Mode Checklist with several production-focused failure families, without changing the existing sections.
PR: #20760

The added sections are specifically aimed at the “everything looks healthy, but answers are still wrong” stage, including:

• embedding / metric mismatch issues
• session and cache related memory drift
• observability gaps (hard to tell what actually happened during a run)
• index lifecycle / deployment ordering pitfalls

That checklist format works well, but in practice many people want an even faster entry point when something breaks.
So I built a lightweight companion: a single visual card you can use as a debug prompt.

RAG 16 Problem Map · Global Debug Card

This is not a replacement for LlamaIndex.
It is meant to be a simple debugging layer you can use after your LlamaIndex app is already running, when you have a real failing run and you need a clearer path to “what likely went wrong and what to try next”.

How to use (super simple)

1. Save the card image .

2. Take one failing run from your app and summarize it briefly:

   • what the user asked
   • what your app retrieved or attempted to use
   • what the final answer was
   • why you believe it is wrong

3. Upload the card image + paste that failing run summary into any strong LLM, then ask:
   “Please follow this debug card to identify the likely RAG failure modes and suggest concrete fixes + quick verification checks.”

I have tested this workflow with ChatGPT, Claude, Gemini, Perplexity, and Grok.
They can all read the card and use it to classify common RAG failures and propose reasonable next-step fixes.

If you are a LlamaIndex user and you have ever hit problems like:

• retrieval looks fine but the answer still hallucinates
• the same question gives inconsistent quality across runs
• production Q&A drifts even though the pipeline “works”
• you need a shared failure vocabulary for debugging with your team

…then this might be useful.

global debug card + short README are here:
https://github.com/onestardao/WFGY/blob/main/ProblemMap/wfgy-rag-16-problem-map-global-debug-card.md

If you try it on a real broken LlamaIndex run, I’d love to hear what failure modes it flagged and whether the suggested fixes helped.

[aniruddhaadak80]

A visual debug companion is a useful next step because long checklists are valuable reference material, but they are slower to use when someone is actively triaging a failure. Turning common RAG failure modes into a quicker diagnostic surface makes the documentation more operational.

What would strengthen this further is feedback from real debugging sessions. If certain failure patterns repeatedly co-occur or lead to the same remediation path, that could shape how the card is organized and keep it from becoming only a prettier checklist.

[onestardao]

Thanks for the thoughtful feedback.

What you are seeing right now is essentially Problem Map 2.0.
I’m currently preparing Problem Map 3.0, which should be released in the next few days. The new version will expand beyond RAG and cover more common failure patterns across broader AI systems.

If you have any interesting ideas or suggestions that could make the diagnostic surface more useful in real debugging sessions, feel free to open an issue in my repo. I’d be happy to discuss and iterate on it.

Appreciate the input.

[Nyrok]

The framing of "everything looks healthy, but answers are still wrong" maps well to an upstream problem too: the query itself.

When retrieval returns the right chunks but the final answer still drifts, the cause is often in how the prompt wraps those chunks, not just what it retrieves. A prompt with no explicit output_format block, no constraints block, no role context gives the model too much room to interpret. The same retrieved data will produce inconsistent answers across runs when the prompt is unstructured.

The debug card is a solid diagnostic surface. One axis worth adding: "Is the retrieved context being interpreted through a structured prompt or a loose one?" That distinction narrows down whether you have a retrieval problem or a framing problem.

I've been building flompt for exactly this, a visual prompt builder that decomposes prompts into 12 semantic blocks and compiles to Claude-optimized XML. Open-source: github.com/Nyrok/flompt