Test your RAG with questions the documents cannot answer
A useful RAG evaluation includes unsupported questions, missing conditions, retired editions and near-match products. Test evidence and response behavior separately.

A knowledge base that answers sentences from its own documents has passed only part of the test. RAG evaluation also needs questions whose answers are absent, ambiguous or withdrawn. Keep them in a separate group so plausible answers cannot hide unsupported claims.
What kind of missing answer is it?
Distinguish absent information, missing user context, retrieval that missed existing evidence, and a retired document. The right response may be to explain a gap, ask for a model or date, fix retrieval, or verify withdrawal. A generic failure response does not tell the user what to do next.
| Example question | Corpus state | Expected behavior |
|---|---|---|
| What voltage does AX-17 use? | Current manual states it | Answer with product-specific evidence |
| What voltage does AX-19 use? | Product absent | Explain the gap and check the model label |
| What voltage does this device use? | Model unspecified | Ask for the model |
| Which policy applied last October? | Historical edition retained | Select the applicable edition by date |
| Is review guaranteed within two hours? | Only “manual review” is documented | Do not invent a deadline or guarantee |
These are proposed cases, not measured results. Keep supported questions too. A system that refuses every question can look good on unsupported cases while remaining useless.
What belongs in an acceptance record?
Record a fixed corpus version, question, required evidence IDs and expected behavior. For supported cases, specify the important fact and conditions. For unsupported cases, state precisely what is missing. Avoid relying on a single reference sentence that rejects valid paraphrases.
An application-defined record could look like this:
{
"case_id": "missing-review-deadline",
"corpus_version": "demo-policy-v2",
"question": "人工审核一定两小时内结束吗?",
"expected_behavior": "state_missing_information",
"required_evidence_ids": ["custom-review"],
"must_not_claim": ["guaranteed_two_hours"]
}
This is not a Dify API format. The Chinese question deliberately tests the same Chinese corpus as the original fixture. The must_not_claim value is a semantic label for a reviewer or a separately validated checking system, not proof that substring matching can understand every formulation.
Microsoft's RAG evaluator guide separates retrieval evaluation from final-response evaluation. It also distinguishes groundedness from completeness: a response can avoid invention while omitting a crucial exception, or cover the topic while adding an unsupported promise.
Inspect retrieval before judging the answer
If the corpus states that custom items require manual review but the exception never reaches the model context, record the evidence-delivery failure. Then inspect whether the response respects the evidence it received. Use candidate coverage and reranking for the retrieval stage and the citation chain for the connection between sources and claims.
A similarity threshold does not define answerability. A highly ranked passage may discuss a neighboring topic; a lower-scoring passage may contain the exact exception. Check thresholds against labeled questions and revisit them after a model or corpus change.
Dify Retrieval Testing helps inspect chunks. Test the application separately for Context and final responses. Test-page settings are temporary; do not combine results from different configurations as though they were one run.
Can a small script summarize the checks?
Yes, for labels already assigned by reviewers. It does not perform semantic assessment. These records are synthetic; the second deliberately has the wrong behavior, which the program should report:
expected = {
"known": {"behavior": "answer", "evidence": {"ax17-power"}},
"unknown": {"behavior": "state_missing_information", "evidence": set()},
"ambiguous": {"behavior": "clarify", "evidence": set()},
}
observed = {
"known": {"behavior": "answer", "evidence": {"ax17-power"}},
"unknown": {"behavior": "answer", "evidence": set()},
"ambiguous": {"behavior": "clarify", "evidence": set()},
}
if expected.keys() != observed.keys():
raise ValueError("Missing or unexpected case IDs")
failed = []
for case_id, target in expected.items():
actual = observed[case_id]
if (actual["behavior"] != target["behavior"]
or not target["evidence"] <= actual["evidence"]):
failed.append(case_id)
print("failed:", failed)
Behavior labels need human review or a separately validated judging process. Letting the evaluated model declare its own answer correct is not an independent check. Evidence IDs establish source presence only, not semantic support. Review important claims for numbers, conditions, negation and commitments.
How should changes be compared?
Fix the corpus and question versions, then change one setting at a time. Report missing evidence and incorrect answers for supported cases, unsupported claims for absent or ambiguous cases, and unnecessary refusals. Save failed questions with their retrieved chunks. After fixing one, rerun it and add a paraphrase so the improvement is not tied to one wording.
Use near-match product identifiers and document retirement to create specific negative cases. The existing Dify FAQ application guide provides the broader workflow in which to run them.
Sources were checked on November 21, 2026. The summary script and synthetic missing-case/evidence branches were tested locally. No model or complete RAG pipeline was run, so this article reports no accuracy, refusal rate or production outcome.


