RAGEvaluationDifyKnowledge Base

Test your RAG with questions the documents cannot answer

A useful RAG evaluation includes unsupported questions, missing conditions, retired editions and near-match products. Test evidence and response behavior separately.

Test your RAG with questions the documents cannot answer

A knowledge base that answers sentences from its own documents has passed only part of the test. RAG evaluation also needs questions whose answers are absent, ambiguous or withdrawn. Keep them in a separate group so plausible answers cannot hide unsupported claims.

What kind of missing answer is it?

Distinguish absent information, missing user context, retrieval that missed existing evidence, and a retired document. The right response may be to explain a gap, ask for a model or date, fix retrieval, or verify withdrawal. A generic failure response does not tell the user what to do next.

Example questionCorpus stateExpected behavior
What voltage does AX-17 use?Current manual states itAnswer with product-specific evidence
What voltage does AX-19 use?Product absentExplain the gap and check the model label
What voltage does this device use?Model unspecifiedAsk for the model
Which policy applied last October?Historical edition retainedSelect the applicable edition by date
Is review guaranteed within two hours?Only “manual review” is documentedDo not invent a deadline or guarantee

These are proposed cases, not measured results. Keep supported questions too. A system that refuses every question can look good on unsupported cases while remaining useless.

Answerable, absent, ambiguous and retired evidence require different actions

What belongs in an acceptance record?

Record a fixed corpus version, question, required evidence IDs and expected behavior. For supported cases, specify the important fact and conditions. For unsupported cases, state precisely what is missing. Avoid relying on a single reference sentence that rejects valid paraphrases.

An application-defined record could look like this:

{
  "case_id": "missing-review-deadline",
  "corpus_version": "demo-policy-v2",
  "question": "人工审核一定两小时内结束吗?",
  "expected_behavior": "state_missing_information",
  "required_evidence_ids": ["custom-review"],
  "must_not_claim": ["guaranteed_two_hours"]
}

This is not a Dify API format. The Chinese question deliberately tests the same Chinese corpus as the original fixture. The must_not_claim value is a semantic label for a reviewer or a separately validated checking system, not proof that substring matching can understand every formulation.

Microsoft's RAG evaluator guide separates retrieval evaluation from final-response evaluation. It also distinguishes groundedness from completeness: a response can avoid invention while omitting a crucial exception, or cover the topic while adding an unsupported promise.

Inspect retrieval before judging the answer

If the corpus states that custom items require manual review but the exception never reaches the model context, record the evidence-delivery failure. Then inspect whether the response respects the evidence it received. Use candidate coverage and reranking for the retrieval stage and the citation chain for the connection between sources and claims.

Inspect candidate evidence and context before judging support, omissions and the next action

A similarity threshold does not define answerability. A highly ranked passage may discuss a neighboring topic; a lower-scoring passage may contain the exact exception. Check thresholds against labeled questions and revisit them after a model or corpus change.

Dify Retrieval Testing helps inspect chunks. Test the application separately for Context and final responses. Test-page settings are temporary; do not combine results from different configurations as though they were one run.

Can a small script summarize the checks?

Yes, for labels already assigned by reviewers. It does not perform semantic assessment. These records are synthetic; the second deliberately has the wrong behavior, which the program should report:

expected = {
    "known": {"behavior": "answer", "evidence": {"ax17-power"}},
    "unknown": {"behavior": "state_missing_information", "evidence": set()},
    "ambiguous": {"behavior": "clarify", "evidence": set()},
}
observed = {
    "known": {"behavior": "answer", "evidence": {"ax17-power"}},
    "unknown": {"behavior": "answer", "evidence": set()},
    "ambiguous": {"behavior": "clarify", "evidence": set()},
}
if expected.keys() != observed.keys():
    raise ValueError("Missing or unexpected case IDs")
failed = []
for case_id, target in expected.items():
    actual = observed[case_id]
    if (actual["behavior"] != target["behavior"]
            or not target["evidence"] <= actual["evidence"]):
        failed.append(case_id)
print("failed:", failed)

Behavior labels need human review or a separately validated judging process. Letting the evaluated model declare its own answer correct is not an independent check. Evidence IDs establish source presence only, not semantic support. Review important claims for numbers, conditions, negation and commitments.

Report supported and unsupported cases separately, including missing evidence and unnecessary refusal

How should changes be compared?

Fix the corpus and question versions, then change one setting at a time. Report missing evidence and incorrect answers for supported cases, unsupported claims for absent or ambiguous cases, and unnecessary refusals. Save failed questions with their retrieved chunks. After fixing one, rerun it and add a paraphrase so the improvement is not tied to one wording.

Use near-match product identifiers and document retirement to create specific negative cases. The existing Dify FAQ application guide provides the broader workflow in which to run them.

Sources were checked on November 21, 2026. The summary script and synthetic missing-case/evidence branches were tested locally. No model or complete RAG pipeline was run, so this article reports no accuracy, refusal rate or production outcome.

Related posts

Run your Agent workflow through Nbility

Get an API key and connect OpenAI-compatible models and developer tools from one place.

Get an API key