Diagnosing a Model Quality Regression Without Guessing Its Identity
Use public tasks, explicit completion checks, and controlled conversation history to investigate quality changes without turning one bad answer into an identity claim.
When yesterday's working model struggles today, or a fresh conversation suddenly fixes the task, “the model got worse” is an understandable reaction. Turn that reaction into a specific observation: which requirement failed, whether the inputs and tool results matched, and when the failure first appeared.
The sources below were checked on November 10, 2026. This article provides a public fixture and investigation method. We did not run a model comparison, produce rankings, or verify any claim of provider model substitution.
Apply the same method when a new model launches. The Fable 5.2 selection guide separates vendor positioning, account access, and a public task trial before changing the current model.
Give the failure a checkable description
Replace “bad answer” with “returned prose around an array that must contain only two fields.” Replace “cannot fix the bug” with “the original failing case still fails and input validation was removed.” A concrete description tells you whether the next change helped.
The official evaluation-design guidance starts with specific, measurable criteria tied to the application. For an individual developer, a few representative tasks are a useful starting point.
Here is a fully public extraction fixture:
Extract unfinished tickets from the records below. Return only a JSON array.
Each item must contain only id and owner, sorted by id ascending.
Use null for a missing owner. Do not infer names or add explanations.
T03 | done | Mei
T02 | open |
T01 | open | Lin
Expected result:
[
{"id": "T01", "owner": "Lin"},
{"id": "T02", "owner": null}
]
Check filtering, order, field names, missing values, and JSON validity separately. This small case says nothing by itself about coding, vision, tools, or long documents. If your application requires enforced JSON structure, see structured output; a plain-text instruction is a different test.
Check conversation state before comparing providers
Run the same public task in a fresh conversation without old instructions, character material, or tools. If that helps, inspect conflicting history, missing facts after compaction, and excessive context. It narrows the investigation without proving that the service never changed.
An agent's inputs also include files, searches, and tool responses. Two runs that read different code revisions are not equivalent even if the last user message matches. Preserve public file versions and tool outcomes. For long conversations, use the context-limit checks.
Change one variable and keep every outcome
Use a record like this; it is a template, not measured data:
| Field | What to keep |
|---|---|
| Time and version | Time zone, client version, configuration date |
| Request path | Service host, protocol, exact model ID, no credentials |
| Input state | Fresh or retained history, public file version, tools |
| Completion rules | Required fields, files, or behavior |
| Observation | Individual pass/fail checks and redacted evidence |
| Change | The one input or setting changed for this attempt |
Hold the task, history, tools, and supported parameters steady. Changing the client, prompt, model setting, and source material together makes attribution difficult. Generation can vary, so retain repeated observations for important tasks while limiting the number of attempts. A single good or bad answer should not carry the whole conclusion.
If the failure follows a tool call, inspect the result the model actually received. If it follows a model switch, inspect IDs, aliases, and capabilities. Do not force unsupported parameters onto a model merely to make settings look identical.
Identity claims need stronger evidence
An October 19 V2EX thread, with 39 replies when checked, contains a poor project-generation experience, suspicions about an unnamed intermediary, and counterarguments about identification tools and model performance. It demonstrates the concern, not a verified substitution.
A separate Chinese practice discussion recommends small tasks, verification, and retained failure cases. Those are useful methods; the author's experience is not population-wide performance data. We do not reuse community prices, attribution probabilities, or unverified performance claims.
Self-reported identity, writing style, or a single drawing task can be influenced by prompts, history, sampling, and task choice. They can suggest follow-up checks but cannot independently establish upstream identity. Service claims need configuration records and whatever provider evidence is actually available.
For Nbility, correlate the request in request logs, then prepare a redacted support record. A version, model, timestamp, and exact unmet condition are much easier to investigate than a general impression of lost intelligence.
