How we made an open model say Not Found instead of guessing

My favourite thing our pipeline has produced so far is a row with nothing in it.

Three aviation incidents went into a run. Two came back filled: every cell carrying a value, the link it came from, and a verdict on how well that document supports it. The third came back with four empty cells, a score of 0.18, and a verdict of mismatch. The empty row is the correct answer. No single incident record exists for that item, so the honest result was nothing, four times over.

Getting gpt-oss-120b to return that nothing took more work than any answer it has ever filled in, and none of the work went where I expected. The prompt was never the problem. Under strict json_schema, a required string field is a contract to emit a string, and the decoder keeps that contract. It never reads the sentence in your prompt asking it to leave the cell blank. The fix had to happen in the schema: every field became a union with null, and the null branch got its own required reason, drawn from a closed list.

Everything below assumes an OpenAI-shaped chat completions endpoint and a JSON Schema you control. Search and ranking are their own posts. A quick word on vocabulary before the table: a verdict is one of four labels on a value (confirmed, confirmed with caveats, uncertain, mismatch), and a score is the confidence behind it, from 0 to 1.

Research Workbench Table View

Filter All3 Confirmed clean 1 With caveats 1 Candidates 0 Filtered 1

Item name	Score	Verdict	Why	Caveats	Date	Official explanation	Primary source	Still unexplained
Kinross incident 2 attempts	0.71	confirmed with caveats	Score 0.71; a USAF determination is on record and is contested, and no government or military file was located. The best available source is a user-submitted database entry.	source_contested uncalibrated_authority_domain	23 November 1953 aviation-safety.net/wikibase/161691	USAF: the F-89C was tracking an off-course RCAF C-47 and was lost over Lake Superior en.wikipedia.org/wiki/Felix_Moncla	None located. ASN WikiBase entry 161691 is user-submitted, not the USAF report aviation-safety.net/wikibase/161691	No. A determination was issued, and the RCAF pilot named in it denied being intercepted en.wikipedia.org/wiki/Felix_Moncla
Shag Harbour incident 1 attempt	1	confirmed	Entity passed quality threshold (score 1.00) with verifying data. Research task succeeded.		4 October 1967 en.wikipedia.org/wiki/Shag_Harbour_UFO_incident	No determination reached; investigated by the RCMP and Canadian Forces en.wikipedia.org/wiki/Shag_Harbour_UFO_incident	RCMP and DND records held at Library and Archives Canada bac-lac.gc.ca/…/list/43130	Yes. No conclusion was reached en.wikipedia.org/wiki/Shag_Harbour_UFO_incident
Lake Michigan 1994 incident 3 attempts	0.18	mismatch	Low score (0.18); zero of 2 evidentiary field(s) filled. The 1994 reports are a cluster of sightings across several towns, so no single incident record resolves.		Not Found	Not Found	Not Found	Not Found

Cell values, sources and caveats here are real and open to checking. The scores, attempt counts and source totals show the shape of a run and get replaced once a captured session record lands.

Lake Michigan 1994 has no incident file to find

“Lake Michigan 1994 incident” looks researchable, which is exactly the trap. There is an encyclopedia page. There is a date, 8 March 1994, and a National Weather Service radar operator at Muskegon County Airport who tracked something over the lake. The Chicago Tribune later counted over 300 witnesses across 42 Michigan counties. What there is not, anywhere, is an incident file: no investigating body, no case number, no determination (1994 Michigan UFO event).

Screening caught the shape problem before a single search ran:

Entity screening flagged ‘Lake Michigan 1994 incident’ (Resolves to a cluster of reports across several Michigan towns rather than one documented incident with a single record.) Researching it anyway because you approved it.

A person approved it anyway, so the run spent three attempts and found what there was to find. Filling those four cells would have been trivial. Each one could have carried a real link to a real page about that night, and the row would have looked as tidy as its neighbours. Abstaining took actual engineering. This post is the bill.

Strict json_schema guarantees shape, not honesty

Strict mode on both providers we use is constrained decoding. Cerebras describes it plainly: it “employs constrained decoding to ensure schema conformance at the token level, making invalid outputs impossible” (Cerebras structured outputs). Read that twice, because invalid is not the same word as wrong. Constrained decoding stops your parser throwing. It does nothing about a well-formed lie.

Groq supports strict: true on openai/gpt-oss-20b and openai/gpt-oss-120b, and requires that “all fields must be required and objects must set additionalProperties: false” (Groq structured outputs). Sit with that requirement for a second. Every field, required, always. The obvious schema for a research cell has no legal way to say nothing, and that is where the abstention problem starts.

Typing the null branch is what makes abstention legal

Here is the schema that produces a guess. It satisfies everything Groq asks for. It also starts the entire “the model ignored my instruction to leave it blank” class of bug.

{
  "type": "json_schema",
  "json_schema": {
    "name": "extracted_cell",
    "strict": true,
    "schema": {
      "type": "object",
      "properties": {
        "value": { "type": "string" },
        "source_url": { "type": "string" }
      },
      "required": ["value", "source_url"],
      "additionalProperties": false
    }
  }
}

Every response parses. So does a date the page never states. There is no token path to an empty answer, so a model that wants to leave a cell blank simply cannot, and you end up string-matching the prose that arrives instead. I know because our evaluator still carries the list from before we fixed it:

_NOT_FOUND_PATTERNS = ("not found", "unknown", "tbd", "none", "error")

The fix is to make nothing a legal value, with a typed reason on the null branch. Reduced to three fields so it fits on one screen:

{
  "type": "json_schema",
  "json_schema": {
    "name": "extracted_cell",
    "strict": true,
    "schema": {
      "type": "object",
      "properties": {
        "value": { "anyOf": [{ "type": "string" }, { "type": "null" }] },
        "source_url": { "anyOf": [{ "type": "string" }, { "type": "null" }] },
        "not_found_reason": {
          "anyOf": [
            { "enum": ["not_applicable", "no_record_found", "blocked_on_dependency"] },
            { "type": "null" }
          ]
        }
      },
      "required": ["value", "source_url", "not_found_reason"],
      "additionalProperties": false
    }
  }
}

An abstained cell is now three keys and no answer:

{ "value": null, "source_url": null, "not_found_reason": "no_record_found" }

Four of those are what the Lake Michigan row amounts to. (Both schemas here are the contract written out for this post, not a payload pasted from a run.)

Judging runs in a separate call that never sees the extraction

A schema-valid answer with a real URL attached is the hallucination that gets through review. So extraction and judgment never share a context. Extraction reads one fetched page and returns one value with the URL it came from. Judging is a separate call that sees the value and the page and little else, because in our runs a model shown its own working defends it.

Judging returns a score, a verdict, and typed caveats: each caveat a code with a severity, never a sentence the model improvised. On another run, a filled cell reading USD 1,186,345,850 carries the code entity_scope_broader_than_item. The CFTC order names Glencore International A.G., Glencore Ltd. and Chemoil Corporation collectively, and the item asked about one of them (CFTC 8534-22). The figure is the sum the order specifies, an $865,630,784 civil monetary penalty plus $320,715,066 in disgorgement, and the headline “$1.186 billion” is a rounding. That cell is correct and it is not clean. The full working is in The link works. The sentence isn’t in it.

Six deterministic gates run before any model opinion

Six checks run as plain code over the extracted values, with no model involved. One overrides the verdict to mismatch outright; the other five are multipliers. Every trigger and penalty is published on the researchers’ page. A result that trips a gate is still shown, scored lower, with the reason recorded. The Lake Michigan row filled zero of two evidentiary fields, and no prose moves that number.

Three attempts cost less than one frontier answer

Every item gets up to three attempts, each accepting a little less than the last, with named exits when a further attempt would not learn anything.

That ladder is only affordable because the model is small. gpt-oss-120b and gpt-oss-20b are American open-weight models: released by OpenAI under Apache 2.0, weights downloadable (gpt-oss-120b model card), served here by Groq and Cerebras. Groq lists gpt-oss-120b at USD 0.15 per million uncached input tokens and USD 0.60 per million output (Groq pricing). Three attempts plus a judging call on each still lands under one frontier answer at list prices. Spending three passes to return four empty cells is rational at that price, and the same model runs on both vendors (Cerebras models), so changing supplier does not change the failure taxonomy.

What this design costs

A small model under hard contracts wins on verifiability and loses on open-ended reasoning. In our own use it is worse at long original chains of thought than a frontier model, and no schema work fixes that. If your question needs reasoning rather than evidence gathered and checked, this is the wrong tool. That is the trade, and we took it knowingly.

Three things we have not measured, written down so nobody mistakes design intent for results:

We are pre-launch: no pilot, no paying customer, and quality checked by hand with a browser attached. What is real and what is illustrative in the table above is marked under the table itself.

Engagement-funded products prefer the invisible failure

An empty cell is a visible failure. A plausible cell is an invisible one.

A product measured on session length or answers per visit prefers the invisible failure, because the visible one gets complained about and the invisible one gets pasted into somebody’s report. The difference sits in what funds the thing, not in what can be built. The same decision shows up in what LoQuery refuses to be.

• Not a chatbot

  There is no conversation with the pipeline. You configure a run, watch it work, steer it while it goes, and get a table. Nothing here is trying to sound like a person.

• Not a search engine

  It orchestrates search engines, through your browser, using your connection. We do not have an index and we are not trying to build one.

• Not trained on a private library

  It reads the live open web, per run. Nothing is retrieved from a corpus we curated in advance, so nothing is as stale as the last time we updated it.

• Not a cache

  Nothing carries over between sessions. Run the same question twice and it is researched twice, because a verdict reused from last month is a verdict nobody checked.

• Not autonomous

  A run needs your browser attached and expects you in the loop. That is the design, not a limitation we are working around: a tool that finished the thinking while you were away would be the thing we built this to replace.

Check your own extraction schema for a null branch

Open the schema your extraction step sends and search it for null. If the string is absent, the model has no legal way to report that it found nothing, and every cell in your output is a cell it was required to fill.

The second check is structural. Save your schema to a file and run this:

"""Check a JSON Schema against the two structural rules strict mode adds."""

import json
import sys


def violations(node, path="$"):
    """Strict mode wants every property named in `required`, and
    `additionalProperties: false` on every object. Neither is the JSON
    Schema default, so a schema can be perfectly valid and still be
    rejected at the provider."""
    if not isinstance(node, dict):
        return
    if node.get("type") == "object":
        properties = sorted(node.get("properties", {}))
        required = sorted(node.get("required", []))
        if properties != required:
            yield f"{path}: required={required} but properties={properties}"
        if node.get("additionalProperties") is not False:
            yield f"{path}: additionalProperties is not false"
    for name, child in node.get("properties", {}).items():
        yield from violations(child, f"{path}.{name}")
    for keyword in ("anyOf", "oneOf", "allOf"):
        for index, child in enumerate(node.get(keyword, [])):
            yield from violations(child, f"{path}.{keyword}[{index}]")
    if "items" in node:
        yield from violations(node["items"], f"{path}[]")


with open(sys.argv[1], encoding="utf-8") as handle:
    document = json.load(handle)

# Takes either a bare schema or a whole response_format argument.
schema = document.get("json_schema", {}).get("schema", document)
found = list(violations(schema))

print("\n".join(found) if found else "strict-mode clean")
sys.exit(1 if found else 0)

Run it on the second schema in this post, then on one dumped from a Pydantic model:

$ python strict_check.py cell.schema.json
strict-mode clean
$ python strict_check.py pydantic.schema.json
$: required=[] but properties=['not_found_reason', 'source_url', 'value']
$: additionalProperties is not false

Those two lines describe a schema the provider will reject. Ours printed exactly that shape for two sessions before anyone looked.

Four questions, answered short

Why not just prompt the model to leave unknown cells blank?

Because under strict structured output the prompt has no vote. Constrained decoding masks every token that would break the schema, so if the schema requires a string, some string is coming back regardless of what the prompt asked for.

Does strict mode make answers more accurate?

No. It guarantees the output parses, and that is all it guarantees. Accuracy has to come from somewhere else. In our pipeline that is a separate judging call plus six deterministic gates.

Why does judging run in a separate call?

In our runs, a model shown its own working defends it. The judge sees the value and the page it came from, not the reasoning that produced the value, so fluency earns nothing.

Is abstaining expensive?

It costs attempts. At Groq list prices for gpt-oss-120b, three attempts plus judging still land under one frontier answer, which is what makes returning nothing affordable.

Early access is open on the waitlist. And if you add a null branch to your own extraction schema, I would genuinely like to hear what breaks: the failure taxonomy is the part of this we most want to compare notes on. We read everything that reaches support@loquery.com.