Toward Zero Invalid State Admissions: A Benchmark Proposal

Status: proposal. This note defines a benchmark. It reports no results. No system, including Tinor's, is claimed to achieve zero invalid admissions; "toward" is the operative word.

Thesis. A stateful AI system should be evaluated on what it admits into shared state, not only on what it answers — and any such benchmark must score invalid admissions and useful completions jointly, so that a system which rejects everything cannot appear reliable.

Research question. How can we measure whether a stateful AI system prevents invalid state admissions without sacrificing useful task completion?

Why answer accuracy is the wrong unit

Most evaluations of language-model systems score answers: is the output correct, faithful to sources, free of unsupported content (Ji et al., 2023)? For a system that writes — records a supplier's certification, updates a capacity figure, marks a claim withdrawn — the consequential event is the write. A wrong answer misleads one reader once. A wrong admission is served to every later reader and agent until someone notices.

Risk frameworks make a similar point in general terms: measure the behaviours that matter in context, including failure modes and their impact, not only aggregate accuracy (NIST AI RMF).

Unit of evaluation

A case is: a starting state (admitted history, Principals, authority grants, disclosure rules), an input (materials and a request, possibly from an agent), and a ground-truth expected outcome for each proposition in the input.

Each proposition's expected outcome is exactly one of:

• ADMIT — should be admitted (with specified attribution, evidence class, scope and time);
• ADMIT-QUALIFIED — should be admitted only as a weaker proposition or evidence-only note;
• HOLD — should not be admitted yet; a specific question or confirmation is required;
• REFUSE — should not be admitted, with a specified reason (unauthorized, unsupported, out of scope, forged source, and so on);
• NO-CHANGE — the request should produce no state change.

For read cases, the ground truth is an expected resolution: the state projection for a stated subject, time point, knowledge cut, viewer and purpose (including value relations such as single, conflicting or indeterminate).

Metrics

Let I be the set of propositions whose ground truth is REFUSE, HOLD or NO-CHANGE (should not be admitted as proposed), and V the set whose ground truth is ADMIT or ADMIT-QUALIFIED.

Reasons matter. A REFUSE for the wrong reason (e.g., "insufficient evidence" when the true issue is "unauthorized") counts against TCR even though no invalid admission occurred, because the remedy offered to the user is wrong.

The always-reject trap

A system that refuses every proposition scores ISAR = 0 — the headline number this benchmark is named after — and is useless. To prevent that reading:

• ISAR is never reported alone; it is always reported together with VSAR and TCR on the same case set.
• A minimum VSAR and TCR is part of any claim (for example, "ISAR ≤ x at VSAR ≥ y"); the thresholds are declared before running.
• The case set must contain enough ADMIT cases that blanket refusal performs visibly badly. An always-reject baseline and an always-admit baseline are run and published alongside every system.

Equally, a system that holds everything for human review is not "reliable"; HOLD where ground truth is ADMIT counts against VSAR and TCR.

Case families

The benchmark should contain at least the following families, each with ADMIT counterparts so that the right answer is not always "no":

Ground truth

Ground truth is the hardest part and must be built deliberately:

• Authored by domain reviewers, not generated by the system under test or by the same model family.
• Written against declared rules: each expected outcome cites the contract, authority policy or disclosure rule that determines it. Where reasonable reviewers would disagree, the case is either removed or marked contested and scored separately.
• Separated from truth in the world. Ground truth is "what the declared rules require given these inputs", not "what is actually true about this fictional factory". The benchmark measures rule-conformant admission, not omniscience.
• Versioned. Cases, rules and expected outputs are versioned together; results are reported against an exact version.

Experiment inputs and expected outputs

For each case, the harness supplies: starting state; materials (text, files with hashes); the acting Principal and its grants; the request; for read cases, the query coordinates.

The system must output, per proposition: outcome (as above), the admitted record or the refusal/hold reason, and for reads, the projection with qualifiers. Free-text explanations are recorded but not scored as the outcome.

Failure criteria

A case fails if any of the following occurs:

• an invalid proposition is admitted, or admitted stronger than allowed;
• a valid proposition is refused, held without a correct question, or admitted with wrong attribution, evidence class, scope or time;
• a read projection differs from ground truth in value relation, standing, a required qualifier, or disclosure;
• history is altered (an earlier admitted record changed or deleted);
• the system reports success while its own state shows otherwise.

Certain failures are critical and reported separately regardless of averages: unauthorized admission, disclosure of withheld content, and history alteration.

Refusal policy

Systems under test must declare their refusal policy up front — which situations produce REFUSE, which produce HOLD, and with which reason vocabulary — and are scored against ground truth, not against their own policy. A declared policy that maps every uncertainty to HOLD is legitimate to declare and will score poorly on TCR, which is the intended signal.

Repeated runs

Systems that use models are non-deterministic. The protocol:

• run each case N times with fixed inputs (N declared in advance), and report per-case outcome distributions, not only the best run;
• report worst-case as well as mean for critical failures: a single unauthorized admission across N runs is a failure for that case;
• fix and publish model versions, prompts, temperatures, contracts and rule versions;
• separate the deterministic layer from the model layer where possible — for example, replay the same candidate through the admission path to check that admission is a function of the candidate, independently of how the candidate was produced.

What would count as evidence

A credible result would include: the exact case-set version; system and model versions; ISAR, VSAR, SRA, UFR and TCR with per-family breakdown and baseline comparisons; critical-failure counts; uncertainty estimates appropriate to the sample; and CDIC with the method used to measure effort. We have not run this benchmark, and this note contains no measured numbers, sample sizes or confidence intervals.

Limits and open questions

• Construct validity. Synthetic cases may not reflect the distribution of real inputs. Mixing in anonymized real cases raises consent and disclosure issues.
• Rule dependence. Results are relative to the declared rules; a lenient rule set makes ISAR easy.
• Cost metrics are soft. CDIC depends on team skill and tooling; it is useful for comparing approaches within one team more than across organizations.
• Gaming. Systems tuned to the case families may overfit. A held-out set and periodic refresh are needed.
• Independence. Results reported by the builder of the system under test are weaker than independent replication.

Where Tinor stands

Tinor's design records include a benchmark section for its reality compilation work and regression vectors for admission and resolution, and its deterministic assessment is designed to be a function of the candidate. None of that is a run of this benchmark, and no comparative or absolute performance is claimed here. Design Proposal.

Related reading

• An AI Answer Is Not a Fact
• Reality Needs a Type System
• Managing Revocation and Conflicting Claims

References

• Ji, Z. et al. (2023). Survey of Hallucination in Natural Language Generation. arXiv:2202.03629
• NIST (2023). AI Risk Management Framework (AI RMF 1.0). nist.gov
• W3C (2025). Verifiable Credentials Data Model v2.0 — validity and status semantics used in families 5, 6 and 8. w3.org