Deterministic, no-LLM gate for LLM output - traces every claim to an evidence bank, scores quality with a fixed rubric, and flags repetition runaway, so the check is auditable and repeatable.
Input: an evidence bank and a list of things the subject may not claim
Output: flagged contradictions, a no-LLM rubric score, repetition-runaway detection
Not a general hallucination detector
from veritas_gate import TruthChecker, rubric_score, is_degenerate
gate = TruthChecker(
experience_evidence="Built a RAG system over a 35,000+ doc corpus that cut research time 85%.",
forbidden_skills=["kubernetes", "lora", "fine-tuning"], # things the subject has NOT done
forbidden_claims="cloud-native\nfull-stack",
)
gate.check("Built a RAG pipeline over a 35,000+ document corpus.").is_valid # True
gate.check("Expert in Kubernetes and LoRA fine-tuning.").is_valid # False (fabrication)
gate.check("I have no experience with Kubernetes.").is_valid # True (honest omission)
is_degenerate("AI futures, AI potentials, AI possibilities, " * 100) # True
Demo: python -m veritas_gate.example
Benchmark: RAGTruth
RAGTruth (Niu et al., ACL 2024): human-annotated LLM responses labeled for hallucination.
python benchmark/fetch_data.py # 36 MB, fetched to a git-ignored dir, not vendored
python benchmark/run.py
| | what it is | what it measures | writes results.json? |
|---|---|---|---|
| python benchmark/run.py | RAGTruth evaluation, fetched corpus | generic-grounding performance: every number below | yes |
| python benchmark/run.py --fixture | offline run, committed synthetic fixture | loader, harness and scoring branches only | |
no, never
Fixture built by benchmark/fixtures/build_synthetic_fixture.py, no corpus text
20.8% of RAGTruth's 14,289 annotated hallucination spans contain a digit
Recall capped near 0.21
Speed, three runs of the full test split: 0.288-0.298 ms per response with a fresh checker and evidence bank, 0.122-0.127 ms for check() on a reused checker
results.json byte-identical across runs, timing excluded
vs 1-3 s LLM judge: 3-4 orders of magnitude cheaper
Use: cheap deterministic pre-filter inside a configured domain
src/veritas_gate/grounding.py. Opt-in, off by default:
TruthChecker(..., enable_broadened_numeric=True, enable_entity_grounding=True, enable_novelty_check=True)
unverified_number: every digit token in the draft absent from the evidence
ungrounded_entity: a capitalized token the evidence never mentions, sentence-openers excluded
novel_content_window: a sliding window of stopword-filtered content words absent from the evidence. Window 10 tokens, 60% novel; train-split grid search over window sizes 3/5/7/10 and thresholds 0.6/0.8/1.0
Verdict: DOES_NOT_WORK (pre-registered rule in benchmark/metrics.py)
WORKS and HIGH_PRECISION_FLAGGER_ONLY need a precision lower bound of at least 60%; this one is 38.1%
Fires on 2,255 of 2,700 responses (84%)
Per task:
| task | n | naive F1 | ensemble F1 | verdict |
|---|---|---|---|---|
| Data2txt | 900 | 78.3% | 78.3% | fires on 100% of responses: ties the floor exactly, no signal |
| QA | 900 | 30.2% | 34.5% | real margin over the floor |
| Summary | 900 | 37.0% | 41.6% | real margin over the floor |
Data2txt: recall 100.0%, precision 64.3% (equals the task base rate)
QA and Summary beat their floors by 4.3 and 4.6 points
Use: pre-filter that routes to a human or a judge model
What it checks
TruthChecker, against a configured evidence bank:
Forbidden and over-claim phrases: substring-robust, whitespace- and hyphen-normalized
Not-claimable skills: word-boundary matched, honest-omission whitelist ("I have no experience with X")
Unverified impact metrics and counts: %/$/×/large-count not in the evidence
Misattribution (optional): personal-project signature under an employer block; inert without markers
Credentials and named entities: asserted but not evidenced
Claim registry (optional): claim_rules=[...], enforced at check time
Enforces configured constraints only; does not parse prose into claims.
rubric_score: deterministic 0-100 quality score. Keyword alignment without stuffing, title and intent alignment, real-metric density, parseable structure
is_degenerate: repetition-runaway output (collapsed vocabulary, or a long run of comma-items sharing a leading word)
check_claim_rules: declarative registry. One row is both the enforced rule and the rendered prompt instruction. Standalone, or via TruthChecker(claim_rules=[...])
Claim registry
from veritas_gate import check_claim_rules, prompt_rules_block
rules = [{
"id": "initiative-attribution",
"violation_type": "attribution_error",
"severity": "high",
"prompt_instruction": "The retrieval platform was self-initiated; the migration was assigned.",
"subjects": {
"self_initiated": {"any_of": ["retrieval platform"]},
"assigned_work": {"any_of": ["migration workstream"]},
},
"forbid": [{
"subject": "self_initiated",
"within_sentence": ["was assigned", "tasked with"],
"message": "the retrieval platform was self-initiated, not assigned",
}],
}]
# TRUE sentence covering both subjects -> passes
check_claim_rules("I initiated the retrieval platform and was assigned to the migration "
"workstream.", rules) # []
# the term attached to the wrong subject -> fires
check_claim_rules("I was assigned to the retrieval platform.", rules) # 1 finding
prompt_rules_block(rules) # the same row, rendered for the system prompt
Nearest-marker attribution: each forbidden term attaches to the subject whose marker sits nearest it
Rules match literal lower-cased substrings, no word boundaries
False positives: short term inside a longer word
False negatives: rule translated out of code loses its boundary logic
Private-corpus run (2026, 9,963 documents, 36.7M chars; not vendored):
rl fired on 3,410 of 9,963 documents as a registry row, 107 for the word-boundary regex (31.9×)
