You can't trust an AI's confidence, so we measure it instead: the six research-backed checks behind every score we give.
Every answer our system gives you comes with a score from 0% to 100%. The score is a statement about reliability: how likely it is that this answer is correct for a person in your situation. Our target is that the number means what it says. A score of 80 should mean that answers we scored 80 have turned out to be right about 80% of the time when checked against reality. That property is called calibration, and the discipline of measuring it comes from the field that perfected it over decades: weather forecasting (Brier, 1950; Murphy & Winkler, 1977).
The score is produced by grading each answer against six criteria drawn from published research on when AI systems are right and when they are wrong. The overall score follows a weakest-link rule: an answer is only as reliable as its worst criterion. An answer with perfect source support that ignores the specifics of your situation is still unreliable for you, and averaging would hide that. The chain breaks at its weakest link, so that is what we report.
You might wonder: why not just ask the AI how confident it is? Because research shows that does not work. When language models are asked to state their own confidence, the numbers they give are poorly calibrated and systematically overconfident, clustering at high values like 80 and 90 regardless of whether the answer is right (Xiong et al., 2023). There is a structural reason: models are trained under grading schemes that reward confident guessing over admitting uncertainty, so confident-sounding answers are a learned style, not a self-report (Kalai et al., 2025). The consequence is the founding premise of this system: you cannot ask an AI how sure it is. You have to measure it.
So our grading never relies on the model's opinion of itself. Two of the six criteria are computed by code with no model judgment involved. Three are evidence-reading tasks over documents retrieved at answer time. One is a database of measured outcomes. And the grading is performed by a separate, more capable judge model, not the one that wrote the answer, because research shows models score their own work too generously (Zheng et al., 2023; Panickssery et al., 2024) and a separate calibrator outperforms self-grading (Kamath et al., 2020).
Do independent, credible sources actually support the claims in this answer?
AI systems produce more accurate answers when their claims can be traced to retrieved evidence, and they hallucinate most when generating from memory alone (Shuster et al., 2021; Lewis et al., 2020). We retrieve current sources for your question and check whether they agree with what the answer claims. A claim no source will stand behind is where fabrication concentrates.
Does the true answer to this question change over time, and how fast?
AI models learn from a snapshot of the world and degrade on facts that change after that snapshot (Lazaridou et al., 2021). Research classifies questions as never-changing, slow-changing, or fast-changing, and shows accuracy collapsing as answers become more mutable (Vu et al., 2023). "How many ounces in a pound" never moves. "What refrigerant does this model take" moved recently. We classify your question's mutability and check the dates on the evidence.
How much does the correct answer depend on details of your situation that the AI cannot see?
A large share of naturally asked questions are ambiguous, and models tend to answer just one interpretation, usually the most common one (Min et al., 2020). When the right answer depends on your wall type, your model year, your climate, the AI answers for the average case, which may be nobody's case. Research shows that detecting ambiguity and declining to answer confidently beats guessing (Cole et al., 2023). We estimate how many unstated variables the correct answer hinges on.
Is this topic well documented in what the AI learned from, or is the AI filling gaps by inference?
Model accuracy tracks how often a topic appears in training data: strong on popular, well-documented subjects, and measurably worse in proportion to a topic's rarity (Kandpal et al., 2023; Mallen et al., 2023). A niche brand plus a model number plus an unusual failure mode lives in the long tail, where the model has little or nothing memorized and reasons its way to an answer instead. Inferred facts are educated guesses.
If we ask the AI the same question several times, does it keep giving the same answer?
We ask the model your question multiple times, independently, and measure whether the answers mean the same thing. High variance in meaning across samples is a measurable predictor of confabulation: when the model is guessing, its guesses scatter (Farquhar et al., 2024, published in Nature; Manakul et al., 2023). This criterion is computed entirely by code. No model is asked anything about itself.
How have answers like this one actually performed when checked against reality?
This is the criterion no one else has. Every answer we score is stored, and its real-world outcome is recorded whenever it can be checked: against manufacturer documentation, against expert review, and against reports from people who used the answer and told us whether it worked. Those measured outcomes continuously correct the other five criteria through standard statistical calibration (Guo et al., 2017; following Brier, 1950). Where our field record disagrees with the model's own estimate, the field record wins. Measurements outrank opinions, including the machine's.
| Score | Meaning |
|---|---|
| 0% | This question has no factual answer to be right about. It is a matter of taste, values, or opinion. We say so plainly instead of dressing an opinion up as a fact. (In the research literature, declining to make a claim is called abstention, and systems that abstain honestly outperform systems that always guess.) |
| 95–100% | A known fact: the question is universal (no dependence on your situation), stable for decades, densely documented, unanimously sourced, and the AI answers it identically every time. A known fact is simply a question that scores at the ceiling on all six criteria. Literal 100% is reserved for definitional truths such as mathematics, definitions, and physical constants. |
| In between | The weakest-link verdict of the six criteria, corrected by our measured record. |
Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1).
Cole, J. R., et al. (2023). Selectively answering ambiguous questions. EMNLP 2023.
Dhingra, B., et al. (2022). Time-aware language models as temporal knowledge bases. TACL.
Farquhar, S., Kossen, J., Kuhn, L., & Gal, Y. (2024). Detecting hallucinations in large language models using semantic entropy. Nature, 630.
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. ICML 2017.
Kadavath, S., et al. (2022). Language models (mostly) know what they know. arXiv:2207.05221.
Kalai, A. T., et al. (2025). Why language models hallucinate. OpenAI.
Kamath, A., Jia, R., & Liang, P. (2020). Selective question answering under domain shift. ACL 2020.
Kandpal, N., et al. (2023). Large language models struggle to learn long-tail knowledge. ICML 2023.
Lazaridou, A., et al. (2021). Mind the gap: Assessing temporal generalization in neural language models. NeurIPS 2021.
Lewis, P., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. NeurIPS 2020.
Mallen, A., et al. (2023). When not to trust language models. ACL 2023.
Manakul, P., Liusie, A., & Gales, M. (2023). SelfCheckGPT. EMNLP 2023.
Min, S., Michael, J., Hajishirzi, H., & Zettlemoyer, L. (2020). AmbigQA. EMNLP 2020.
Murphy, A. H., & Winkler, R. L. (1977). Reliability of subjective probability forecasts. Applied Statistics, 26(1).
Panickssery, A., Bowman, S. R., & Feng, S. (2024). LLM evaluators recognize and favor their own generations. NeurIPS 2024.
Rashkin, H., et al. (2021). Measuring attribution in natural language generation models (AIS). arXiv:2112.12870.
Shuster, K., et al. (2021). Retrieval augmentation reduces hallucination in conversation. Findings of EMNLP 2021.
Sun, K., et al. (2024). Head-to-Tail: How knowledgeable are large language models? NAACL 2024.
Vu, T., et al. (2023). FreshLLMs: Refreshing large language models with search engine augmentation. arXiv:2310.03214.
Wang, X., et al. (2023). Self-consistency improves chain of thought reasoning. ICLR 2023.
Xiong, M., et al. (2023). Can LLMs express their uncertainty? ICLR 2024.
Zheng, L., et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. NeurIPS 2023.
The answer you see is generated by Claude Sonnet, a current, capable general model, so the answer we grade is a fair representation of what a leading AI would tell you. The grading is done by a more capable model, Claude Opus 4.8, kept separate from the one that wrote the answer. As stronger and cheaper models ship, we update which ones we use; the method above does not depend on the specific model.
Draft v0.1. Citations are drawn from the published literature as known to the authors; exact titles, venues, and links will be verified against the original publications before this document is released publicly. This rubric is versioned; each revision carries a changelog explaining what changed and why.
← Back to Trust