Disclaimer: This article documents a personal technical experiment on my own infrastructure. Configurations reflect my specific environment and should not be treated as security advice or guaranteed protection. Always validate independently and consult qualified security professionals before implementing any security measure. NosisTech LLC accepts no liability for outcomes arising from use of this content.
A hallucination detection agent checks whether an AI response is grounded in the material it was given. It asks: did the model answer from the source, or did it invent something? I built this version as a small proof of architecture, not as a production risk system. The value is in the pattern: source context in, AI answer in, claim-level review out.
What This Agent Does
This agent compares an AI response against a source paragraph. It asks the model to identify factual claims and label each one as SUPPORTED, UNSUPPORTED, or CONTRADICTED.
The examples use fictional companies so the model is less likely to lean on outside training knowledge. That matters because this agent is testing whether the answer is grounded in the supplied source, not whether the model knows something from somewhere else.
Why I Built This
I built this because hallucination risk is often discussed like a model problem, but I see it as an architecture problem. Everyone wants models to be more truthful, while too few systems check whether each claim is actually tied to the source material.
The standard I want to set is simple: if an AI system is summarizing, reviewing, or explaining source material, the response deserves an audit layer. This build shows that the audit layer does not need to be huge to be useful.
What Actually Happened
What actually happened: One test case was too narrow. The build became stronger when I added grounded, unsupported, and contradicted examples.
What broke: Gemini returned output that the first version did not parse as JSON.
What surprised me: Gemini and DeepSeek agreed on the important verdicts after the parser fix, even when their claim wording differed slightly.
What I would change next: I would add a stricter schema validator and a human review queue for higher-risk outputs.
Why this matters: A response that looks polished can still contain unsupported claims. Claim-level review gives operators a way to see where the risk is hiding.
The Architecture: How It Works
The architecture has five parts.
First, the agent loads model settings from .env. This keeps the model name, endpoint, and API key outside the code.
Second, it stores a few fictional test cases. Each case has a source context and an AI response.
Third, it sends both pieces of text to the model with instructions to return JSON. JSON means structured text that other software can read reliably.
Fourth, it extracts the JSON object if a provider wraps the answer in extra formatting.
Fifth, it prints the review result so the builder can compare model behavior across providers.
The Build: Step by Step
SYSTEM_PROMPT = """
You compare an AI response against its source context.
Return only JSON:
{
"score": 0.0,
"verdict": "GROUNDED | MIXED | HIGH_RISK",
"claims": [
{
"claim": "claim from the response",
"status": "SUPPORTED | UNSUPPORTED | CONTRADICTED",
"reason": "short reason"
}
]
}
Score 0.0 means mostly supported. Score 1.0 means mostly unsupported or contradicted.
"""What This Code Is Actually Doing
This is the instruction card for the reviewer model. It tells the model to compare the answer against the source and return a structured report. The score is a rough risk meter, where lower means more grounded and higher means more unsupported or contradicted.
DEMOS = [
{
"label": "Grounded",
"source": "Marble Ridge Clinics is a fictional healthcare group with 12 clinics. Its 2026 plan focuses on appointment reminders and patient portal training.",
"response": "Marble Ridge Clinics has 12 clinics and is focusing on appointment reminders and patient portal training.",
},
{
"label": "Unsupported",
"source": "Harbor & Finch Logistics is a fictional freight company. It operates 42 refrigerated trucks across three states.",
"response": "Harbor & Finch Logistics operates refrigerated trucks across three states. It also runs cargo flights, owns warehouses in Brazil, and reported 80 million USD in revenue.",
},
{
"label": "Contradicted",
"source": "Northstar Ledger is a fictional bookkeeping firm. It serves small retailers and does not provide tax filing services.",
"response": "Northstar Ledger provides tax filing services for enterprise manufacturers.",
},
]What This Code Is Actually Doing
These are the test scenarios. The first checks whether the agent recognizes a clean answer. The second checks whether it flags invented details. The third checks whether it catches a response that says the opposite of the source.
def detect_hallucinations(client, model, source, response):
"""Ask the model to check response claims against the source."""
prompt = f"Source context:\n{source}\n\nAI response:\n{response}"
result = client.chat.completions.create(
model=model,
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": prompt},
],
temperature=0,
)
return parse_json(result.choices[0].message.content)What This Code Is Actually Doing
This function sends the source and response to the model in one request. The model receives the rules first, then the material to review. temperature=0 asks for a more consistent answer, which matters when comparing providers.
def parse_json(text):
"""Parse JSON even when a model wraps it in extra formatting."""
cleaned = text.strip()
if cleaned.startswith("```"):
lines = cleaned.splitlines()
if lines[0].startswith("```"):
lines = lines[1:]
if lines and lines[-1].startswith("```"):
lines = lines[:-1]
cleaned = "\n".join(lines).strip()
start = cleaned.find("{")
end = cleaned.rfind("}")
if start != -1 and end != -1:
cleaned = cleaned[start : end + 1]
return json.loads(cleaned)What This Code Is Actually Doing
This is the cleanup layer. Some models return JSON inside a formatted code block. This function strips that wrapper and keeps the JSON object so Python can read it.
Errors I Hit During This Build
The first real runtime error was: The model did not return valid JSON. Gemini produced output that the first version did not parse cleanly. I fixed it by adding a small JSON extraction function before calling json.loads.
I also hit a local syntax-test issue while checking fenced JSON from the command line. The shell interpreted formatting characters in the test command. I fixed the test by generating those characters programmatically instead of typing them directly.
A local bytecode check also tried to write a cache folder and hit an access error in my environment. That was a tooling issue, not an agent issue. I verified syntax with an in-memory compile check instead.
Why This Matters
This pattern matters because many business failures in AI do not look dramatic. They look like one invented number in a confident paragraph. They look like a source summary that quietly adds a detail the source never said.
For legal, compliance, security, medical, or financial workflows, that kind of error changes the risk profile. This agent does not make the model safe by itself. It adds a review layer that helped me see which claims were grounded, unsupported, or contradicted in my test environment.
Tools Used
- LiteLLM (MIT): github.com/BerriAI/litellm
- OpenAI Python SDK (Apache 2.0): github.com/openai/openai-python
- python-dotenv (BSD-3-Clause): github.com/theskumar/python-dotenv
- NosisTech Agent Engineering (MIT): github.com/nosistech/agent-engineering
(c) 2026 NosisTech LLC. Licensed under CC BY 4.0. Use freely, just credit us.