Disclaimer: This article documents my personal exploration of AI governance frameworks and regulatory concepts. Nothing in this post constitutes legal advice, compliance certification, or a guarantee of regulatory conformance. AI governance requirements vary by jurisdiction, sector, and use case. Consult qualified legal and compliance professionals before making governance decisions. NosisTech LLC accepts no liability for outcomes arising from use of this content.
I built this agent around one uncomfortable truth: an AI system can sound confident while being numerically wrong.
The Verification and Validation Agent is my small inspection station for AI-generated claims. It takes a claim, asks a model to extract the factual parts, then uses deterministic Python logic to check those parts against trusted reference data.
What This Agent Does
This agent checks whether a business claim matches a trusted reference.
The model does not get to decide whether the claim is true. The model only extracts the structure of the claim: the metric, the organization, the time period, the claimed value, and the unit. Python then compares the extracted value against a trusted reference and returns one of three outcomes: PASS, REVIEW, or FAIL.
Analogy
Think of this build like an inspection station in a warehouse.
A package arrives with a label that says what is inside. The worker at the first desk reads the label and writes down the important details. But that worker does not approve the package. The approval happens at the second desk, where the written details are checked against the official order record.
That is the architecture here. The AI reads the claim. The code checks the claim. This keeps the creative language model away from the final verification decision.
Vocabulary You Need
Verification means checking whether a claim matches a trusted source.
Validation means deciding whether the claim is acceptable for the workflow based on the verification result.
A trusted reference is the source of truth used by the agent during the demo.
Drift means the difference between the claimed value and the trusted value.
A verdict is the final status returned by the agent: PASS, REVIEW, or FAIL.
What This Is Not
This is not a full compliance platform.
It does not connect to live regulatory databases, enterprise data warehouses, audit dashboards, or legal review systems. It does not certify that a company is compliant. I built a small reference implementation that demonstrates the control point: an AI claim passes through a verification gate before it reaches the user.
Why I Built This
A lot of AI governance discussed as if logging a model output is enough.
It is not enough. A log tells me what the model said. It does not tell me whether the number was right.
So I built the smallest version of the missing layer: extract the claim, compare it to trusted data, calculate the drift, and return a verdict.
What Actually Happened
What actually happened: The first draft was close, but it allowed loose matching that could have created false confidence. I tightened the reference matching so missing fields go to REVIEW instead of accidentally matching trusted data.
What surprised me: Gemini handled the structured extraction cleanly in the pasted terminal output. The deterministic Python layer produced the intended PASS, REVIEW, and FAIL results.
What I would change next: I would add a local reference file and an audit export. That would make the pattern easier to reuse across more claims without turning this demo into a full platform.
Why this matters: Governance is stronger when the AI is not grading its own claim. This build separates reading from verifying.
The Architecture: How It Works
The agent has two layers.
The first layer is the model extraction layer. It receives plain English and returns structured JSON.
The second layer is the verification layer. It checks the extracted value against trusted data using regular Python math.
That separation is the whole point. The model helps interpret language, but the verification decision comes from code.
What to Watch When You Run It
The important output is not the wording. The important output is the verdict path.
In the final test, the agent produced three different outcomes:
PASS for a claim that matched the trusted value.
REVIEW for a claim that was close but drifted.
FAIL for a claim that was far outside the trusted value.
That tells me the control logic is working. Changing the model through MODEL_NAME changes the provider, but the verification rule stays the same.
The Build: Step by Step
def required_settings():
"""Validate required settings and return LiteLLM connection values."""
names = ["LITELLM_BASE_URL", "MODEL_NAME", "LITELLM_API_KEY"]
missing = [name for name in names if not os.environ.get(name)]
if missing:
print("ERROR: Missing required environment variables:", ", ".join(missing))
raise SystemExit(1)
print("Active model:", os.environ["MODEL_NAME"])
return os.environ["LITELLM_BASE_URL"], os.environ["MODEL_NAME"], os.environ["LITELLM_API_KEY"]
def build_client(base_url, api_key):
"""Create an OpenAI-compatible client pointed at LiteLLM."""
from openai import OpenAI
return OpenAI(base_url=base_url, api_key=api_key)What This Code Is Actually Doing
This section checks the agent configuration before any model call happens. The agent needs a LiteLLM base URL, a model name, and an API key from the environment. An environment variable is a setting stored outside the code, which keeps secrets and provider choices out of the Python file.
The client is pointed at LiteLLM, not at one fixed model provider. That is what makes the build model-agnostic.
def extract_claim(client, model_name, claim_text):
"""Ask the model to extract a claim into structured fields."""
if not isinstance(claim_text, str) or not claim_text.strip():
return {"error": "invalid_input", "original_claim": claim_text}
from openai import APIConnectionError, APIStatusError, RateLimitError
prompt = (
"Return only JSON with these fields: metric, entity, period, "
"claimed_value, unit, original_claim.\nClaim: "
)
delay = 2
for attempt in range(3):
try:
response = client.chat.completions.create(
model=model_name,
messages=[{"role": "user", "content": prompt + claim_text.strip()}],
temperature=0,
)
raw_text = response.choices[0].message.content or ""
start, end = raw_text.find("{"), raw_text.rfind("}") + 1
if start < 0 or end <= start:
return {"error": "invalid_json", "original_claim": claim_text}
try:
return json.loads(raw_text[start:end])
except json.JSONDecodeError:
return {"error": "invalid_json", "original_claim": claim_text}
except RateLimitError:
if attempt == 2:
return {"error": "rate_limit", "original_claim": claim_text}
time.sleep(delay)
delay *= 2
except APIConnectionError:
return {"error": "endpoint_unreachable", "original_claim": claim_text}
except APIStatusError:
return {"error": "provider_error", "original_claim": claim_text}
return {"error": "extraction_failed", "original_claim": claim_text}What This Code Is Actually Doing
This is the model-facing part of the agent.
The claim starts as normal language. The model is asked to turn it into JSON, which is structured data that Python can reliably read. The code also handles providers that wrap the JSON in extra text by finding the first JSON object inside the response.
Rate limits are handled with a short retry loop. A rate limit means the provider is temporarily refusing more requests. The agent waits, tries again, and returns REVIEW if extraction still cannot complete.
def trusted_reference_data():
"""Return fictional trusted values used by the demo."""
return [
{"entity": "northstar bakery", "metric": "revenue", "period": "q1 2026", "value": 1250000, "unit": "usd"},
{"entity": "harbor clinic", "metric": "patient satisfaction", "period": "april 2026", "value": 91.0, "unit": "percent"},
{"entity": "ridgeway logistics", "metric": "delivery accuracy", "period": "march 2026", "value": 98.4, "unit": "percent"},
]
def find_reference(extracted_claim, references):
"""Match extracted claim fields to trusted reference fields."""
fields = {}
for key in ("metric", "entity", "period", "unit"):
fields[key] = str(extracted_claim.get(key) or "").strip().lower()
if not all(fields.values()):
return None
for reference in references:
trusted = {key: str(reference[key]).lower() for key in fields}
if trusted == fields:
return reference
return NoneWhat This Code Is Actually Doing
This is the trusted data layer for the demo.
I used fictional organizations so the build teaches the pattern without making claims about real companies. The matching step is intentionally strict. If the model misses a field, the agent does not guess. It returns no reference, which sends the claim to REVIEW.
That matters because loose matching can create false confidence. A governance gate is more useful when uncertain claims pause instead of sliding through.
def verify_claim(extracted_claim, reference):
"""Compare extracted claim data against trusted reference data."""
base = {"claimed_value": extracted_claim.get("claimed_value"), "trusted_value": None, "drift": None, "unit": extracted_claim.get("unit")}
if reference is None:
return {**base, "verdict": "REVIEW", "reason": "No matching trusted reference was found."}
try:
claimed_value = float(str(extracted_claim.get("claimed_value")).replace(",", "").strip())
except (TypeError, ValueError):
return {**base, "verdict": "REVIEW", "reason": "The claimed value was not numeric.", "trusted_value": reference["value"], "unit": reference["unit"]}
drift = abs(claimed_value - reference["value"])
unit = reference["unit"]
pass_limit, review_limit = {"usd": (5000, 50000), "percent": (0.5, 2.0)}.get(unit, (0, 0))
verdict = "PASS" if drift <= pass_limit else "REVIEW" if drift <= review_limit else "FAIL"
return {"verdict": verdict, "reason": "Compared against trusted reference data.", "claimed_value": claimed_value, "trusted_value": reference["value"], "drift": round(drift, 4), "unit": unit}What This Code Is Actually Doing
This is the verification gate.
The claimed value is converted into a number. Then Python calculates drift, which is the distance between the model-extracted value and the trusted value. The verdict comes from the drift threshold, not from the model.
For this demo, I used one threshold set for dollars and another for percentages. Those thresholds are environment-specific design choices in this build, not universal compliance rules.
claims = [
"Northstar Bakery revenue was 1250000 USD in Q1 2026.",
"Harbor Clinic patient satisfaction was 89.7 percent in April 2026.",
"Ridgeway Logistics delivery accuracy was 91 percent in March 2026.",
]What This Code Is Actually Doing
These three claims are the test path.
The first claim matches the trusted reference and returns PASS. The second is close enough to require REVIEW. The third is far enough away to return FAIL. That gives the agent a small but meaningful verification range instead of repeating the same case three times.
Errors I Hit During This Build
The final agent runtime did not fail.
The first issue appeared during review, not runtime. The first draft used loose reference matching. If a model omitted a field, an empty value could match a trusted record by accident. I fixed it by requiring metric, entity, period, and unit to be present before any reference can match.
The second issue was endpoint error handling. The first draft converted several provider and connection failures into a generic extraction failure. I changed it so unreachable LiteLLM endpoints, provider API errors, and rate limits produce clearer REVIEW reasons.
During local verification, one PowerShell scan failed with this message: The string is missing the terminator: “. The cause was a quoted search pattern. I fixed the command by changing the quoting style and reran the scan successfully.
A generated Python cache folder also resisted cleanup with an access-denied message. I removed it after confirming it was only a generated artifact, not a project file.
Why This Matters
This agent does not just record what a model said. It checks the claim before accepting it. That difference matters in reporting, compliance review, analytics, customer support, finance, and any workflow where a confident wrong number can create damage.
The larger opportunity is a reusable claim verdict layer. Every serious agent system needs a place where language becomes structured data, structured data gets checked, and the result is visible before anyone acts on it.
Tools Used
- LiteLLM (MIT): https://github.com/BerriAI/litellm
- NosisTech Agent Engineering (MIT): https://github.com/nosistech/agent-engineering
(c) 2026 NosisTech LLC. Licensed under CC BY 4.0. Use freely, just credit us.