When a single AI makes a decision, there is no check on its reasoning. It produces an output and the pipeline moves forward. That may be fine for low-risk summarization. It is not enough for decisions that affect people, money, approvals, denials, claims, or compliance workflows.
I wanted to know what happens when you force two AI agents to evaluate the same problem independently, compare their conclusions, and only proceed automatically when they genuinely align. What I built is a conflict resolution system. The important part is not insurance claims. The important part is the checkpoint pattern.
What This Agent Does
The Conflict Resolution Agent processes sample insurance claims by routing each claim through two independent reviewer agents. The first reviewer is a Coverage Reviewer. Its job is to evaluate whether the claim appears valid and covered. The second reviewer is a Risk Reviewer. Its job is to scrutinize the same claim for weak evidence or unjustified payout risk.
That difference in framing matters. If both reviewers are told to think the same way, the second reviewer adds very little. The point is to create two legitimate perspectives. One is asking, “Does this look covered?” The other is asking, “Where could this go wrong?”
After both reviewers return their recommendation and confidence score, the system compares them. If they agree and the stronger confidence score is high enough, the system can approve or reject automatically. If they disagree, or if the confidence gap is too large, or if the confidence is too low, the claim does not move forward automatically. It becomes HUMAN_REVIEW_REQUIRED.
The current repo does not block the terminal waiting for a person to type a decision. That was removed on purpose. Instead, escalated cases are marked as requiring review. If you want to simulate a human override during testing, you can set HUMAN_REVIEW_DECISION=APPROVE or HUMAN_REVIEW_DECISION=REJECT in the environment.
Every decision gets written to a JSONL audit log with timestamps, claim ID, reviewer outputs, conflict status, final decision, and who decided it: AUTO, CHECKPOINT, or HUMAN.
Why I Built This
Most AI decision systems I had seen up to this point were single-agent pipelines. One model receives the input, reasons about it, and returns a verdict. The problem with that architecture is the same problem you have with any single point of judgment: there is no independent check. The model can be confident and still be wrong, and nothing in the pipeline catches the weakness.
What I wanted to build was a system that treated disagreement as a signal rather than an error. In human organizations, when two experienced reviewers reach different conclusions about the same case, that disagreement is valuable information. It means the case is ambiguous, incomplete, or risky. Those are exactly the cases that should not move automatically.
The architecture uses separation as its core principle. The reviewers generate assessments. The conflict logic compares those assessments. The decision router decides whether the case can move automatically or needs a human checkpoint. Those jobs are deliberately separate.
Think of it like a loan committee. One person reviews the application for eligibility. Another reviews it for risk. If both are aligned and confident, the process can move. If they disagree, the disagreement goes to a human decision-maker. The disagreement is not a nuisance. It is the reason the checkpoint exists.
What Actually Happened
What I expected: the two reviewers would frequently disagree, producing human-checkpoint cases on most claims.
What actually happened: the simplified system made the routing logic clearer than the older version. The current build is not trying to create dramatic disagreements. It is proving the flow: independent reviews, comparison, automatic decision only when aligned and confident, checkpoint when not.
What broke: the older version had more moving parts than the teaching goal required. Duplicate reviewer functions, retry scaffolding, and blocking terminal input made the post feel more complex than the architecture needed to be.
What surprised me: the important lesson was not the insurance domain. It was the checkpoint boundary. A claim does not need to be “hard” for the pattern to matter. The system needs a visible rule for when autonomy stops.
What I would change next: I would add richer reviewer reasoning and structured explanation fields, but only after preserving the simple routing rule. Verdict and confidence are enough to teach the pattern. Reasons would make the human review screen stronger.
Why this matters: “human in the loop” is too vague by itself. A useful system needs to define exactly when the human enters, what information they receive, and what gets logged afterward.
The Architecture: How It Works
The system has four distinct layers.
The first layer is the claim data. In this demo, the claims are built into the Python file. Each claim has a claim ID, description, amount, and claimant. The examples are simple on purpose because the claim domain is not the main lesson.
The second layer is the reviewer pair. Two reviewers receive the same claim but different perspectives. One looks for coverage and validity. One looks for risk and weak evidence. They do not talk to each other. They each return a recommendation and confidence score.
The third layer is the conflict check. The system compares the two recommendations and confidence scores. If the recommendations differ, that is a conflict. If the confidence gap is larger than the configured threshold, that is also a conflict.
The fourth layer is the decision router. If there is no conflict and the strongest review is confident enough, the system uses that recommendation automatically. If there is conflict or low confidence, the final decision becomes HUMAN_REVIEW_REQUIRED unless a simulated human override is configured.
That is the whole pattern. Two independent reads. One comparison. One routing decision. One audit record.
The Core Components
The reviewer pair uses the same function with different names and perspectives. That matters because the code is simpler and less repetitive. Instead of maintaining two nearly identical reviewer functions, the system keeps one reviewer function and changes the role instructions.
The conflict detector is basic arithmetic and set comparison. It checks whether the recommendations disagree and whether the confidence scores are too far apart. Simple math is doing governance work here.
The decision router is the traffic controller. It does not evaluate the claim itself. It reads the two reviews, checks the conflict result, checks the auto-decision confidence threshold, and routes the claim to AUTO, CHECKPOINT, or simulated HUMAN.
The audit log writes every decision to a JSONL file. JSONL means one JSON object per line. It is useful for audit logs because each decision becomes a separate record that can be reviewed later.
The Model-Agnostic Layer
Every model call routes through a LiteLLM-compatible HTTP endpoint. The code reads LITELLM_BASE_URL, MODEL_NAME, and LITELLM_API_KEY from the environment.
The current repo does not import the OpenAI SDK. It does not import the LiteLLM Python package. It sends a direct HTTP request using Python’s standard library.
That keeps the project small. The model can change behind the endpoint, but the conflict-resolution architecture stays the same.
The Build: Step by Step
Step 1: The reviewer roster.
REVIEWERS = [
("Coverage Reviewer", "Evaluate whether the claim appears valid and covered."),
("Risk Reviewer", "Scrutinize the claim for weak evidence or unjustified payout risk."),
]What This Code Is Actually Doing
This is the two-person review panel. One reviewer is assigned to coverage. The other is assigned to risk.
For a beginner, think of this like assigning two people to inspect the same invoice. One checks whether the invoice matches the purchase order. The other checks whether anything about the invoice looks suspicious. They are both looking at the same document, but they are not looking for the exact same thing.
That difference is the point. Independent review only helps if the reviewers have distinct roles.
Step 2: Calling the model.
def call_llm(prompt: str) -> str:
payload = json.dumps(
{
"model": os.getenv("MODEL_NAME"),
"messages": [{"role": "user", "content": prompt}],
"temperature": 0,
}
).encode("utf-8")
request = Request(
os.getenv("LITELLM_BASE_URL").rstrip("/") + "/chat/completions",
data=payload,
headers={
"Authorization": "Bearer " + os.getenv("LITELLM_API_KEY"),
"Content-Type": "application/json",
},
)
with urlopen(request, timeout=60) as response:
return json.loads(response.read().decode("utf-8"))["choices"][0]["message"]["content"].strip()What This Code Is Actually Doing
This function is the phone line to the AI model. It packages the prompt, model name, and temperature into a JSON request and sends it to the configured LiteLLM-compatible endpoint.
Temperature is set to 0 because this is a decision workflow, not a creative writing task. Lower temperature pushes the model toward more consistent responses.
The important architecture point is that every reviewer uses the same model-call function. If the endpoint changes, the code has one place to update. The rest of the decision logic stays untouched.
Step 3: Parsing reviewer output.
def parse_review(raw: str) -> dict:
cleaned = raw.strip().removeprefix("```json").removeprefix("```").removesuffix("```").strip()
try:
data = json.loads(cleaned)
except json.JSONDecodeError:
data = {}
recommendation = data.get("recommendation", "REJECT")
confidence = float(data.get("confidence", 0))
if recommendation not in {"APPROVE", "REJECT"}:
recommendation = "REJECT"
return {"recommendation": recommendation, "confidence": max(0.0, min(1.0, confidence))}What This Code Is Actually Doing
This function is the translator between messy model output and clean system input.
Even when you tell a model to return only JSON, some providers wrap the answer in markdown code fences. The first line strips common code fences with string methods, so the agent does not need the regex module. Then the function tries to load the response as JSON.
If the recommendation is missing or invalid, it defaults to REJECT. If confidence is outside the valid range, it gets clamped between 0.0 and 1.0. Clamped means forced back inside the allowed range.
Step 4: Running one reviewer.
def review_claim(claim: dict, reviewer_name: str, perspective: str) -> dict:
prompt = (
f"You are {reviewer_name}. {perspective}\n"
"Return only JSON with keys recommendation and confidence. "
"recommendation must be APPROVE or REJECT. confidence must be 0.0 to 1.0.\n\n"
f"Claim:\n{json.dumps(claim, indent=2)}"
)
result = parse_review(call_llm(prompt))
result["reviewer"] = reviewer_name
return resultWhat This Code Is Actually Doing
This function sends one claim to one reviewer.
It builds the reviewer’s assignment, includes the claim details, asks for a strict JSON response, and then sends the raw model output through the parser. After parsing, it adds the reviewer name to the result.
The same function can run both reviewers because the role changes through the reviewer_name and perspective values. That keeps the code smaller and easier to trust. There is one review mechanism, used twice with different instructions.
Step 5: Deciding whether the claim can move automatically.
def decide(claim: dict, reviews: list[dict]) -> dict:
threshold = float(os.getenv("CONFLICT_THRESHOLD", "0.25"))
recommendations = {review["recommendation"] for review in reviews}
confidence_gap = abs(reviews[0]["confidence"] - reviews[1]["confidence"])
conflict = len(recommendations) > 1 or confidence_gap > threshold
if conflict:
final = os.getenv("HUMAN_REVIEW_DECISION", "HUMAN_REVIEW_REQUIRED")
decided_by = "HUMAN" if final in {"APPROVE", "REJECT"} else "CHECKPOINT"
else:
best = max(reviews, key=lambda item: item["confidence"])
final = best["recommendation"] if best["confidence"] >= float(os.getenv("AUTO_APPROVE_CONFIDENCE", "0.85")) else "HUMAN_REVIEW_REQUIRED"
decided_by = "AUTO" if final in {"APPROVE", "REJECT"} else "CHECKPOINT"
return {
"timestamp": datetime.now(timezone.utc).isoformat(),
"claim_id": claim["claim_id"],
"final_decision": final,
"decided_by": decided_by,
"conflict_detected": conflict,
"reviews": reviews,
}What This Code Is Actually Doing
This is the traffic controller.
First, the function checks whether the two reviewers recommended different outcomes. If one says approve and the other says reject, that is a conflict.
Then it checks whether their confidence scores are too far apart. If one reviewer is very confident and the other is unsure, that gap matters even if they technically gave the same recommendation.
If there is a conflict, the system does not make an automatic decision. It checks whether a simulated human decision was configured. If not, it marks the claim as HUMAN_REVIEW_REQUIRED.
If there is no conflict, the system looks at the more confident reviewer. If that confidence is high enough, the decision can move automatically. If not, the claim still goes to the checkpoint.
The logic is simple, but the governance meaning is strong: agreement alone is not enough. Confidence matters too.
Step 6: Appending the audit log.
def append_audit(decision: dict) -> None:
with audit_path().open("a", encoding="utf-8") as file:
file.write(json.dumps(decision) + "\n")What This Code Is Actually Doing
This function writes the decision to the audit log.
The file opens in append mode, which means the new decision is added to the end without deleting the older ones. Each decision becomes one JSON line.
Think of this like a claims ledger. Every claim decision gets its own row. Later, a reviewer can inspect the sequence of decisions and see exactly what the system did.
Step 7: Processing one claim.
def process_claim(claim: dict) -> dict:
print(f"\n[CLAIM] {claim['claim_id']}: ${claim['amount']:,}")
reviews = [review_claim(claim, name, perspective) for name, perspective in REVIEWERS]
for review in reviews:
print(f"{review['reviewer']}: {review['recommendation']} ({review['confidence']:.2f})")
decision = decide(claim, reviews)
append_audit(decision)
print(f"Final: {decision['final_decision']} via {decision['decided_by']}")
return decisionWhat This Code Is Actually Doing
This function runs the full workflow for one claim.
It prints the claim, sends it to both reviewers, prints each reviewer’s recommendation and confidence, decides whether the claim can move automatically or needs a checkpoint, writes the audit record, and returns the decision.
This is the whole system in miniature: review twice, compare, route, log.
Errors I Hit During This Build
Malformed JSON from model output. Different providers may format JSON differently. Some return clean JSON. Some wrap it in code fences. The parser strips markdown wrappers before calling json.loads, which makes the output more consistent across providers.
Low-confidence or invalid model output. If the model returns an invalid recommendation, the parser defaults the recommendation to REJECT. If the confidence score is missing or outside the allowed range, the parser forces it into the valid 0.0 to 1.0 range.
Blocking human input removed. The older flow asked a human to type APPROVE or REJECT directly into the terminal. That is awkward for a repeatable demo because the script can hang waiting for input. The current repo uses HUMAN_REVIEW_REQUIRED as the checkpoint output and allows a simulated override through HUMAN_REVIEW_DECISION.
Dependency weight removed. The current build has no Python package dependencies. It uses the standard library for environment loading, HTTP requests, JSON parsing, timestamps, and logging.
What This Means
Every organization deploying AI for decisions that affect people or money eventually faces the same question: what happens when the AI is wrong?
The standard answer is “put a human in the loop.” But that answer is incomplete. A serious system has to define when the human gets involved, what information the human receives, and what the system records afterward.
The conflict resolution pattern answers those questions structurally. The human checkpoint appears when the reviewers disagree, when confidence differs too much, or when the aligned review is not confident enough for automatic routing. The human is not being asked to start from scratch. They are being handed a specific disagreement or uncertainty signal.
The architecture is also a cost-control mechanism. Automatic decisions are fast and cheap. Human review is slower and more expensive. A well-calibrated conflict threshold helps reserve human attention for cases where it actually adds value.
For insurance claims, loan reviews, vendor approvals, hiring screens, compliance triage, or any workflow where a bad automated decision can create real harm, that distinction matters. The goal is not to remove human judgment. The goal is to protect human judgment for the moments where it is most needed.
A single AI answer can sound confident. Two independent reviews plus a checkpoint gives you something stronger: a decision path you can inspect.
Tools Used
- LiteLLM (MIT): github.com/BerriAI/litellm
- NosisTech Agent Engineering (MIT): github.com/nosistech/agent-engineering
(c) 2026 NosisTech LLC. Licensed under CC BY 4.0. Use freely, just credit us.