Disclaimer: This article documents my personal exploration of AI governance frameworks and regulatory concepts. Nothing in this post constitutes legal advice, compliance certification, or a guarantee of regulatory conformance. AI governance requirements vary by jurisdiction, sector, and use case. Consult qualified legal and compliance professionals before making governance decisions. NosisTech LLC accepts no liability for outcomes arising from use of this content.
Every AI system eventually faces the same question: why did it decide that? Most tools today cannot answer it clearly. The system takes in data, produces an output, and the reasoning stays locked inside. That might be tolerable when an AI recommends a product. It becomes a serious business risk when AI is involved in loan review, hiring workflows, eligibility screening, or any outcome that affects someone’s financial situation. Regulators in multiple jurisdictions are increasingly focused on that gap.
What This Agent Does
This agent evaluates a synthetic small business application. You give it four data points about a business: revenue growth, debt ratio, months in business, and credit score. The agent runs those numbers through a transparent scoring model, produces a recommendation, and immediately explains how that recommendation happened.
The important word is transparent. This is not a black box model pretending to be a lender. The scoring rules are visible in the code. Each factor has a maximum number of points and a target value. The agent calculates how many points the application earned for each factor, totals the score, identifies the strongest and weakest factors, and generates one counterfactual improvement.
A counterfactual is the “what would need to change?” part of the explanation. In this build, the counterfactual does not promise approval or suggest a real lending action. It simply says which weak factor, if improved toward its target, would most improve the result.
Then the agent asks the language model to write two explanations from the same evidence. The analyst version includes the score, confidence, strongest factor, weakest factor, and counterfactual. The applicant version avoids scores, point values, probabilities, target numbers, and financial-advice language. Same facts, different audience.
Why I Built This
The standard approach to AI scoring systems is a black box wrapped in a dashboard. A score comes out, a decision gets made, and if someone asks why, the explanation is often vague: multiple factors were considered.
That is not good enough for serious systems. If a compliance team, partner, auditor, applicant, or executive asks how the system reached a conclusion, the answer needs to be visible. Not mystical. Not “the model said so.” Visible.
The earlier version of this build used heavier explainability ideas like SHAP and LIME. Those are powerful concepts, but they also create a lot of machinery for a teaching post. The simplified repo makes a sharper point: explainability starts with architecture, not with a library.
If the scoring system is transparent, the explanation can be transparent. You can see the inputs. You can see the factor weights. You can see the earned points. You can see which factor helped most, which factor hurt most, and what improvement would matter most next.
Think of it like a school report card. A parent does not want to hear, “Your child’s academic score was produced by a complex system.” They want to see the grades by subject. Math: strong. Reading: improving. Attendance: needs work. That is the level of clarity business AI systems need before they start affecting real outcomes.
What Actually Happened
What actually happened: the simplified version became more useful as a teaching tool after the custom SHAP and LIME-style classes were removed. The current agent does not pretend to be a production lending system. It shows the entire scoring path with plain arithmetic and then uses the model only to write the explanation.
What broke: the old article described a trained classifier and explainer classes that no longer exist in the repo. The code now uses a synthetic scoring table instead of training a model from sample data.
What surprised me: the demo application lands just below the example threshold. The score is close enough that the system does not present the outcome as highly certain. That is exactly the kind of result where explanation matters most. Near-threshold cases need clarity, not false confidence.
What I would change next: I would add persistence. Every explanation should eventually be written to an append-only audit log with the input, factor scores, final score, confidence, and both summaries. Explainability is stronger when the explanation survives the session.
Why this matters: an AI scoring workflow does not become trustworthy because it gives a confident answer. It becomes more reviewable when it can show the path from input to score to explanation.
The Architecture: How It Works
Think of this agent as a scoring panel with a built-in court reporter. The scoring panel evaluates the application. The court reporter writes down what happened in language different audiences can understand. Neither side replaces the other.
The agent has five layers. First, the application data layer provides the four synthetic business values. Second, the scoring layer calculates points for each factor. Third, the attribution layer identifies which factor helped most and which factor hurt most. Fourth, the counterfactual layer explains the most useful improvement area. Fifth, the summary layer asks the model to write an analyst explanation and an applicant explanation.
The language model does not calculate the score. It does not decide the recommendation. It does not invent the explanation from scratch. The code produces the scoring evidence first. The model turns that evidence into readable language.
The Core Components
The application record contains the four example values: revenue growth, debt ratio, months in business, and credit score. These are synthetic values for demonstration only. They are not real applicant data and should not be used for real lending decisions.
The feature table defines the scoring rules. Each factor has a label, a maximum point value, a target value, and a direction. Some factors are better when higher, like credit score. One factor is better when lower, debt ratio.
The scoring function turns each factor into earned points. It compares the applicant’s value against the target, limits the result between zero and one, and multiplies by the maximum points for that factor.
The explanation function totals the score, decides whether the synthetic application meets the example threshold, calculates confidence, identifies the strongest and weakest factor, and writes a counterfactual improvement.
The summary function sends the finished report to the model. The analyst version can mention scores and confidence. The applicant version is constrained to plain language and avoids turning the output into financial advice.
The Model-Agnostic Layer
Every LLM call runs through a LiteLLM-compatible endpoint. The code reads LITELLM_BASE_URL, MODEL_NAME, and LITELLM_API_KEY from .env. Changing the model happens in configuration, not in the scoring logic.
The current repo does not use the OpenAI SDK, LiteLLM Python package, NumPy, scikit-learn, SHAP, LIME, or dotenv. It uses the Python standard library to load configuration, build JSON requests, call the model endpoint, and print the output.
That keeps the architecture visible. The scoring happens in Python. The explanation writing happens in the model. Those two jobs stay separate.
The Build: Step by Step
Step 1: The synthetic application.
APPLICATION = {
"application_id": "APP-DEMO",
"revenue_growth": 18.5,
"debt_ratio": 0.72,
"months_in_business": 14.0,
"credit_score": 610.0,
}What This Code Is Actually Doing
This is the example application. It is not pulled from a database and it is not real applicant data. It is a fixed synthetic record used to demonstrate how explainability works.
Think of it like a practice case in a training manual. The point is not whether this specific business should qualify for anything. The point is to show how the agent evaluates four inputs and explains the result.
Step 2: The feature table.
FEATURES = [
("revenue_growth", "Revenue growth", 25, 25.0, True),
("debt_ratio", "Debt ratio", 25, 0.35, False),
("months_in_business", "Months in business", 20, 24.0, True),
("credit_score", "Credit score", 30, 680.0, True),
]What This Code Is Actually Doing
This is the scoring rubric. Each row tells the agent five things: which field to read, what label to show, how many points the factor is worth, what target value to compare against, and whether higher is better.
For revenue growth, higher is better. For credit score, higher is better. For months in business, higher is better. For debt ratio, lower is better.
This is like a loan review scorecard. Revenue growth might be worth 25 points. Credit score might be worth 30. The rule is not hidden in a model. It is visible in the table.
Step 3: The scoring function.
def score_feature(value: float, target: float, higher_is_better: bool) -> float:
ratio = value / target if higher_is_better else target / max(value, 0.01)
return max(0.0, min(1.0, ratio))What This Code Is Actually Doing
This function calculates how close one factor is to its target. If higher is better, the agent divides the applicant value by the target. If lower is better, like debt ratio, it divides the target by the applicant value.
The result is capped between zero and one. Capped means it cannot go below zero or above one. If a factor beats the target, it does not earn infinite points. It earns full credit for that factor.
For a beginner, think of it like a grading scale. If the assignment is worth 25 points, the student can earn up to 25. Doing extra well does not turn it into a 90-point assignment.
Step 4: The explanation engine.
def explain(application: dict) -> dict:
factors = []
total = 0.0
for field, label, max_points, target, higher_is_better in FEATURES:
earned = round(score_feature(application[field], target, higher_is_better) * max_points, 2)
total += earned
factors.append({"field": field, "label": label, "earned": earned, "max": max_points, "target": target})
strongest = max(factors, key=lambda item: item["earned"] / item["max"])
weakest = min(factors, key=lambda item: item["earned"] / item["max"])
return {
"application_id": application["application_id"],
"decision": "approve" if total >= 70 else "decline",
"score": round(total, 2),
"confidence": round(max(0.1, min(0.99, abs(total - 70) / 30)), 2),
"factors": factors,
"strongest": strongest,
"weakest": weakest,
"counterfactual": f"Improving {weakest['label']} toward {weakest['target']} would most improve the result.",
}What This Code Is Actually Doing
This is the heart of the agent. It walks through the feature table, scores each factor, adds the points, and builds the explanation record.
The strongest factor is the one that earned the highest share of its possible points. The weakest factor is the one that earned the lowest share of its possible points. That matters because a 20-point factor and a 30-point factor should be compared by percentage of possible credit, not raw points alone.
The decision threshold in this educational example is 70. If the total score is at least 70, the synthetic result is approve. If it is below 70, the synthetic result is decline. The confidence is based on distance from that threshold. A score far away from 70 is more confident. A score close to 70 is less confident.
That is a smart design choice because near-threshold cases are exactly where a human should be cautious. If the score is 69.23, the system should not behave like the answer is carved in stone.
Step 5: The analyst summary.
def summarize(report: dict, audience: str) -> str:
if audience == "applicant":
result = "met" if report["decision"] == "approve" else "did not meet"
prompt = (
"Write an applicant-friendly explanation for this educational example. "
"Do not include scores, point values, probabilities, target numbers, JSON, "
"or the words declined, approve, approved, or financing. Say it is not a "
"final lending decision or financial advice. Keep it under 100 words.\n\n"
f"Example result: {result} the example threshold\n"
f"Positive factor: {report['strongest']['label']}\n"
f"Main improvement area: {report['weakest']['label']}"
)What This Code Is Actually Doing
This part builds the applicant-facing prompt. The applicant version is intentionally careful. It avoids scores, point values, probabilities, target numbers, JSON, and certain decision words. It also tells the model to say this is not a final lending decision or financial advice.
That is not just softer wording. It is audience routing. The applicant does not need the same explanation as an internal analyst. The applicant needs plain language, no false certainty, and no implication that this educational example is an actual lending decision.
Step 6: The analyst version.
else:
prompt = (
"Write an analyst explanation for this educational synthetic example. "
"Mention score, confidence, strongest factor, weakest factor, and counterfactual. "
"Keep it under 120 words.\n\n"
+ json.dumps(report, indent=2)
)
return call_llm(prompt)What This Code Is Actually Doing
This part builds the analyst-facing prompt. The analyst version is allowed to mention the score, confidence, strongest factor, weakest factor, and counterfactual because the analyst needs the mechanics.
This is the same evidence packet, but written for a different reader. That is a major explainability lesson. A system may need one explanation for internal review and another explanation for the person affected by the outcome.
The facts should stay consistent. The language should change based on the audience.
Step 7: The model call.
def call_llm(prompt: str) -> str:
payload = json.dumps(
{
"model": os.getenv("MODEL_NAME"),
"messages": [{"role": "user", "content": prompt}],
"temperature": 0.2,
}
).encode("utf-8")
request = Request(
os.getenv("LITELLM_BASE_URL").rstrip("/") + "/chat/completions",
data=payload,
headers={
"Authorization": "Bearer " + os.getenv("LITELLM_API_KEY"),
"Content-Type": "application/json",
},
)
with urlopen(request, timeout=60) as response:
return json.loads(response.read().decode("utf-8"))["choices"][0]["message"]["content"].strip()What This Code Is Actually Doing
This function sends the explanation prompt to the configured model. The model name and endpoint come from .env, not from hardcoded values.
Temperature is set to 0.2, which keeps the output focused. This is not a creative writing task. The model is being asked to explain a fixed evidence packet.
The model does not get to invent the score. It receives the score. It does not get to decide the strongest factor. It receives the strongest factor. It does not get to decide the counterfactual. It receives the counterfactual. Its job is translation.
Step 8: Printing the full result.
def main() -> None:
load_env()
require_env()
report = explain(APPLICATION)
print("=== MODEL OUTPUT ===")
print(json.dumps(report, indent=2))
print("\n=== ANALYST SUMMARY ===")
print(summarize(report, "analyst"))
print("\n=== APPLICANT SUMMARY ===")
print(summarize(report, "applicant"))What This Code Is Actually Doing
This function runs the whole demonstration. It loads configuration, checks required values, generates the scoring report, prints the raw model output, then prints the analyst and applicant summaries.
The order matters. The structured scoring report appears before the model-written explanations. That means the reader can see the evidence first and the language second.
For explainable systems, that order is the difference between “trust the AI” and “inspect the record.”
Errors I Hit During This Build
Variable name mismatch. The older version of the article referenced environment variable names that no longer match the current repo. The simplified version standardizes on LITELLM_BASE_URL, MODEL_NAME, and LITELLM_API_KEY.
Overbuilt explainer classes. The old version described SHAP, LIME, NumPy, and a trained classifier. Those pieces were removed. The current build uses one transparent scoring model and one attribution table, which is easier to read and better aligned with the teaching goal.
Dependency weight. The simplified repo has no Python package dependencies. That removed the installation and compatibility problems that can distract from the architecture.
Audience safety. The applicant summary needed stronger constraints than the analyst summary. The current code handles that by using two different prompts from the same underlying report.
What I Would Do Differently
The biggest production gap is persistence. Every explanation this agent generates is printed to the terminal. A serious system would write every input, factor score, final score, confidence value, counterfactual, analyst summary, and applicant summary to an append-only audit log.
The second gap is policy review. The scoring table is transparent, but transparency does not automatically make a scoring policy fair, legal, or appropriate. A qualified legal and compliance team would need to review the factors, weights, thresholds, applicant language, and operational use case before anything like this touched a real workflow.
The third gap is data. This build uses one synthetic application and fixed scoring rules. A production system would need a carefully governed dataset, documented feature selection, bias testing, monitoring, and independent validation. The explanation is only as good as the scoring design behind it.
I would also add a compliance-officer audience. The current agent writes for analysts and applicants. A compliance version would be stricter: full factor table, timestamp, model name, threshold, confidence value, counterfactual, and a complete audit record formatted for review.
That is where this becomes more than a demo. Explainability plus audit history becomes governance infrastructure.
What This Means
This build demonstrates one thing: explainability is an architectural choice, not a magic feature you sprinkle on afterward. A scoring system can produce a result and a documented explanation at the same time if it is designed that way from the beginning.
The code here is a learning example. It uses synthetic data, a simple scoring table, and a fixed example application. It is not a lending system. It should not be used to score real people, real businesses, or real financial applications.
But the question it asks is the right one: can your system explain itself in a way an analyst can review, an applicant can understand, and a compliance team can audit?
If the answer is no, that is not just a technical gap. It is a business risk. The fix starts with architecture: visible inputs, visible scoring logic, visible attribution, clear counterfactuals, audience-specific explanations, and records that survive after the model finishes speaking.
Tools Used
- LiteLLM (MIT): github.com/BerriAI/litellm
- NosisTech Agent Engineering (MIT): github.com/nosistech/agent-engineering
(c) 2026 NosisTech LLC. Licensed under CC BY 4.0. Use freely, just credit us.