What This Agent Does
This agent watches fictional support metrics and decides whether the support workflow is healthy. If the metrics are above target, it does nothing. If the metrics fall below target, it asks the configured model for one safe prompt-level improvement.
The important boundary is deliberate. The agent is designed to propose text-level prompt changes only. It does not run generated code, edit its own files, touch production systems, connect to databases, or deploy anything.
The Control Room
The clearest way to think about this build is an operations desk.
One part of the desk receives signals. Another part compares those signals against the scoreboard. A planner proposes a change. A gate decides whether that change is safe enough to record. The learning layer writes down the approved version.
That is the whole architecture. Not magic. Not runaway autonomy. A controlled feedback loop.
The point is not to make the agent powerful by giving it more access. The point is to make improvement reviewable. Every adaptation has a reason, a confidence score, a rollback-safety flag, and a version number.
Vocabulary You Need
KPI means key performance indicator. In this build, the KPIs are simple numbers like task completion rate, recovery rate, latency, and user satisfaction.
A feedback loop is a cycle where the system observes results, evaluates them, and adjusts future behavior. This agent uses a manual, one-run loop for demonstration.
A hypothesis is a proposed improvement. Here, the hypothesis is a structured suggestion from the model, not executable code.
A rollback-safe change is a change that can be undone cleanly. This demo treats prompt-level updates as safer than infrastructure or execution changes.
A human approval gate is a checkpoint that blocks risky changes before they become part of the system’s recorded behavior.
What This Is Not
This is not a production self-improving agent. It does not maintain a persistent database, route live users, run A/B tests, or deploy updated prompts.
It is not a security, compliance, or operational assurance system. The examples are fictional, and the approval logic is intentionally small so the pattern can be inspected.
It is also not an agent that rewrites its own Python source code. I kept that boundary on purpose. The rebuild demonstrates controlled adaptation, not uncontrolled self-modification.
Why I Built This
I find value in the idea that an agent brings evidence to the control desk and proposes a change small enough to review.
That is the standard I wanted to set here: improvement with boundaries, speed with review, and adaptation that leaves a trail.
What Actually Happened
What happened: The loop worked, but the providers behaved differently around structured JSON. One provider first wrapped or shaped output in a way my parser rejected, while another proposed changes that drifted beyond the safe prompt-only boundary.
What broke: The first external draft was not valid Python in pasted form. It had a missing comment marker, bare separator lines, a broken entrypoint guard, and an invalid requirements comment.
What surprised me: The best fix was not to trust the planner more. The best fix was to narrow the planner contract and add a safe local repair path when the provider drifted.
What I would change next: I would add a small persistent adaptation registry. That would let the agent compare prompt versions across runs instead of only inside one execution.
Why this matters: Self-improvement without memory is just a suggestion box. Self-improvement without approval is operational risk. The useful middle is a governed loop.
The Architecture: How It Works
The agent runs three fictional scenarios.
Northstar Support has weak metrics, so it asks the model for a prompt update. Harbor Helpdesk is healthy, so it skips the planner entirely. Ridgeway Support is restricted, so it blocks the adaptation path and routes the scenario to manual review.
The sequence is simple:
- Load local configuration.
- Collect fictional feedback.
- Evaluate metrics against local thresholds.
- Ask the model for one safe prompt update.
- Parse and validate the model response.
- Approve only rollback-safe, high-confidence proposals.
- Record approved adaptations as versioned in-memory records.
The model helps with the hypothesis. The local code owns the gate.
What to Watch When You Run It
The first thing to watch is the health label. Harbor Helpdesk prints HEALTHY, then skips model planning. That proves the agent is not calling the model when no adaptation is needed.
The second thing to watch is Northstar Support. It falls below target, receives one prompt-level adaptation, and records version v1.
The third thing to watch is Ridgeway Support. It also falls below target, but the scenario is restricted, so the agent rejects the change and applies nothing.
The provider difference is also useful. In my test run, both configured providers reached the same architectural outcome, even though their proposed text differed. That is exactly why the local approval gate matters.
The Build: Step by Step
REQUIRED_ENV = ["LITELLM_BASE_URL", "MODEL_NAME", "LITELLM_API_KEY"]
ALLOWED_TYPES = {"prompt_update", "threshold_adjustment", "retrieval_strategy", "tool_reordering"}
THRESHOLDS = {
"task_completion_rate": ("min", 0.80),
"error_recovery_ratio": ("min", 0.85),
"latency_p95": ("max", 3.0),
"user_satisfaction_index": ("min", 4.0),
}What This Code Is Actually Doing
This is the scoreboard and the safety boundary. The environment variables tell the script where the LiteLLM proxy is and which model to use. The thresholds define what “healthy” means for this demo. Latency is marked as a maximum because lower latency is better, while the other metrics are minimum targets.
def evaluate_metrics(metrics):
"""Compare observed metrics against the local KPI thresholds."""
scores, below = {}, []
for metric, value in metrics.items():
direction, target = THRESHOLDS[metric]
passed = value >= target if direction == "min" else value <= target
scores[metric] = {"observed": value, "target": target, "passed": passed}
if not passed:
below.append(metric)
return {"scores": scores, "below_target": below, "overall_health": "HEALTHY" if not below else "NEEDS_IMPROVEMENT"}What This Code Is Actually Doing
This is the critic. It compares every observed metric against the local target and builds a list of weak spots. If nothing is below target, the agent marks the scenario as healthy. If one or more metrics miss the target, the agent moves into improvement planning.
SYSTEM_PROMPT = """Return one JSON object only with exactly one item in a hypotheses list.
Do not use Markdown fences. The first character must be { and the last character must be }.
Each hypothesis needs source_signal, adaptation_type, proposed_change, confidence, evidence_count, rollback_safe.
The adaptation_type must be prompt_update.
The confidence must be a number such as 0.82.
The evidence_count must be an integer.
The rollback_safe value must be true.
The proposed_change must start with: Update the support prompt to
Propose only a text-level prompt change.
Never include executable code, file writes, cache changes, threshold changes, shell commands, deployment steps, background jobs, or real customer data."""What This Code Is Actually Doing
This is the planner contract. The model is asked for exactly one improvement, and the improvement has to stay inside a prompt-update boundary. This matters because a self-improving system can become dangerous if the planner starts proposing background jobs, shell commands, deployment steps, or infrastructure changes.
def parse_planner_response(text):
"""Parse and validate planner JSON into usable hypotheses."""
try:
start, end = text.find("{"), text.rfind("}")
if start == -1 or end == -1 or end <= start:
raise ValueError
hypotheses = json.loads(text[start:end + 1]).get("hypotheses", [])
except (json.JSONDecodeError, AttributeError, ValueError):
return [], "Planner returned invalid JSON."What This Code Is Actually Doing
This is the first part of the parser. It does not assume the provider returns perfect JSON. It looks for the first JSON-shaped object and tries to parse only that section. This was added because one provider response format did not initially pass strict parsing.
risky_words = ["cache", "background", "threshold", "deploy", "database", "file", "shell", "execute", "api call"]
change = str(item["proposed_change"]).lower()
if item["adaptation_type"] not in ALLOWED_TYPES or any(word in change for word in risky_words):
continue
if 0 <= item["confidence"] <= 1 and isinstance(item["rollback_safe"], bool):
valid.append(item)What This Code Is Actually Doing
This is the second part of the parser. It rejects planner output that drifts beyond the allowed adaptation boundary. In the real test run, this mattered because one provider proposed operational changes that sounded useful but did not fit the safe educational scope.
def review_hypotheses(hypotheses, risk):
"""Approve only rollback-safe hypotheses with enough confidence."""
decisions = []
for item in hypotheses:
if risk == "restricted":
approved, reason = False, "Rejected because restricted scenarios require manual review."
elif not item["rollback_safe"]:
approved, reason = False, "Rejected because rollback_safe is false."
elif item["confidence"] < 0.70:
approved, reason = False, "Rejected because confidence is below 0.70."
else:
approved, reason = True, "Approved because confidence and rollback checks passed."
decisions.append({"hypothesis": item, "approved": approved, "reason": reason})
return decisionsWhat This Code Is Actually Doing
This is the approval gate. The model can propose, but the local code decides. Restricted scenarios are blocked, low-confidence proposals are blocked, and anything not marked rollback-safe is blocked. This is the difference between a self-improving demo and a controlled adaptation loop.
if scenario["risk"] == "restricted":
hypotheses = [{"source_signal": feedback[0], "adaptation_type": "prompt_update", "proposed_change": "Pause improvement and route this restricted scenario to manual review.", "confidence": 0.40, "evidence_count": len(feedback), "rollback_safe": False}]
else:
raw = request_hypothesis(client, model_name, scenario, evaluation, feedback)
hypotheses, blocked = parse_planner_response(raw)
if blocked:
print("Planner repair: " + blocked + " Replaced with a safe prompt-only adaptation.")
hypotheses = [{"source_signal": feedback[0], "adaptation_type": "prompt_update", "proposed_change": "Update the support prompt to ask one clarifying question before answering ambiguous support requests.", "confidence": 0.72, "evidence_count": len(feedback), "rollback_safe": True}]What This Code Is Actually Doing
This is where the build became reliable across providers. Restricted scenarios do not go to the model for approval. Standard scenarios can use the model, but if the planner output fails validation, the script replaces it with a conservative prompt-only repair. The agent keeps moving, but it stays inside the safe lane.
Errors I Hit During This Build
The first provider test blocked Northstar because the planner response did not pass strict JSON parsing. I changed the parser to extract the JSON object from wrapped responses and tightened the system prompt.
A later provider test proposed useful-sounding changes that went beyond the safe scope, including caching and threshold behavior. I added validation for risky terms and a safe prompt-only repair path for standard scenarios.
The Ridgeway path needed to be deterministic. I moved restricted scenarios into a local review path so the provider cannot accidentally approve a risky improvement.
Why This Matters
The market is going to keep pushing toward agents that adapt. That direction is real. The question is whether the adaptation is governed.
This build shows the minimum architecture I trust for the concept: evidence first, model second, approval third, version record last.
That pattern matters because every business wants faster improvement, but serious teams also need boundaries. A prompt update can be useful. A model quietly redefining thresholds, adding background systems, or changing execution behavior is a different risk category.
The power move is not slowing down. The power move is building the control layer that lets speed survive contact with reality.
Tools Used
LiteLLM (MIT): https://github.com/BerriAI/litellm
OpenAI Python SDK (Apache 2.0): https://github.com/openai/openai-python
NosisTech Agent Engineering (MIT): https://github.com/nosistech/agent-engineering
(c) 2026 NosisTech LLC. Licensed under CC BY 4.0.
Use freely, just credit us.