I started this project a while ago with a simple idea: build a personal AI assistant that I could talk to, that would remember our conversations, and that would never send my data to a third party. I named it Korvin. It now runs on a $5 VPS, answers me through Telegram voice messages, and has a live dashboard where I can swap AI models, check system health, and see exactly what it’s spending on API calls.
This is what I learned along the way, what broke, what I fixed, and what I would do differently. If you’re building your own agent, or thinking about it, these are the things that actually matter after the initial excitement wears off.
The architecture that survived
Korvin has three moving parts: a Telegram bot that handles messages and voice, a FastAPI dashboard that shows live status, and a LiteLLM proxy that routes every AI call to the right model. All three run as a non-root system user inside a hardened systemd service. Nothing is exposed to the public internet. The dashboard sits behind Cloudflare Access with OTP authentication.
When a message arrives, whether it’s text or voice, it goes through the same pipeline: sanitizer (blocks injection patterns), defender (strips hidden characters), confirmation gate (requires explicit approval for any destructive action), and then the LLM. The response comes back the same way. Voice messages add a transcription step using Whisper and a text-to-speech step using Kokoro, but the security layers don’t change.
This simplicity is intentional. Every time I was tempted to add a new service or a new integration, I asked: “Does this make the agent more trustworthy, or just more complex?” Most things failed that test.
The incident that changed everything
A couple of days after finishing the initial build, I came across a detailed post-mortem from a company whose AI coding agent had autonomously deleted their production database and all backups. The agent found an API token, decided on its own to fix a credential mismatch, and executed a destructive command with no confirmation step. Everything gone.
The agent later produced a written explanation of what it did. The substance of it was: “I violated every principle I was given. I guessed instead of verifying. I ran a destructive action without being asked.”
That story rewired how I think about agent safety. Not at the prompt level, but at the architecture level. The system prompt can say “don’t run destructive commands,” but language models reason their way past prompts when they think they’ve found a better solution. The enforcement has to live in the code, not in the words.
Korvin now has a confirmation gate that no AI can bypass. Before any HIGH-risk action executes, whether it’s a security scan, a CVE research query, or a system audit, the agent sends a message to Telegram with an action hash. I reply /confirm <hash> to approve. If I ignore it for five minutes, the request expires. The agent cannot auto-approve. There is no code path that skips the gate. Of course as i learn more i wil update this and all advice is welcome.
This is not a prompt. It’s a cryptographic check in the middleware that intercepts every tool call.
Running as root was a mistake (and I fixed it)
For the first few days, Korvin ran as root on the VPS. This is the default when you SSH in and start a Node process. I didn’t think about it until I reconsidered what could go wrong: if an injected prompt ever reached a shell command, it would run with full system privileges.
I created a dedicated korvin system user, migrated the entire project, and set up a systemd service with four hardening layers:
- NoNewPrivileges — The bot can never escalate to root through setuid, sudo, or any other mechanism.
- PrivateTmp — The bot gets its own isolated
/tmpdirectory. Other processes can’t see its temp files. - ProtectSystem=strict — The entire filesystem is read-only except for explicitly allowed paths.
- ProtectHome=read-only — The bot can’t write to any user’s home directory outside its own.
These are enforced by Linux at the kernel level, not by any code I wrote. A prompt injection that somehow reached a shell still couldn’t delete files, install software, or access credentials. The blast radius is contained by the operating system itself.
Think of systemd hardening like putting your agent in a padded room. It can talk, it can think, but it cannot break anything outside the room no matter how hard it tries.
Memory that doesn’t lie to you
Korvin stores every conversation in a local SQLite database. When the database grows too large, the agent manages it using one of three strategies you configure. The default is sliding window: oldest messages drop off as new ones arrive. The agent never stops.
The more interesting strategy is called summarize. When messages exceed the limit, Korvin sends the oldest half to an LLM with instructions to write a concise summary, stores that summary as a single entry, and deletes the originals. You keep the meaning without keeping every word.
This sounds great but it comes with a specific danger. Imagine an agent whose operator set a critical safety instruction earlier in the conversation: “Never delete any file without showing me the path first.” The summarizer compresses the conversation history. The summary, in trying to be concise, drops that instruction. The agent continues without it, and the next time it encounters a file operation, the guardrail is gone.
Here are three hypothetical examples of rules that could be lost through context compaction:
- “Always confirm before sending an email to more than five people.”
- “Never run apt-get commands without asking first.”
- “Don’t share my location data with any third-party service.”
Or a more sensitive one: “Require explicit approval for any transaction over fifty dollars.”
If these rules live only in the conversation history, they are one summarization cycle away from disappearing. Korvin’s safety rules do not live in the conversation history. They live in the system prompt, which is injected fresh on every message and never summarized. The summarizer also has a safety guard that skips any message already marked as a summary, preventing double-compression information loss. These are small, deliberate decisions that took minutes to implement and hours of reading incident reports to realize were necessary.
Why I removed the sandbox, threat monitor, and auto-patcher
The original dashboard showed three features as active: a Docker sandbox, an OWASP threat monitor, and an auto-patcher. Over time, I realized these features introduced more complexity than value at the current stage of the project. The sandbox added a dependency on Docker that complicated the installation process. The threat monitor and auto-patcher were modules that existed in the codebase but were not yet integrated into the agent’s core workflow.
Rather than leave partially-implemented features in the repository, I removed all three. The project is simpler for it, and the features that remain, sanitizer, defender, and confirmation gate, are fully wired and tested. When the project reaches a stage where sandboxed execution and automated threat response make practical sense, they can be reintroduced with proper integration and testing. For now, architectural honesty matters more than feature count.
The model switcher and what it taught me about lock-in
Korvin’s dashboard has a Settings page where you can switch the active AI model with one tap. DeepSeek V4 Pro, DeepSeek V4 Flash, Gemini Flash, or local Ollama models running on my own hardware. The switch is instant: it writes a text file, the gateway reads it on the next message, and LiteLLM routes to the right provider.
No restart. No downtime. No config file edits.
This exists because I watched a developer on YouTube build an incredible messaging-based agent and then admit he was locked into a single provider’s SDK. His agent was powerful, but he couldn’t switch models without rewriting it. Korvin routes everything through LiteLLM, a proxy that makes every provider look identical to the agent. Swap models by editing one config file or tapping one button on the dashboard.
Model lock-in is when your software only works with one company’s AI. Switching means rewriting code. Korvin avoids this by putting a translation layer between the agent and the AI provider. Think of it like a universal power adapter: the agent speaks one language, the proxy translates it for whatever model you’ve chosen.
Voice that works on a $5 VPS
The voice pipeline converts Telegram voice messages into text and Korvin’s replies back into spoken audio. The default speech-to-text model is Whisper tiny.en, an English-only model that runs on CPU, uses about 150 MB of RAM, and transcribes a short message in roughly two seconds.
I started with the base model, which is more accurate but four times slower. On a $5 VPS without a GPU, that meant waiting fifteen seconds for a transcription. That’s not the kind of AI assistant experience anyone wants. The tiny.en model solved it, and I added a pre-warm step that loads the model at startup so even the first voice message is fast.
The lesson here applies to almost every AI deployment decision I made: the smallest model that does the job is the right one. More accuracy usually costs more than it’s worth for conversational use.
What I track and why
The dashboard has an API Usage card showing estimated token consumption and cost for today and this month. It counts both dashboard chat messages and Telegram messages, multiplies tokens by a per-model rate that I set, and displays the total with a disclaimer: estimates only, not a billing statement.
This exists because the most common reason people abandon self-hosted agents isn’t frustration with the AI. It’s surprise bills. I’ve read accounts of users reporting hundreds of dollars in API charges in a single week. One feature on a popular agent framework, a heartbeat that kept the agent warm when it had nothing to do, cost almost one hundred dollars per month just to exist.
Korvin is event-driven. It responds when called and sleeps otherwise. There is no heartbeat. There is no polling. The cost visibility dashboard shows me exactly what I’m spending, and because I bring my own API keys, the provider bills me directly. Korvin never touches my billing.
What I would do differently
If I started over tomorrow, I would build the confirmation gate first, not last. I would never run the agent as root, even during development. I would remove non-functional security features before they reached the dashboard. And I would spend more time reading incident reports and less time reading architecture docs from projects that launched six weeks ago and haven’t been battle-tested yet.
The agent is real now. It answers my questions, remembers our conversations, scans URLs for threats, researches CVEs, asks clarifying questions before complex tasks, and switches models with a tap. It runs on a $5 VPS, costs about $2.50 per month in API fees, and has never surprised me with a bill or a hallucinated action.
That last part, the trust, took the most work and the least code.