How This Started — Not with a Plan
I didn’t sit down one day and decide to build a local AI stack.
It happened the way most things in my work happen: I was in the middle of something, hit a wall, and started looking for a way around it.
The wall in this case was simple, I was spending real money on API credits and cloud model subscriptions, I was working late at night in Honduras where electricity and internet are not always reliable, and I was starting to get a little uncomfortable sending client-adjacent thinking to a cloud endpoint I don’t control.
So I bought an Acer Predator laptop with an RTX 5070 Ti and 32GB of DDR5 RAM because I needed a machine that could keep up with everything I do and somewhere in the middle of setting it up I noticed Ollama was a thing. I pulled a model to try it. Then another. Then I spent a weekend getting Open WebUI running locally. I did not fully understand what I was doing at first. I just knew it worked, and that felt like enough.
Weeks later, after the setup had become part of my daily workflow, I went back and actually read about the architecture, what MoE models are, why Qwen3 30B-A3B runs so fast despite its nominal size, how quantization affects quality. That is my process. Try it, use it, make it work. Then understand it. Then decide if it belongs in something I build for myself or a client.
This article is that post-fact research. It is also a comparison I wish someone had written for me before I started, an honest article about what local models can and cannot do, and practical about how to close the gaps.
The Setup
On the local side, I am running Ollama with Open WebUI on the Predator. The models I use most are:
- Qwen3 30B-A3B (q4_k_m) — this is the main one. 41 tokens per second on my hardware, which is fast enough that it doesn’t feel like waiting.
- Gemma 4 26B (Q4_K_M) — Google’s model. Slower at around 26 tokens per second, but it supports image input, which matters for some of my work.
- nomic-embed-text — an embedding model for when I need to do RAG locally.
On the cloud side, I use Claude Sonnet 4.6 most of the time, with occasional use of Opus for complex strategy work. I have also tested Gemini Pro 3.1 in certain situations, particularly for long-context document work.
What Is Qwen3 30B-A3B, Actually
When I first pulled this model, I thought I was running a 30-billion-parameter model. I was confused because of my laptop’s specifications. The 30B refers to the total parameter count, but the A3B part is what changes everything: this is a Mixture-of-Experts (MoE) architecture, meaning only about 3 billion parameters are active for any given token. The rest are dormant.
Think of it like a large team where only a few specialists are ever working on your specific problem at once. You get the depth of a big team’s knowledge with the speed of a small one. That is why a nominally 30B model runs at 41 tokens per second on my GPU when a dense 26B model runs at 26. The math seems backwards until you understand the architecture.
What does this mean for quality? Qwen3 30B-A3B has knowledge depth well above what you would expect from a 3B model — it has seen the training data of a 30B model. But for any given reasoning task, it activates less compute than a dense model of comparable size. In practice: excellent on many tasks, occasionally thin on tasks requiring deep sustained multi-step reasoning across a very long output.
💡 The /think flag
Qwen3 has a built-in chain-of-thought mode. If you add /think to the start of your prompt in Open WebUI, the model reasons step-by-step before writing its answer. For complex templates, legal distinctions, or multi-variable decisions, this dramatically improves output quality. It is the single most useful thing to know about this model.
Comparison
This table reflects real tasks I run repeatedly — not benchmarks. Everything here is based on actual use.
| Capability | Qwen3 30B-A3B (local) | Claude Sonnet 4.6 (cloud) |
|---|---|---|
| Multi-rule template following | Drifts when holding 10+ rules simultaneously — drops one or two | Reliable across 12+ rules in a single output |
| Long-form document coherence | Starts losing thread past ~4,000 tokens | Consistent across 30,000+ tokens |
| Strategic multi-variable reasoning | Good for single-domain questions, misses cross-domain edge cases | Connects legal, financial, and technical threads reliably |
| Persistent memory across sessions | None — every session starts blank | Full memory across all conversations |
| Tool use (web, calendar, files) | None natively — requires additional setup | Native — web search, Google Calendar, Drive, DOCX/PPTX creation |
| Privacy / data sovereignty | Complete — nothing leaves the machine | Cloud-based — data goes to Anthropic’s servers |
| Spanish-language fluency | Excellent — trained on substantial Spanish data | Very good |
| Code generation and debugging | Strong — especially Python and JS | Strong — with tool access for live testing |
| Cost at scale | Zero marginal cost after hardware — unlimited usage | Per-token pricing — adds up on heavy workloads |
| Offline / unreliable internet | Works fully offline | Requires internet connection |
Where Does Gemini Pro 3.1 Fit?
Gemini Pro 3.1 is legitimately strong at two things: very long document input (its context window is enormous) and structured data extraction from PDFs and complex documents. If I were regularly processing 100-page regulatory filings or reviewing stacks of client contracts, Gemini would be a serious option alongside Claude.
In my current workflow, I use it occasionally for comparison when I want a second opinion on a governance analysis, or when I have a document that pushes the limits of what Sonnet can hold in context. It is not a daily tool for me, but it earns its place in specific situations.
When Does Opus Make Sense?
Claude Opus is expensive relative to Sonnet, and most of the time Sonnet is enough. Where Opus earns its cost is on genuinely complex strategy work, the kind of reasoning that requires holding a large number of interacting variables and producing an output you will actually hand to a client or use to make a significant decision.
The Real Pitfalls – And How to Partially Fix Them
None of the gaps below are permanent. Some are fully solvable today. Some require a bit of setup. A few are genuinely hard and the right answer is to keep using Claude and Gemini for those specific tasks rather than patching around them indefinitely.
Pitfall 1 – No Memory
This is the biggest one. Every time I open Open WebUI and start a new chat with Qwen3, the model has no idea who I am, what I do, what my taste, preferences, rules are, or what my approach for anything. Early on I just re-explained things every session and paid the cost in time. Eventually I got tired of it and solved it properly.
✅ The Fix — System Prompt in Open WebUI
Go to Settings → Models → Edit your Qwen3 model → paste a master system prompt. This loads every session automatically. Mine includes my identity, rules, taste, preferences, and general information. One-time setup. Permanent benefit. This is the single highest-leverage change you can make to a local model setup.
A backup approach is keeping a context.md file on your desktop. Paste it at the start of any session where full context matters. It takes 30 seconds and removes a lot of frustration.
Pitfall 2 – Template Drift on Complex Outputs
In a lot of projects I have 12 simultaneous rules. Qwen3 consistently drops one or two of these. It is not random — it tends to drop rules that come later in the instruction list, which tells me it is weighting earlier instructions more heavily.
⚙️ The Fix — Numbered Checklist + /think
Convert your template from prose instructions to a numbered checklist in the system prompt. Then use /think at the start of any prompt where the template applies. The model reasons through each rule explicitly before writing. This cuts drift significantly. One-shot examples in the system prompt — showing a complete correct output, works even better than abstract rules alone.
Pitfall 3 – Long-Document Coherence
When I tried to use Qwen3 to generate a full study guide in one pass, the model would start strong and lose structural coherence around the 4,000 token mark. Sections would start repeating themes, the internal logic would drift, and the document would need substantial editing to be usable.
⚙️ The Fix — Section-by-Section Generation
Generate one section at a time with explicit handoff context: “We have covered sections 1–3. Section 3 concluded with X. Now write section 4, which should cover Y.” Qwen3 holds quality reliably at the section level. The assembly overhead is maybe 20% more work than a single-pass generation, but the output quality is dramatically better. For documents that genuinely need to be produced in one pass, Claude is the right tool.
Pitfall 4 – No Tools
This one surprised me the most. I had gotten used to asking Claude to pull up my Google Calendar, check nosistech.com live, search for current pricing, or create a downloadable DOCX file. Qwen3 locally cannot do any of this out of the box. It is a language model in a box — it knows a lot, but it cannot act on the world.
- Web search: Open WebUI has built-in web search support. Go to Settings → Web Search → enable DuckDuckGo or SearXNG. Free, no API key needed, works immediately.
- Document creation: Use a two-step workflow. Qwen3 generates the draft (fast, private). Claude applies the final template and produces the DOCX. Best of both worlds.
- Calendar and file access: Not solvable cleanly today for local models. Keep these tasks in Claude.
How I Actually Divide the Work Now
After a few months of running both stacks in parallel, here is where I have landed:
🖥️ Use Qwen3 locally for
First drafts of documents and blog content · Private research you don’t want in the cloud · Spanish-language content drafts · Fast brainstorming when you need 20 variations quickly · Offline work when internet is unreliable · Processing sensitive documents that should not leave the machine
☁️ Use Claude for
Final polished output with templates applied · Long-form documents (study guides, governance frameworks, roadmaps) · Strategic multi-variable decisions · Anything with legal or financial stakes · Calendar, web search, DOCX/PPTX creation · Tasks that require your full context without setup overhead
These are not permanent categories. As the system prompt, knowledge base, and eventually the n8n automation layer come together, the local stack will be able to handle more. The goal is not to replace the cloud stack — that is not the point. The goal is a capable private compute layer that handles the right subset of work and gradually expands what it can do.
One More Model Worth Mentioning – Gemma 4 26B
I also have Gemma 4 26B running locally, and it deserves a note because it has one capability Qwen3 does not: image input. I can drop a screenshot, a certificate image, a diagram, or a chart into Gemma and ask questions about it. For processing cert badge screenshots or reviewing visual content without sending images to a cloud, this matters.
Gemma runs at about 26 tokens per second on my hardware — slower than Qwen3 but not painfully so. I think of them as complementary: Qwen3 for text-heavy work, Gemma when I need to process an image locally.
Quick Reference – Gaps and Fixes
| Gap | The Problem | The Fix |
|---|---|---|
| No memory | Every session starts blank — no context, no rules, no positioning | Master system prompt in Open WebUI — 15 minutes to set up, permanent benefit |
| Template drift | Drops rules on complex 10+ rule templates | Numbered checklist in system prompt + /think flag before template-heavy prompts |
| Long documents | Quality degrades past ~4,000 tokens | Section-by-section generation with explicit handoff context |
| No web search | Cannot look up current information | Enable DuckDuckGo/SearXNG in Open WebUI Settings → Web Search — free and immediate |
| No file creation | Cannot produce DOCX, PPTX, or PDF files | Two-step workflow: Qwen3 drafts → Claude applies template and creates file |
| No calendar access | Cannot check or set calendar events | Keep calendar tasks in Claude — not fixable locally in the near term |
| Generic copy | Without brand context, output reads as boilerplate | Brand voice block in system prompt: target client, positioning angle, tone rules |
What This Has Actually Taught Me
Running local models has made me better at using cloud models too. When you work with a model that has no memory, no tools, and no context about you, you get very precise very fast about what you actually need from a prompt. You stop relying on the model to fill in the gaps from prior conversation. You write cleaner instructions.
I also understand the architecture much better now than I did six months ago. MoE versus dense models, quantization tradeoffs, context window behavior, RAG versus fine-tuning; these were abstract concepts I knew in theory. Running models locally made them concrete. When you see a 30B model running faster than a 26B model and have to explain why, you actually learn what A3B means.
The last thing I have learned is that the right question is not which model is better, it is which model is right for this specific task, this specific constraint, and this specific moment. Sometimes the answer is Qwen3 because I am on a plane. Sometimes it is Gemma because I need to look at an image privately. Sometimes it is Claude Sonnet because I need my full context and a downloadable DOCX in five minutes. And very occasionally it is Opus because I am doing something that actually requires that level of reasoning.
Building the local stack did not replace the cloud stack. It extended what I can do. That was worth every hour of setup.