My health, personal, or financial data must never leave my machines, and every AI component I run stays a dependency on a vendor I do not control. Those two facts together set how I route work between a local model and the cloud. This is the ground behind Sovereign & local AI. Self-hosting the model is the response, chosen for privacy, cost predictability, and freedom from rate limits.
TL;DR
Local-first by default across every profile, cloud narrowed to specifically-scoped profiles, the local tier enforced on sensitive tasks, and the whole thing sized to one resident model on a 24 GB machine.
The problem, and why the obvious answer lost
The constraint is a boundary, not a preference: health, personal, or financial data must never leave the machine, and no cloud browser is in the picture. My default pointed the other way.
The obvious answer for a stack this size is a cloud-default with a few local overrides: send everything to a hosted model and carve out the exceptions. I named that option, ran it as my starting position, and rejected it, because it fails in the wrong direction. On a cloud-default stack the sensitive task is the exception, and an exception is something I have to remember. I wanted the safe path to be the one that runs when nothing is remembered.
The decision: local-first by default, cloud only where scoped
The decision inverted the default. I set HERMES_PROVIDER_OVERRIDE=custom:ollama-local in ~/.zshrc, so every profile points at the Ollama daemon on the machine before any per-profile config is read. Cloud is kept, but only as a named exception: the Project Mining Team profiles carry Cortecs as their primary and OpenRouter as their mirror, through their own profiles/*/config.yaml.
That kept exception is what makes this a judgement and not a doctrine. A local-first default with no cloud anywhere would be an absolute; a default with one scoped, named exception is a decision the data can argue with. The split follows the material: routes that touch health, personal, or financial data run on the local tier, and the profiles that mine public material keep a hosted model where it earns its place.
How the tier is enforced
The override does most of the work by removing a choice. For Makoto and Kodoku, the profiles that hold that data, there is nothing to switch to: the environment forces the local daemon, so the privacy path is the only path that loads. /health mode then enforces the local tier on any task that touches health, personal, or financial data, automatically. I do not pick the tier for those tasks; the guardrail does, because a rule I have to recall under time pressure is a rule I will eventually forget.
The cloud exception has the same shape from the other side. It is set per profile in profiles/*/config.yaml, not held as a global default, so widening it means editing a named file rather than drifting back to the hosted path everywhere.
What the local tier costs to run
Running the model myself has a hard budget, and the budget is the machine I already own, not a bill I can scale. The MacBook Pro M4 Pro carries 24 GB of unified memory, and with one model resident at a time (OLLAMA_MAX_LOADED_MODELS=1), about 17 GB is usable for weights plus the key-value (KV) cache after macOS overhead. Context costs memory on top of the weights, roughly 0.5 GB per extra 16K tokens.
That budget sets the size bands I choose from:
- 4B for classification, formatting, and single-shot transforms, not agent work.
- 7–9B for fast completions, quality assurance (QA), and tool-calling.
- 12–14B as the daily-driver sweet spot.
- 26–32B dense runs solo, with swap under long context.
Past roughly 20 GB of weights, nothing fits. The sharpest consequence is that multi-agent work runs sequentially: one model resident, unload, run the next. An orchestrator and a sub-agent both resident does not fit at the 12B tier and up. Which hosted model serves the scoped cloud exception is a decision of its own.
What broke: q8_0 with an -mlx build
The quantization posture is what made the failure worth chasing. Weights are quantized to Q4_K_M, a mixed precision that holds the critical layers higher than a uniform low-bit build would, and the daemon-global cache is set to OLLAMA_KV_CACHE_TYPE=q8_0, which roughly halves cache memory at a quality cost too small to measure on GGUF weights. That setting is daemon-global: one value for every model on the machine.
The failure came from pairing it with a build that could not live with it. Combining OLLAMA_KV_CACHE_TYPE=q8_0 with an Apple MLX -mlx runner build drove unbounded memory growth as context accumulated. I watched it climb from 8 GB to 24 GB in a single session. The fix was to stop carrying both: the -mlx builds are parked, none on disk, and the daemon stays GGUF-only so q8_0 stands.
What this does not solve
Local-first is a judgement I keep re-deciding, not a doctrine I apply. The scoped cloud exception and the parked -mlx build are the guardrail: the split stays open to argument, which is exactly what keeps it a judgement rather than a rule. Local-first answers where a task runs, and only that.
It does remove one class of risk. A local tier with no live vendor in the loop has no vendor to cut it off, and that is the vector the cutoff-survival test grades every dependency by. Everything else is another register, linked rather than re-derived here. Where the sensitive path lands fully on the device is the dictation stack, where inference never leaves the machine. Moving mail, docs, and calendar out of the cloud is its own method, and so is the deterministic two-stage render, the routing of each task by its shape, and the boundary the automation account runs behind.
Discuss the pattern in your context
The pattern travels: choose the default for the data, name the exception out loud, and size the local tier to what one machine will hold. If this is the shape of your problem, let’s talk.