What breaks when you run local LLMs for coding agents

I wanted every coding agent I use to run against a model I host myself. No hosted API, no per token bill, no code leaving my network. On paper it is a weekend project: a laptop as the client, a gaming PC with a decent GPU as the server, an OpenAI compatible endpoint in between. In practice it turned into a tour of every layer except the one I expected to fight with. Unified memory ran out and took the whole laptop down with it. Two capable models failed as agents because the inference engine could not parse their tool calls. A third got stuck repeating itself. And at least one hard freeze hit the laptop when no model was loaded on it at all. The model's raw capability was almost never the problem.

The setup

Two machines, two roles.

On the client side there is VS Code with GitHub Copilot Chat pointed at a custom endpoint, and Pi as the terminal coding agent. The server is nothing exotic, just the gaming PC I already had. 16 GB of dedicated VRAM is enough for a 24B coding model at 4 bit, and Ollama exposes it on the LAN as an OpenAI compatible API. Point every tool at http://<server-ip>:11434/v1 instead of a hosted endpoint and, in theory, nothing else changes.

The first phase looked different. The MacBook ran its own MLX server for models small enough to fit in unified memory, and that phase is where most of the trouble started.

Where MLX on the M5 Pro fell short

Unified memory is the whole appeal of Apple Silicon for local inference and also its hard limit. There is no separate VRAM pool. The GPU and CPU share the same 24 GB, and that budget has to cover macOS, the editor, the browser and the model. mlx_lm.server has the best Apple Silicon support of anything I tried, so speed was never the issue. Memory was, and so was tool call parsing, which gets its own section below.

This is roughly how I ran it, with every knob turned toward a smaller footprint:

mlx_lm.server --model <model-id> --port 8080 \
  --max-tokens 2048 \
  --decode-concurrency 1 --prompt-concurrency 1 \
  --prefill-step-size 512 \
  --prompt-cache-size 1 --prompt-cache-bytes 900000000

A 14B coding model at 4 bit, around 8 GB of weights, left plenty of headroom and became my safe fallback. Past roughly 14 GB of weights things got fragile. An agentic session keeps growing (system prompt, tool definitions, every turn so far) on top of weights that are already resident, and with the 17 GB Qwen3 Coder 30B MoE running under Pi the failures escalated in two steps. The first was a clean error:

RuntimeError: [METAL] Command buffer execution failed: Insufficient Memory
(kIOGPUCommandBufferCallbackErrorOutOfMemory)

The second was worse. The machine got hot, locked up completely and needed a hard restart, with nothing on screen to act on.

What helped, without removing the ceiling, was capping how much memory the GPU can wire, so the process fails with an error instead of freezing the machine. The value resets on reboot, and it should sit a few GB under total RAM:

sudo sysctl iogpu.wired_limit_mb=16000

On top of that I kept models small and quantized, capped the prompt cache explicitly, and started fresh agent sessions instead of letting one grow for hours. I also had an opencode config ready and never made it my daily driver. It is agent only and sends a heavy system prompt plus the full tool list on every request, which under my original theory was exactly the load that pushed the laptop over the edge.

None of this raises the ceiling. It only makes you hit it more predictably. On 24 GB of shared memory, a big model and a long agentic session compete for the same space, and sooner or later one of them loses.

Same model, different engine, different outcome

A model can be perfectly capable of agentic tool use and still fail as an agent, because the inference engine's chat template does not round trip its tool call format. From the client side this looks exactly like a weak model, and that is what makes it expensive.

Same weights, same task, and the outcome depends on which process parses the output. If your agent stalls after a tool result, or silently ignores a tool call, check chat template compatibility before blaming the model or your prompt.

The client integration has sharp edges of its own. In VS Code the custom endpoint lives in chatLanguageModels.json, and it is best to let VS Code generate it through Manage Language Models. Two details matter. The vendor has to be customendpoint, not openrouter, because the OpenRouter vendor expects OpenRouter's model list schema and mlx-lm's minimal /v1/models response fails with an invalid response format error. And the API key has to be the keychain backed ${input:...} reference VS Code creates. A plaintext key makes the whole provider group disappear without any error. Once configured, Ask mode works well with local models. Agent mode is where the memory and tool call fragility comes back.

Pi needs two compat flags for local models, supportsDeveloperRole: false and supportsReasoningEffort: false, since mlx-lm understands neither. Pi also applies file edits without a confirmation step by default, so I only run it inside a clean git repo where every change is easy to review and revert.

Getting the gaming PC right

Ollama picks up the RX 7800 XT through its ROCm backend. The first thing worth checking is that it actually runs on the GPU and has not quietly fallen back to the CPU. The startup log shows whether the GPU was detected, and ollama ps shows the GPU/CPU split of the loaded model.

The rest is unglamorous Windows plumbing. Binding to all interfaces has to be a permanent system environment variable, not a $env: variable in a PowerShell session:

[System.Environment]::SetEnvironmentVariable("OLLAMA_HOST", "0.0.0.0:11434", "Machine")

Confirm it in the startup log, which should say Listening on 0.0.0.0:11434 and not 127.0.0.1. The same goes for OLLAMA_CONTEXT_LENGTH. The log prints the VRAM based default context at startup, and if it still shows a small value like 4096 after a restart, the variable did not take effect. After any change, reopen the terminal and quit the Ollama tray app or service, because a process that is already running keeps its old environment.

Then open the port in the firewall and test from the client using the LAN IP instead of the hostname, which rules out mDNS as a source of confusing failures:

New-NetFirewallRule -DisplayName "Ollama" -Direction Inbound -Protocol TCP -LocalPort 11434 -Action Allow

curl http://<server-ip>:11434/v1/models

Pi then points at it with a plain OpenAI compatible provider in ~/.pi/agent/models.json:

"ollama": {
  "baseUrl": "http://<server-ip>:11434/v1",
  "api": "openai-completions",
  "apiKey": "ollama",
  "models": [ ... ]
}

None of this is hard. All of it looks like a model problem from the client side until you read the server log.

A model that loops instead of finishing

Devstral on Ollama got past the tool call round trip and then hit a different problem. No crash and no parsing error. Driven from Pi, the model degenerates into repeating the same output again and again. I have not closed this one yet, but these are the candidates, roughly in order of likelihood.

1. Context truncation. Even with OLLAMA_CONTEXT_LENGTH=16384, a long Pi session can outgrow the window. If Ollama silently drops the oldest turns, including the part of the prompt that anchors the task, the model loses the thread and repeats what is left. ollama ps does not show the context size directly, so the safer check is restarting ollama serve right after setting the variable and making sure no old instance is still running with Get-Process ollama.
2. A weak repeat penalty. Ollama's default is fairly soft, and Mistral family models, Devstral included, are prone to loops without a stronger one.
3. Temperature close to zero on a repetitive task, like similar boilerplate across several files. Nudging it to 0.2 to 0.4 alongside the repeat penalty usually breaks the pattern.
4. A tool call quirk. Agentic models with weak tool call closure sometimes retry the same call forever when a tool result does not match what they expected.

The first diagnostic step is still open: reproduce the loop with plain ollama run devstral:24b, no Pi and no tools. If it loops there too, it is a model or sampling problem. If it does not, the cause is in the tool calling flow.

For the sampling side, the plan is to test interactively first:

ollama run devstral:24b --verbose

/set parameter repeat_penalty 1.3
/set parameter repeat_last_n 256

Those settings only live in that REPL session. If they stop the loop, they have to be baked into a Modelfile to apply when Pi calls the model through the API:

@"
FROM devstral:24b
PARAMETER num_ctx 16384
PARAMETER repeat_penalty 1.3
PARAMETER repeat_last_n 256
"@ | Out-File -Encoding utf8 Modelfile

ollama create devstral-tuned -f Modelfile

After that, Pi's models.json points at devstral-tuned instead.

When the laptop itself is the problem

The hardest bug in this project was not a model or a config. The MacBook went down hard more than once while Pi sat open and unattended in a terminal, pointed at the Ollama server on the gaming PC and not at a local model.

My first explanation was the one from the MLX phase: a big local model exhausts unified memory and takes the machine down. The symptoms matched that pattern exactly. Hot machine, full lockup, hard restart, no panic log. The mechanism did not, because at least one of these freezes happened with no local model loaded at all.

So I stopped guessing and started ruling things out.

• Look for a kernel panic report before assuming there was one. ~/Library/Logs/DiagnosticReports/ had no panic file for any of the incidents, only unrelated app faults. A hard hang that never gets logged as a panic is a different failure class than a clean out of memory crash.
• Confirm it was an unclean stop. last reboot showed a crash marker for each session, so these were real crashes and not manual restarts. log show around each window had only routine logging, with no panic trace and no explicit shutdown cause.
• Look at what the incidents had in common. The constant was Pi left open for hours with an idle connection to a remote endpoint. That narrows the hypothesis to how the client process handles a stale connection over time, which is a very different investigation from "the model is too big".

The root cause is still unconfirmed, and the absence of any panic log also leaves a hardware level shutdown, thermal for example, on the table. The next steps are boring on purpose. Finish removing the MLX models from the Mac, so that a new freeze would rule out local models entirely. Check pmset -g thermlog for throttling events. And before leaving Pi unattended again, run a small logger in another terminal tab so the next incident comes with evidence:

while true; do date; ps aux | grep -i "[p]i" ; sleep 300; done >> ~/pi_monitor.log

That is also why the laptop is becoming a pure client. It is as much a stability decision as a resource one: fewer moving parts on the machine that keeps failing.

Where it stands now

The gaming PC is the solid part. Ollama runs on the RX 7800 XT over ROCm, is reachable on the LAN and handles Devstral's tool calls correctly. The MacBook is being stripped down to a pure client, with the MLX models on their way out. Two problems are still open: the freeze, which comes first because it is a stability issue, and the Devstral loop, parked until the client side is simplified and the freeze investigation has more signal. If tool calling stays unreliable with every local model, the fallback is aider, which does not use tool calls at all. The model writes diffs as plain text, and aider applies them and commits to git.

If you are building something similar, budget time for the plumbing and not just for the model choice. Memory headroom, tool call formats, sampling defaults and connection handling each broke something here. The model's raw capability almost never did.

References

• MLX LM
• Ollama FAQ, including OLLAMA_HOST and context length
• Ollama Modelfile reference
• aider