Mattia Peretti

Local-llm

  • Published on

    What breaks when you run local LLMs for coding agents

    aillmlocal-llmcoding-agentsself-hosted

    I moved my coding agents off hosted APIs and onto two machines I own, a MacBook Pro M5 Pro as the client and a gaming PC with an RX 7800 XT as the model server. Model capability was rarely the blocker. The blockers were unified memory limits, inference engines that mangle tool call formats, a model stuck in a loop, and a laptop that kept freezing for reasons I still have not pinned down.

  • Published on

    We just hit the local LLM tipping point

    aillmlocal-llmopen-sourceinference

    Colibrì, a ~1,300-line pure C inference engine with zero dependencies, runs the 744B-parameter GLM 5.2 MoE model on a laptop with 25 GB of RAM and no GPU by streaming int4-quantized experts from NVMe. Cold-cache throughput is 0.05 to 0.1 tokens per second, so speed is the honest limiter, but it is an existence proof that frontier-scale local inference works on commodity hardware.