Running local LLMs on a Mac Mini M4 (32GB) for Hermes agent

I use Hermes daily with cloud models - Kimi K2.6, Minimax M3, GLM 5.2 via Opencode Zen or Ollama Cloud. For certain tasks (personal or financial files) I wanted a local model instead, so nothing gets relayed through external routing/inference services (with unknown privacy policies).

This is a log of the models and backends I tried while getting Hermes Agent running locally on a base Mac Mini M4.

Hardware

Mac Mini M4 base model, 32GB RAM (120GB/s memory bandwidth). Hermes runs in its own Ubuntu VM on Proxmox.

Ollama and Llama 3.1

ollama run llama3.1:8b

Chatting directly with the Ollama desktop app got an instant reply. Hermes took 6 minutes to respond to a plain “Hello”.

Hermes injects around 20K tokens of tool schemas and system prompt on every call, which this model was too slow to prefill.

llama.cpp and Gemma-4-12B-it-GGUF:Q8_0

llama serve \
	--host :: --port 8080 \
	-c 65536 \
	--cache-type-k q8_0 --cache-type-v q8_0 \
	-fa on \
	-np 1 \
	-ngl 99 \
	--hf-repo unsloth/gemma-4-12b-it-Q8_0.gguf
  • Context limit: 64k - set via -c 65536, kept the same for every model/backend below.
  • Flash attention:
    • Without -fa: prefill 26–53 tok/s.
    • With -fa on: prefill 100–170 tok/s.
  • Speed: decode 6–8 tok/s.
    • This is a dense 12B model, so it’s close to the real ceiling for 120GB/s bandwidth (~9.5 tok/s is the theoretical maximum).
    • I tried 4 concurrent request slots vs. a single slot (n_slots=4 default) - no difference to decode speed.
  • Verdict: Worked well with no stalling, and tool calling worked great. It was just slow on my M4 base model.

MLX-LM and Qwen3.6-35B-A3B-4bit

mlx_lm.server \
	--host 0.0.0.0 --port 8000 \
	--model mlx-community/Qwen3.6-35B-A3B-4bit

I tried a MoE model to use bandwidth more efficiently (~3.8B active params/token vs. Gemma’s dense 12B).

It was much faster. Tool calling worked OK, though a couple of times the model appeared to stop responding until I prompted again (5+ min gaps).

During one longer session, context grew to ~46K tokens and mlx_lm.server crashed the whole Mac with a kernel panic (IOGPUGroupMemory, GPU out of memory) rather than failing gracefully.

panic(cpu 7 caller 0xfffffe00517057e8): "IOGPUGroupMemory::remove_memory_object() memory object not found" @IOGPUGroupMemory.cpp:323
  • Speed: 40 tok/s decode.
  • Verdict: This was the fastest model by far, but tool calling wasn’t stable enough to trust from a Signal chat session. Not sure yet if there’s a CLI option to handle the memory-related crash more gracefully.

llama.cpp and Qwen3.6-35B-A3B-GGUF:UD-Q4_K_XL

This is the one I’ve settled on for now ✅

I tried Q5_K_XL first (unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q5_K_XL) but it was too big for the machine. Insufficient Memory (kIOGPUCommandBufferCallbackErrorOutOfMemory) on load, with or without -ngl 99. For reference, Q5_K_XL is 26.6GB and Q4_K_XL is 22.4GB.

Q4 quant worked though:

llama serve \
	--host :: --port 8080 \
	-c 65536 \
	--cache-type-k q8_0 --cache-type-v q8_0 \
	-fa on \
	-np 1 \
	--no-mmap \
	--hf-repo unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_XL
  • Speed:
    • prefill starts 400 tok/s, settles at 200 tok/s.
    • decode 15–25 tok/s - which is quite usable 👍.
  • Stability: multiple long debugging sessions (8+ sequential tool calls - grep, reading files, tracing a fallback chain, web_search), no stalls or crashes.
  • Verdict: Best I’ve tried so far. 2–3x faster than the Gemma setup and more stable. llama.cpp has the option to set a hard context ceiling too (with -c) which mlx-lm.server currently doesn’t have.

Thoughts

  • On the M4 base model’s 120GB/s bandwidth, MoE > dense.
  • Settled on llama.cpp + Qwen3.6-35B-A3B-GGUF Q4_K_XL.
  • As better MoE models keep landing - it feels like only a matter of time before something in this size class behaves a lot better on not-flagship hardware.

References