Local Codex on a 16 GB M5: gpt-oss-20b decoded at 0.3 tokens per second
I tried codex --oss with gpt-oss-20b on my 16 GB M5. The weights are 12.82 GiB, Metal gives the GPU 11.84 GiB, and Ollama defaults to a 4k context. Measurements, the math, and the config I ended up with.
In bf16, a decay of 0.9995 is 1.0
MLX cast the gated delta net decay to the activation dtype. In bf16 that rounds 27% of gate values to exactly 1.0, and a state that never decays is not a bug the loss curve will tell you about.
llama.cpp's BF16 attention kernel: the prefill number is the headline, the accumulator is the story
PR #30491 adds BF16 instantiations of llama.cpp's Metal tensor-API flash attention. The +44% prefill is prefill-only. The real finding is that the simdgroup BF16 kernel accumulates attention output in half, with the fp32 variant commented out.
llama.cpp's MoE expert cache: read the eligibility predicate before the benchmark table
llama.cpp PR #29887 adds --moe-cache-mib, an LRU of MoE experts in GPU memory with 1.57-2.20x decode speedups over PCIe. All three Metal buffer types report is_host = false, so on Apple Silicon the cache disables itself.
Speculative decoding on Apple GPUs was slower than not using it. The fix landed this morning.
Metal had no mat-mul kernel for a handful of rows, so llama.cpp verified speculative drafts with the mat-vec kernel. On an M3 Ultra that made DFlash2 decoding slower than serial decoding: 16.8 t/s against 32. PR #29869 takes it to 64.4.
1918 live tensors at the end of prefill: ExecuTorch's MLX backend never released a temp slot
gemma4-e2b on an iPhone 16 finished prefill holding 3.1 GB of intermediates and was killed by jetsam. The cause is the AOT memory plan, not a refcount bug, and set_cache_limit(0) moves peak memory by exactly zero.
MLX's nvfp4 prefill went 1,267 → 4,256 tok/s in two PRs. Decode went 81.9 → 81.7.
Two MLX PRs merged September 14 took nvfp4 prefill on Qwen3.6-35B-A3B from 1,267 to 4,256 tok/s on M5 Max. Decode went 81.9 to 81.7. That split is the whole lesson.
MiniCPM5-2B's 131k context is 5.25 GiB of KV cache on a phone. The config.json tells you more than the leaderboard.
Two KV heads, 42 layers, head_dim 128 — 43 KB per token. The honest window on a phone is 8k–16k, and the 130k vocab is a fixed surcharge on every token you decode.
llama.cpp's Metal 4 fast path now ships as two metallibs, and on iPhone the second one is the whole point
The silent tensor-API probe failure was a shader-compiler default. The fix disables the fast path for precompiled libraries — which is everything on iOS — so the xcframework now ships a second metallib gated on SDK 26.
Apple's 4.3x is a prefill number, and one init-time check decides whether you get it
The M5 Ultra Mac Studio landed this week. A Metal 4 capability check that fails silently can cost you 2–3x of its headline gain — and generation speed looks fine either way.
Speculative decoding is not a free 2x on Apple Silicon, and MoE makes it worse
Liquid AI shipped an on-device result with 85% draft acceptance and a 1.04x speedup. Why sparse experts and unified memory break the assumption that acceptance rate predicts throughput.
ExecuTorch's Apple Silicon path is MLX and hand-written Metal. Not Core ML, not MPS.
Meta's flagship on-device model demo routes around both delegates Apple ships in the OS. What that means if you're choosing a backend today, plus the K-quant repacking trick worth stealing.
ONNX Runtime just deprecated WebGL and JSEP. Read the changelog, not the WebNN press release.
Two browser-ML milestones landed a day apart and point in opposite directions. Thirty WebGPU changes against one WebNN line — including PagedAttention in a browser.
Production Blueprint
CI/CD pipeline configuration templates engineered for rapid deployment of isolated environments.