On-device and edge machine learning. Apple Silicon, MLX and Metal, quantization, and the inference runtimes that actually ship.

01 // Recent Publications

Local Codex on a 16 GB M5: gpt-oss-20b decoded at 0.3 tokens per second

I tried codex --oss with gpt-oss-20b on my 16 GB M5. The weights are 12.82 GiB, Metal gives the GPU 11.84 GiB, and Ollama defaults to a 4k context. Measurements, the math, and the config I ended up with.

#codex #openai #gpt-oss #apple-silicon

In bf16, a decay of 0.9995 is 1.0

MLX cast the gated delta net decay to the activation dtype. In bf16 that rounds 27% of gate values to exactly 1.0, and a state that never decays is not a bug the loss curve will tell you about.

#mlx #metal #numerics #linear-attention

llama.cpp's BF16 attention kernel: the prefill number is the headline, the accumulator is the story

PR #30491 adds BF16 instantiations of llama.cpp's Metal tensor-API flash attention. The +44% prefill is prefill-only. The real finding is that the simdgroup BF16 kernel accumulates attention output in half, with the fp32 variant commented out.

#llama-cpp #metal #tensor-api #numerics

llama.cpp's MoE expert cache: read the eligibility predicate before the benchmark table

llama.cpp PR #29887 adds --moe-cache-mib, an LRU of MoE experts in GPU memory with 1.57-2.20x decode speedups over PCIe. All three Metal buffer types report is_host = false, so on Apple Silicon the cache disables itself.

#llama-cpp #moe #unified-memory #apple-silicon

Speculative decoding on Apple GPUs was slower than not using it. The fix landed this morning.

Metal had no mat-mul kernel for a handful of rows, so llama.cpp verified speculative drafts with the mat-vec kernel. On an M3 Ultra that made DFlash2 decoding slower than serial decoding: 16.8 t/s against 32. PR #29869 takes it to 64.4.

#llama-cpp #metal #speculative-decoding #kernels

1918 live tensors at the end of prefill: ExecuTorch's MLX backend never released a temp slot

gemma4-e2b on an iPhone 16 finished prefill holding 3.1 GB of intermediates and was killed by jetsam. The cause is the AOT memory plan, not a refcount bug, and set_cache_limit(0) moves peak memory by exactly zero.

#executorch #mlx #ios #memory

MLX's nvfp4 prefill went 1,267 → 4,256 tok/s in two PRs. Decode went 81.9 → 81.7.

Two MLX PRs merged September 14 took nvfp4 prefill on Qwen3.6-35B-A3B from 1,267 to 4,256 tok/s on M5 Max. Decode went 81.9 to 81.7. That split is the whole lesson.

#mlx #nvfp4 #quantization #m5

MiniCPM5-2B's 131k context is 5.25 GiB of KV cache on a phone. The config.json tells you more than the leaderboard.

Two KV heads, 42 layers, head_dim 128 — 43 KB per token. The honest window on a phone is 8k–16k, and the 130k vocab is a fixed surcharge on every token you decode.

#on-device #ios #slm #kv-cache

llama.cpp's Metal 4 fast path now ships as two metallibs, and on iPhone the second one is the whole point

The silent tensor-API probe failure was a shader-compiler default. The fix disables the fast path for precompiled libraries — which is everything on iOS — so the xcframework now ships a second metallib gated on SDK 26.

#on-device #apple-silicon #metal #llama-cpp

Apple's 4.3x is a prefill number, and one init-time check decides whether you get it

The M5 Ultra Mac Studio landed this week. A Metal 4 capability check that fails silently can cost you 2–3x of its headline gain — and generation speed looks fine either way.

#on-device #apple-silicon #metal #llama-cpp

Speculative decoding is not a free 2x on Apple Silicon, and MoE makes it worse

Liquid AI shipped an on-device result with 85% draft acceptance and a 1.04x speedup. Why sparse experts and unified memory break the assumption that acceptance rate predicts throughput.

#on-device #apple-silicon #metal #llama-cpp

ExecuTorch's Apple Silicon path is MLX and hand-written Metal. Not Core ML, not MPS.

Meta's flagship on-device model demo routes around both delegates Apple ships in the OS. What that means if you're choosing a backend today, plus the K-quant repacking trick worth stealing.

#on-device #apple-silicon #executorch #mlx

ONNX Runtime just deprecated WebGL and JSEP. Read the changelog, not the WebNN press release.

Two browser-ML milestones landed a day apart and point in opposite directions. Thirty WebGPU changes against one WebNN line — including PagedAttention in a browser.

#on-device #webgpu #webnn #npu
02 // Projects & Open Source

Production Blueprint

CI/CD pipeline configuration templates engineered for rapid deployment of isolated environments.

Stack: GitHub Actions, Docker, Shell