M5 Max MLX Stack Doubles Local LLM Speed vs Cloud
Apple M5 Max with MLX-optimized Gemma 4 and Qwen 3.5 hits 118 tokens/sec vs GGUF's 60, 15-50% faster than M4 Max, exposing cloud APIs as overpriced for many workloads.
Hardware Leap: M5 Max Delivers 15-50% Wall Clock Gains Over M4 Max
The core opportunity was reducing dependency on unreliable cloud APIs like Claude and OpenAI, which failed mid-recording. IndyDevDan unboxed a fully specced M5 Max MacBook Pro (128GB RAM) against an M4 Max equivalent, benchmarking real workloads to prove local inference viability. Decision: Prioritize Apple Silicon's unified memory and GPU neural accelerators for private, zero-cost runs.
Key metrics from cold/warm prompts and live benchmarks:
- M5 Max prefill/decode speeds consistently higher across models.
- Wall clock (end-to-end time) 15-50% faster on M5, with quieter fans and lower thermal load.
- Peak RAM: Gemma 4 MLX fits in 16GB; full runs peak at 42-55GB on 128GB systems.
Tradeoff: Gains shine on simple prompts but scale with hardware limits—future M5 Ultra or M6 with 500GB RAM could dominate more.
"The M5 is about 15 to 50% faster than the M4, which is a pretty massive jump." (IndyDevDan, highlighting wall clock improvements after first benchmark, emphasizing real-world usability over raw specs.)
MLX vs GGUF: 2x Performance Edge Makes GGUF Obsolete on Apple Silicon
Problem: Engineers overlook MLX (Apple's ML framework) for GGUF (Ollama standard), leaving speed on the table. Options evaluated: GGUF (universal but slower) vs MLX (Apple-optimized with Nvidia NVFP4 quantization).
Decision: Always use MLX on Apple Silicon—nearly double prefill (550 t/s Gemma GGUF vs MLX variants) and decode (118 t/s MLX Qwen/Gemma vs 60 t/s GGUF Qwen). Why? MLX leverages mixture-of-experts and NVFP4 for efficiency; GGUF lacks hardware-specific tuning.
Benchmarks via live-bench tool (https://github.com/disler/live-bench):
- Simple prompts (e.g., "explain hash table"): MLX Qwen 118 t/s decode, Gemma MLX 100+ t/s.
- Minimum viable: >30 t/s usable; <20 t/s "dead zone."
Tradeoffs: MLX prefill slower on tiny prompts (less relevant for agents); locked to Apple. GGUF more portable but halves speed—"If you're running GGUF on Apple Silicon in 2026, you're leaving 2x performance on the table."
"MLX smokes the GGUF format. Not by a little. By a LOT. 118 tokens per second vs 60." (IndyDevDan, post first benchmark, calling out the format gap as the video's controversial finding, urging switch for production local stacks.)
Model Showdown: Gemma 4 Edges Qwen 3.5 in Speed and Density
Compared Gemma 4 (Google, US-open) vs Qwen 3.5 (Alibaba, 35B MoE). Both in MLX/NVFP4.
Decision: Gemma 4 wins for packed efficiency (max intelligence/parameter, 16GB footprint) and prefill speed; Qwen competitive but slower decode. Why? Gemma's architecture prioritizes throughput; both excel over cloud for privacy/speed.
Metrics:
- Prefill: Gemma GGUF/MLX ~550 t/s > Qwen.
- Decode: Both MLX ~100-118 t/s.
- Wall time: Gemma blitzes simple tasks.
Tradeoff: Gemma US-origin appeals politically, but no quality gap—choose by workload. "Google actually cooked here."
"Gemma 4 is an incredibly packed model... maximizing intelligence per parameter." (IndyDevDan, praising post-benchmark, noting US competitive edge without discriminating origins.)
Context Scaling Cliff: Local Hits Wall Past 16K Tokens
Opportunity: Agents need long contexts (graph walks benchmark: BFS on scaling graphs, 200-32K tokens).
Findings: Prefill dominates time as prompts grow; M5/MLX holds ~117 t/s decode but accuracy drops (e.g., errors at 8K vs cloud's 80% at 1M). Wall clock balloons—32K prompts fully tax GPUs (100% util, fans on).
Decision: Limit local to <16K for speed; use cloud hybrids for massive context. Why? KV cache/RAM constraints; innovations needed.
"The context window cliff: why local inference falls off HARD past 16K tokens." (IndyDevDan, from intro, warning on overlooked scaling pain in agent pipelines.)
Agentic Coding Reality: Pi Agent Proves Local Viability with Caveats
Tested Pi coding agent (pi.dev) on full workflows. Decision: Local micro-agents win for engineering/personal tasks (privacy, zero latency/API bills). Why? Handles coding despite context limits; beats cloud downtime.
Results: M5/MLX succeeds on agentic tasks (e.g., rate limiter design), but non-deterministic—varies by prompt complexity.
Tradeoff: Full agents need context hacks; ideal for "task tier" sub-agents. "Local agents actually do agentic coding? (Yes. With caveats.)"
Future-Proofing: Local-First for Cost/Control as Tipping Point Nears
Big picture: Cloud pricing/deprecation risks vs local ownership. Thesis: Micro-agents on-device for 80% workloads; benchmark your hardware now.
"Every time Claude goes down... you're getting reminded who actually owns your stack. Spoiler: it's not you." (IndyDevDan, opening rant, framing local as rebellion against API racket amid live outage.)
Key Takeaways
- Switch to MLX models on Apple Silicon for 2x speed over GGUF—118 t/s decode benchmark.
- M5 Max yields 15-50% wall clock wins vs M4; monitor RAM (42-55GB peaks).
- Favor Gemma 4 MLX for density/speed; >30 t/s minimum for usable local inference.
- Cap contexts at 16K to avoid cliffs; hybrid cloud for longer.
- Build micro-agents with Pi-like tools for private coding—test via live-bench.
- Benchmark your stack: Prep for M5 Ultra/M6 obliterating APIs.
- Ditch cloud for tasks valuing privacy/speed over massive context.