A developer has demonstrated a Gemma 4 mixture-of-experts model running on a recycled server built around a 2016 Intel Xeon E5-2620 v4, 128GB of DDR3 memory and no graphics processor. The result is less a claim that old hardware matches modern accelerators than a detailed account of how specialized inference settings can make an otherwise impractical configuration usable.
The system pairs a roughly 25.2-billion-parameter Gemma 4 26B-A4B verifier with a smaller draft model. The verifier contains 128 experts but activates eight for each token, amounting to about 3.8 billion active parameters at a time. That sparse structure reduces computation, although the full weights still have to fit in memory and be accessed efficiently.
According to the author, token generation on this machine is primarily limited by memory bandwidth. During decoding, the processor repeatedly moves model weights from relatively slow DDR3 memory toward its caches. Matrix operations can finish before the next data arrives, leaving compute units waiting. The decade-old CPU and memory therefore make default settings extremely slow even though the server has enough capacity to hold the model.
The experiment uses `ik_llama.cpp` rather than a simplified model runner because it exposes more optimization controls. Speculative decoding lets the small model propose up to three tokens, which the larger verifier can evaluate together. The author argues that this is especially useful on CPUs: additional computation for a drafter that fits in cache costs less than repeatedly streaming the verifier's weights from main memory. Automatic tuning adjusts the proposed sequence length for the workload.
Other settings address the cache hierarchy directly. A CPU mixture-of-experts option attempts to improve expert routing, while merging the up and gate projections combines operations that would otherwise require separate memory transfers. The process uses eight threads to match the machine's eight physical cores rather than all 16 simultaneous-multithreading threads, because more workers do not solve the shared bandwidth bottleneck. Runtime repacking reorganizes weight matrices for the processor's preferred cache layout, and memory locking is intended to prevent model data from being swapped to disk.
The report is a single-machine experiment, and the supplied material does not provide a standardized speed, power-consumption or quality comparison. Its useful finding is narrower: sufficient RAM alone does not determine whether large-model inference is practical. Model sparsity, speculative decoding, cache behavior, thread count and memory layout can collectively extend the useful life of old server hardware, albeit with more manual engineering than mainstream tools expose.\n\nThe amount of manual tuning is part of the result. It makes the setup relevant to experimenters with spare servers, but less suitable as a drop-in recommendation for ordinary users. Reproducing it requires matching flags to the exact processor, memory system, model files and inference build.
