Cactus Needle: 26M-Parameter Model Runs Locally at 6,000 Tokens/Second

Cactus Needle is a 26-million-parameter Simple Attention Network distilled from Google's Gemini 3.1, designed to run entirely locally on consumer hardware. On a Mac, it achieves 6,000 tokens per second for prefill and 1,200 tokens per second for decode — speeds that make real-time on-device AI practical for the first time at this quality level.

The model's weights are fully open, enabling developers to fine-tune, compress, and deploy it across edge devices without any licensing restrictions. The architecture uses a simplified attention mechanism that dramatically reduces computational overhead while maintaining useful reasoning capabilities.

The strategic significance extends beyond the model itself. Cactus Needle represents the emerging path to truly on-device AI for consumer electronics — phones, smartwatches, AR glasses, and IoT devices. At 26 million parameters, the model is small enough to fit comfortably on edge silicon while being smart enough to handle tasks like text generation, summarization, and basic reasoning without any cloud dependency.

For MENA-based developers, this is particularly relevant. On-device AI eliminates latency issues caused by distance from hyperscaler data centers and works reliably regardless of internet connectivity — a practical advantage in regions with variable network infrastructure. The open weights also mean developers can fine-tune the model for Arabic language tasks without relying on external API providers.

The model joins a growing ecosystem of small, efficient AI models that are making edge AI a commercial reality rather than a research curiosity. Combined with Apple's neural engine chips and Qualcomm's on-device AI hardware, the infrastructure for capable local AI is maturing rapidly.