By gentic.news Staff | March 2026
Independent AI researcher Matthew Weinbach has reported preliminary findings suggesting that "near lossless compression of LLMs is possible on Apple Silicon," specifically achieving storage at 3.5 bits-per-weight (bpw) while maintaining performance within 1-2% the quality of bf16 (Brain Floating Point 16) precision.
The claim, shared via social media, points to a significant potential advancement in running large language models efficiently on consumer Apple hardware without substantial quality degradation.
What Happened
On March 15, 2026, Matthew Weinbach (@mweinbach) posted: "I'm testing but it does look like near lossless compression of LLMs is possible on Apple Silicon... like store in 3.5bpw and get within 1-2% the quality of bf16."
The tweet indicates ongoing experimentation with quantization and compression techniques specifically optimized for Apple's M-series architecture. While no detailed methodology, specific models tested, or exact benchmark numbers were provided, the core claim is clear: a compression ratio reducing weights from 16 bits to approximately 3.5 bits (a ~78% reduction in model size) while preserving nearly all original model accuracy.
Context: The Push for Efficient LLM Deployment
This work fits into the broader industry trend of making large models feasible to run on-device. Apple has been aggressively pursuing this path with its Neural Engine and unified memory architecture. The ability to store a model at 3.5 bpw would dramatically reduce the memory footprint required, potentially allowing models previously requiring 8-16GB of RAM to run on devices with 4-8GB.
Current state-of-the-art quantization techniques, such as GPTQ, AWQ, and GGUF's Q4_K_M (4.5 bpw), typically aim for 4-bit quantization. Achieving comparable quality at 3.5 bits would represent a meaningful step forward in the compression frontier, especially if the "1-2%" quality drop holds across diverse benchmarks and model families.
Key Implication: If validated, this compression level could make high-performance 7B-13B parameter models (like Mistral or Llama variants) comfortably run on base-model MacBooks and iPads, unlocking more capable local AI assistants and tools.
Technical Questions & Next Steps
Weinbach's announcement is a teaser, not a full research release. Critical details remain unknown:
- Which models were tested? (e.g., Llama 3.1 8B, Phi-3, Qwen2.5)
- Which benchmarks define "quality"? (MMLU, ARC, HellaSwag, GSM8K)
- What is the exact compression method? Is it a novel quantization scheme, pruning, or a combination?
- Does "Apple Silicon" optimization rely on specific hardware features (e.g., AMX, Neural Engine)?
- What is the inference speed vs. memory trade-off?
Typically, pushing quantization below 4 bits requires sophisticated techniques like mixed-precision, where sensitive layers or weights are kept at higher precision. The claim of "near lossless" at 3.5 bpw suggests a potentially clever allocation of these precious bits.
gentic.news Analysis
This development, while preliminary, aligns directly with two major trends we've been tracking. First, it follows Apple's strategic pivot towards on-device AI, a direction solidified with the launch of their Apple Intelligence platform in 2024 and subsequent hardware optimizations. Second, it intersects with the intense research focus on post-training quantization, an area where companies like OctoAI and academic groups have been pushing the limits of 4-bit and below.
If Weinbach's results are reproducible and generalizable, they could slightly disrupt the current competitive landscape. Apple's ecosystem advantage hinges on seamless, private, on-device experiences. Efficient 3.5 bpw storage would allow Apple to ship more capable base models in its operating systems or enable developers to include larger models within app size limits. This creates a tighter integration loop that cloud-dependent competitors cannot easily match.
However, caution is warranted. The "1-2%" quality drop needs rigorous verification. A drop on simple benchmarks might mask larger degradation on complex reasoning tasks. Furthermore, the technique's applicability may be limited to specific model architectures that Apple's silicon and software stack (via Core ML and MLX) are optimized for. We'll be watching for a full technical report or code release to assess the true impact.
Frequently Asked Questions
What does "3.5 bits-per-weight" mean?
It means each parameter (weight) in the neural network is stored using an average of 3.5 bits of memory. For comparison, full precision is typically 32 bits (FP32), half-precision is 16 bits (BF16/FP16), and common "4-bit" quantization uses 4 bits. Storing at 3.5 bpw reduces the model's memory footprint by approximately 78% compared to BF16.
How significant is a "1-2%" quality drop?
Context is crucial. A 2% absolute drop on a benchmark like MMLU (where top models score ~85%) is noticeable but often acceptable for massive gains in efficiency. However, a 2% drop on a benchmark where the model scores 50% could represent a 4% relative drop in error, which is more substantial. The specific tasks affected matter greatly for practical utility.
Is this only for Apple Silicon?
The initial claim specifies "on Apple Silicon." This suggests the method may leverage hardware-specific features like Apple's Advanced Matrix Extensions (AMX) or the Neural Engine for efficient low-bit arithmetic. It may not achieve the same efficiency-quality trade-off on x86 or NVIDIA GPUs without modification.
When will this technique be available to use?
As of now, this is an informal result shared by a researcher. There is no public code, paper, or product integration. If the findings are robust, we might see the techniques appear in popular quantization libraries like llama.cpp (which supports GGUF formats) or Apple's own MLX framework in the coming months.
