Detailed explanation
Moving a model from 16 or 32 bits down to 8, 4 or even 2 bits sharply cuts memory and speeds up inference, at the cost of a small quality drop. Essential to run large LLMs on consumer GPUs or locally.
Examples
Llama 70B quantized to 4-bit on an M3 Mac
GGUF models for llama.cpp
Edge inference on mobile
Frequently asked questions
Is there a quality loss?
Yes but often marginal at 8-bit and acceptable at 4-bit for most use cases.
Related terms
Distillation
A process that trains a small "student" model to mimic a large "teacher" model, keeping quality at lower cost.
Fine-tuning
Fine-tuning is the process of further training a pre-trained model on a smaller, task-specific dataset to specialize it for a particular domain, style, or task.
LLM (Large Language Model)
An LLM is a large language model trained on massive text corpora, capable of understanding and generating natural language across many tasks: writing, summarization, translation, code, reasoning.
Deep Learning
A subfield of machine learning using deep neural networks to learn complex representations.
Last updated: 7/15/2026