Shrinking a model by storing its weights in fewer bits, sixteen down to eight or four, so it fits on a laptop or a phone with a small loss of quality. The word is old in signal processing. It became hobbyist vocabulary in March 2023, when llama.cpp let people run a leaked Meta model on a MacBook, and 'Q4' turned into a file suffix people argued about.
Quantisation is not distillation and not pruning. Nothing is removed and nothing is retrained; the same weights are stored with less precision. A quantised model has the same knowledge as its parent, slightly blurred. The blur is what the suffixes measure.
Sighted in the file names of 2023: model-7b-Q4_K_M.gguf, model-13b-Q5_K_S.gguf, and a forum thread eleven pages long about which letter meant what. The record notes that people who had never heard the word in February were comparing perplexity tables by May.
I remember running a seven-billion-parameter model on a Raspberry Pi in the spring of 2023, at one word every few seconds, and finding it more thrilling than any cloud demo. It was slow and slightly stupid and it was mine. That is what quantisation bought.
For the record: a model's weights are numbers, and numbers can be rounded. Sixteen-bit to four-bit cuts the file by three-quarters. Below that, the model starts to forget things unevenly, like a photograph saved too many times. The community's job was finding out how far you could go.
The record is wrong. Show what it should say, and why.
