The Mistral model of December 2023 that made 'mixture of experts' a phrase civilians used, being eight small models with a router that picks two per token, released as a torrent on a Friday and topping the open leaderboards by the following week. The record notes it matched or beat GPT-3.5 on the usual tables while running on a well-equipped home machine.
The record notes that the 8x7B in the name was misread as 56 billion parameters by most of the internet, and that the correct figure, around 47 billion with shared layers, was explained in every thread by one patient person.
Correction: Mixtral did not invent mixture-of-experts. The technique is decades old and Google had published on it at scale; Mixtral was the first open-weight version people could download and run.
December 2023, hosting providers racing to serve the model before Mistral had written documentation for it, and prices per million tokens dropping for a week in public. The record notes the model was cheaper than its own launch.
What does it mean? Write it the way you would say it out loud.
