Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

GGUF

GGUF is a binary file format for storing a model's weights, tokenizer and metadata in one file, built for fast loading and local inference with GGML-based tools such as llama.cpp, often in quantized form.

Also called: GGUF file

By Divya Sharma · Checked Nov 15, 2026 · How we review

If you download a model to run locally, you will often see files ending in .gguf. The format comes from the GGML project, the tensor library behind llama.cpp, and it is the format used by Ollama and LM Studio.

What is inside

A GGUF file holds the model's tensors plus key-value metadata: the architecture, the tokenizer, context length and quantization details. The specification lists single-file deployment as a goal, so the model loads with no extra config files. Models are usually trained in PyTorch and then converted to GGUF.

GGUF and quantization

GGUF can store full-precision weights, but most GGUF downloads are quantized. One model is often published as several files, such as Q8_0, Q5_K_M and Q4_K_M, each trading size for quality. Pick the largest one that fits in your memory with room left for context. The GPU calculator helps here.

GGUF vs safetensors

Safetensors is the usual format for full-precision weights on Hugging Face, used by Python libraries and GPU servers such as vLLM. GGUF targets local, CPU-friendly and Apple Silicon inference. The two serve different runtimes, and conversion scripts move models between them.

Related terms

Quantization
Quantization is a technique that stores a model's weights, and sometimes activations, with fewer bits, such as 8 or 4 instead of 16, so the model needs less memory and often runs faster, at some cost in accuracy.
Safetensors
Safetensors is a file format from Hugging Face for storing model weights. Unlike Python pickle files, loading it cannot run hidden code, and it supports fast, memory-mapped loading of individual tensors.
Local LLM
A local LLM is a language model that runs on your own computer, phone or server instead of a provider's cloud API, so prompts and data stay on your hardware and it works without an internet connection.
Model weights
Model weights are the learned numbers inside a neural network, set during training, that decide how it turns input into output. A checkpoint is a saved file or set of files containing those weights.
Self-hosting (AI models)
Self-hosting an AI model means running it on hardware you control, such as your own servers, rented GPUs or a laptop, instead of calling a provider's API. You choose the model and keep the data, but you run the infrastructure.

Sources