Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

Running and deploying

Fine-tuning, serving and hosting models, and what the hardware and pricing terms mean.

AI training data
AI training data is the collection of examples a model learns from, such as web text, code, images, transcripts or labelled pairs. Its size, quality, mix and licensing shape what the model can do and where it fails.
API key
An API key is a secret string an application sends with each request so that a service can tell which account is calling, apply its limits and bill it. Anyone holding the key can usually use it.
Batch API
A batch API lets you submit a large set of model requests in one job and collect the results later, instead of waiting on each call. Providers run the job when capacity allows and usually charge less for it.
Continuous batching
Continuous batching is a model-serving technique that adds new requests to the running GPU batch and removes finished ones after every generation step, instead of waiting for a whole batch to complete. It keeps the GPU busy and raises throughput.
CPU offloading
CPU offloading runs a model that does not fit in GPU memory by keeping some of its layers in system RAM, or on disk, and using the GPU for the rest. The model runs, but more slowly.
Direct preference optimization
Direct preference optimization (DPO) is a way to align a language model with human preferences by training directly on pairs of preferred and rejected answers, using a simple classification-style loss instead of a reward model and reinforcement learning.
Fine-tuning
Fine-tuning is further training of a pretrained model on a smaller, task-specific dataset, so the model adopts a particular skill, format, tone or domain without being trained from scratch.
GPU
A GPU (graphics processing unit) is a processor with many small cores built to run many calculations in parallel. Its speed at matrix maths makes it the standard hardware for training and running AI models.
GPU cloud
A GPU cloud is a service that rents out GPU machines by the hour or second for AI training, fine-tuning and inference. Specialist providers, sometimes called neoclouds, focus on GPUs and often charge less than the large general clouds.
Inference
Inference is running a trained model to produce output from new input, such as an LLM generating a reply to a prompt. It is the stage users pay for every time they call a model, as opposed to training.
Inference provider
An inference provider is a company that runs AI models on its own hardware and sells access to them through an API, usually charged per token. You send prompts and get outputs without managing GPUs yourself.
KV cache
A KV cache stores the attention keys and values a transformer has already computed for earlier tokens, so each new token only needs work for itself. It speeds up generation but takes GPU memory that grows with context length.
Local LLM
A local LLM is a language model that runs on your own computer, phone or server instead of a provider's cloud API, so prompts and data stay on your hardware and it works without an internet connection.
LoRA
LoRA (low-rank adaptation) is a fine-tuning method that freezes a model's original weights and trains small low-rank matrices added to its layers, so far fewer parameters change and much less GPU memory is needed.
Model fallback
Model fallback is a rule that sends a request to a backup model or provider when the first choice fails, times out, hits a rate limit or rejects the input. It keeps an AI feature working through outages.
Model serving
Model serving is running a trained model as a service that applications call over a network, with software that loads the weights onto GPUs, queues and batches requests, manages memory and returns results.
NPU
An NPU (neural processing unit) is a chip, or a block inside a larger chip, built to run neural network maths quickly and with little power. Phones and laptops use NPUs to run small AI models on the device.
Numeric precision (FP16, BF16, FP8)
Numeric precision is the number format used to store a model's weights and run its maths, such as FP32, FP16, BF16, FP8 or INT4. Fewer bits per number means less memory and faster inference, at some risk to accuracy.
OpenAI-compatible API
An OpenAI-compatible API is an endpoint from another provider or tool that accepts the same request and response format as OpenAI's API, so code written for OpenAI can switch to it by changing the base URL, key and model name.
PagedAttention
PagedAttention is a method for storing a language model's KV cache in small fixed-size blocks that need not sit next to each other in GPU memory, like pages in an operating system. It cuts wasted memory so a server can run more requests at once.
Parameter-efficient fine-tuning
Parameter-efficient fine-tuning (PEFT) adapts a pretrained model by training only a small number of new or selected parameters, leaving the rest frozen. It cuts the memory, compute and storage that full fine-tuning needs.
Prompt caching
Prompt caching is an API feature that stores the processed form of a repeated prompt prefix, such as a long system prompt or document, so later requests that start with the same prefix are cheaper and faster.
Rate limit (API)
A rate limit caps how much an account can use an API in a time window, typically as requests per minute (RPM) and tokens per minute (TPM). Requests over the limit are rejected, usually with an HTTP 429 error.
Reward model
A reward model is a separate model trained on human preference data to score how good a response is. Reinforcement learning then uses that score to tune a language model towards answers people prefer.
RLHF
RLHF (reinforcement learning from human feedback) fine-tunes a language model using human rankings of its answers: a reward model learns those preferences, then reinforcement learning pushes the model toward answers that score well.
Self-hosting (AI models)
Self-hosting an AI model means running it on hardware you control, such as your own servers, rented GPUs or a laptop, instead of calling a provider's API. You choose the model and keep the data, but you run the infrastructure.
Semantic cache
A semantic cache stores past LLM responses and returns one when a new request means roughly the same thing as an earlier one, judged by embedding similarity rather than an exact text match. It saves cost and latency on repeated questions.
Serverless GPU
Serverless GPU is a cloud model where you deploy code or a model and the platform starts GPU workers only when requests arrive, billing for the time they run and scaling to zero when idle.
Speculative decoding
Speculative decoding speeds up text generation by letting a small, fast draft model propose several tokens, which the large model then checks in a single pass, keeping the accepted ones. The output matches what the large model alone would produce.
Streaming (LLM output)
Streaming sends an LLM's output to the client piece by piece as tokens are generated, instead of waiting for the full response. Users see text appear within moments, even when the complete answer takes much longer.
Supervised fine-tuning
Supervised fine-tuning (SFT) trains a pretrained model further on example prompts paired with the responses you want. It is the usual first step that turns a base model into one that follows instructions.
Synthetic data
Synthetic data is training or test data generated by software, often by another AI model, rather than collected from real-world sources. It is used to fill gaps, add examples cheaply, or avoid exposing real personal data.
Tensor parallelism
Tensor parallelism splits each layer of a neural network across several GPUs, so that every GPU holds part of each weight matrix and computes its share of every step. It lets a model too large for one GPU run across many.
Time to first token
Time to first token (TTFT) is how long a user waits between sending a request to a language model and receiving the first token of its reply, covering queuing, prompt processing and network time.
Token pricing
Token pricing is how most AI model APIs charge: a set price per million input tokens you send and a separate, usually higher, price per million output tokens the model generates, with discounts for cached input and batch jobs.
Tokens per second
Tokens per second (tok/s) measures how fast a language model produces output. It is quoted either per user, the speed one reply streams at, or per system, the total output across all concurrent requests.
TPU
A TPU (tensor processing unit) is Google's custom chip for machine learning, built around large matrix-multiply units and high-bandwidth memory. Google offers TPUs to other companies as a rented service on Google Cloud.
Unified memory
Unified memory is a design in which the CPU and GPU share one pool of memory instead of each having its own. On Apple Silicon Macs and similar chips, it lets a local AI model use most of the system RAM as GPU memory.
VRAM
VRAM is the memory on a graphics card. For AI work it holds the model weights, the KV cache and working data, and whether a model fits in VRAM usually decides which GPU you need to run it.