Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

Safety, security and evaluation

How AI systems are attacked, guarded, governed and measured, including the benchmarks.

Abliteration
Abliteration is a technique that stops a language model from refusing requests by finding the single internal direction that carries refusal and editing the weights so the model can no longer write to it, without any retraining.
Agent permissions
Agent permissions are the rules that decide which tools, files, commands, networks and accounts an AI agent may use, and which actions need a human's approval first, so a mistaken or manipulated agent can do limited damage.
AI alignment
AI alignment is the work of making an AI system's behaviour match what its users and developers actually intend, so it is helpful, honest and avoids harm, rather than just predicting likely text or optimising a narrow score.
AI benchmark
An AI benchmark is a fixed, public set of tasks with an agreed scoring method, used to compare models on the same test, for example coding issues solved, exam questions answered or retrieval quality across many datasets.
AI red teaming
AI red teaming is the practice of deliberately attacking a model or AI application before and after release, with human testers or automated tools, to find harmful outputs, jailbreaks, injections and other failures so they can be fixed.
AI supply-chain attack
An AI supply-chain attack compromises an AI application through something it depends on, such as a downloaded model, a LoRA adapter, a dataset, a Python package, an MCP server or an agent skill, rather than attacking the application directly.
AI watermarking
AI watermarking is embedding a hidden, machine-detectable signal in text, images, audio or video at the moment an AI model generates it, so software can later identify the content as AI-generated.
AIME (benchmark)
AIME, the American Invitational Mathematics Examination, is a 15-question US maths competition with whole-number answers from 0 to 999; AI labs use its past papers, such as AIME 2025, as a maths reasoning benchmark.
ARC-AGI
ARC-AGI is a series of benchmarks, started by François Chollet and run by ARC Prize, that tests whether an AI system can solve new reasoning puzzles it has not seen before, measuring how efficiently it learns rather than what it knows.
Artificial Analysis Intelligence Index
The Artificial Analysis Intelligence Index is a composite score from the benchmarking company Artificial Analysis that combines several evaluations, spanning knowledge work, coding, science and general knowledge, into one number for comparing language models.
Benchmark contamination
Benchmark contamination is when a model has seen a benchmark's test questions or answers during training, so its score on that benchmark overstates how well it handles new problems of the same kind.
BrowseComp
BrowseComp is an OpenAI benchmark of 1,266 hard questions that tests whether a browsing agent can search the web persistently for tangled, hard-to-find facts, with short answers that are easy to check.
C2PA content credentials
C2PA content credentials are signed provenance records, defined by an open standard from the Coalition for Content Provenance and Authenticity, that travel with an image, video, audio file or document and record its origin and edits.
Data exfiltration (AI agents)
Data exfiltration, for AI agents, is when an attacker tricks an agent into reading private data it can access and sending it out, through a web request, an email, an image link or another channel the attacker controls.
Data residency
Data residency is a control that keeps a customer's data stored, and often processed, in a chosen geographic region such as the EU, so a team can meet its own legal, contractual or customer rules on where data lives.
Deepfake
A deepfake is image, audio or video content made or altered with AI so that it convincingly shows a real person, place or event in a way that did not happen and could pass as authentic.
Elo rating (model arenas)
An Elo rating, in model arenas, is a score computed from many head-to-head votes between pairs of models, so that the gap between two scores predicts how often one model's answer will be preferred over the other's.
EU AI Act
The EU AI Act, Regulation (EU) 2024/1689, is the European Union's law on artificial intelligence. It sets risk-based rules for those who build and deploy AI systems, from outright bans to transparency duties, phased in from 2025.
Golden dataset
A golden dataset is a fixed, carefully checked set of example inputs paired with the answers or behaviour you expect, used as the reference when you evaluate or regression-test an AI system.
GPQA
GPQA (Graduate-Level Google-Proof Q&A) is a benchmark of 448 hard multiple-choice questions in biology, physics and chemistry, written by experts so that skilled non-experts cannot answer them even with unlimited web search.
Guardrails (AI)
Guardrails are runtime checks placed around a language model that inspect inputs and outputs and block, rewrite or redirect them, so an application stays on topic, avoids harmful content and follows rules the model alone would not reliably keep.
Human evaluation
Human evaluation is judging AI outputs with people rather than automated metrics, for example raters scoring answers against a rubric or picking the better of two anonymous responses in a blind test.
HumanEval
HumanEval is a code-generation benchmark released by OpenAI in 2021: 164 hand-written Python problems, each with a function signature, docstring and unit tests, used to check whether a model writes code that actually works.
Humanity's Last Exam
Humanity's Last Exam (HLE) is a benchmark of 2,500 expert-written questions across dozens of subjects, built to be very hard for AI models, with answers that are clear and checkable but cannot be found quickly online.
Jailbreak (LLM)
A jailbreak is a prompt or technique that gets a language model to produce output its safety training is meant to refuse, such as harmful instructions, by role-play, obfuscation, odd encodings or long adversarial text.
Leaderboard (AI)
An AI leaderboard is a public ranking of models by their scores on one or more benchmarks or on human preference votes, updated as new models are tested, so readers can compare models side by side.
LLM evaluation
LLM evaluation, often called evals, is measuring how well a model or AI application performs a task by running it on a set of test cases and scoring the outputs against defined criteria, by code, by humans or by another model.
LLM observability
LLM observability is recording what an AI application does in production, including prompts, responses, tool calls, latency, token use and cost, as traces you can search, so you can debug failures and see how the system behaves over time.
LLM-as-a-judge
LLM-as-a-judge is an evaluation method where a strong language model grades the outputs of another model or application, using a rubric, a reference answer or a side-by-side comparison, in place of or alongside human reviewers.
LMArena
LMArena, first known as Chatbot Arena, is a public platform that ranks AI models by crowdsourced human votes: users see two anonymous models answer the same prompt and pick the better response.
MMLU
MMLU (Massive Multitask Language Understanding) is a multiple-choice benchmark covering 57 subjects, from elementary maths and US history to law and computer science, that tests a language model's broad knowledge and problem solving.
Model card
A model card is a short document published with a machine learning model that explains what it is for, how it was trained and evaluated, where it performs poorly, and how it should and should not be used.
MTEB
MTEB, the Massive Text Embedding Benchmark, is a public benchmark and leaderboard that scores text embedding models across many task types, such as retrieval, clustering, classification and reranking, over many datasets and languages.
OAuth for AI agents
OAuth for AI agents is the use of the OAuth authorization standard to let an agent or MCP client act on a user's behalf with a limited, revocable access token, instead of holding the user's password or a broad API key.
OSWorld
OSWorld is a benchmark for computer-use agents that runs real operating systems such as Ubuntu, Windows and macOS and scores agents on 369 open-ended tasks across web and desktop apps, checked by scripts.
pass@k
pass@k is a metric for code benchmarks that gives the chance at least one of k generated solutions to a problem passes its unit tests; pass@1 is the chance that a single attempt succeeds.
PII redaction
PII redaction is the detection and removal or masking of personally identifiable information, such as names, phone numbers, addresses and card numbers, from text or images before it is sent to a model, stored in logs or shown to others.
Prompt injection
Prompt injection is an attack where text fed to a language model, typed by a user or hidden in a web page, file or tool result, changes the model's behaviour in ways its developer did not intend.
Regression testing (LLM apps)
Regression testing for LLM apps means re-running a fixed set of evaluation cases after every change to a prompt, model, retrieval setup or tool, to catch cases that used to work and now fail.
SOC 2
SOC 2 is an audit report, produced by an outside CPA audit firm under AICPA standards, on a service organisation's controls relevant to security, availability, processing integrity, confidentiality or privacy.
SWE-bench
SWE-bench is a coding benchmark built from real GitHub issues in popular Python projects; a model or agent must edit the repository so that the issue is fixed and the project's hidden tests pass.
Terminal-Bench
Terminal-Bench is a benchmark for AI agents that work in a command-line terminal, where each task gives the agent a real environment and is scored by tests that check whether the job was actually done.
Tool poisoning
Tool poisoning is an attack where a malicious tool, often an MCP server, hides instructions in its tool description or metadata; the AI model reads them as guidance and may leak data or misuse other tools without the user noticing.
Zero data retention
Zero data retention (ZDR) is an arrangement in which an AI provider does not keep a customer's prompts and outputs after a request is answered, usually offered only to approved customers and only for eligible API features.
τ-bench
τ-bench (tau-bench) is a benchmark that tests AI agents in simulated customer-service conversations, where a language model plays the user and the agent must use domain API tools while following written policy rules.