Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

AI news & research

Research ·

miHoYo's Gamer agent plays 3D games from screen pixels with keyboard and mouse

Gamer, a preprint from miHoYo's Honkai AI team and university partners, is a vision-language-action model that sees only the screen and acts through keyboard and mouse. It was trained on tens of thousands of hours of gameplay, then on about 1,000 hours of annotated trajectories and reinforcement learning. Over roughly 50 hours of tests it played on its own for more than 95% of the time.

A computer-use agent that can only click buttons in tidy web forms is of limited use. Games are a harder test: busy 3D scenes, quests that last hours, and controls that need precise timing. Gamer, a preprint from the Honkai AI team at miHoYo with several Chinese universities and the National University of Singapore, is a model that plays them using only what appears on screen and the same keyboard and mouse a person uses.

The training happens in stages. First the team continued pre-training on tens of thousands of hours of device-level keyboard and mouse recordings from games and simulated environments, so the model learns how actions change what it sees. They then built about 1,000 hours of instruction data from curated play of Honkai: Star Rail, using a language model to add task goals, reasoning at key decisions and running memory summaries. A step they call sparse-thinking tuning cuts unnecessary reasoning, and a multi-turn reinforcement learning method handles long sessions, limited context windows and rare rewards. The base model is Qwen3-VL-8B.

Speed is as important as accuracy, since the game keeps moving while the model thinks. The team built Streaming-vLLM, a serving engine that runs many game sessions at once, reuses cached state between turns and keeps response times low, tested on Nvidia H200 and A800 hardware. Over roughly 50 hours of evaluation, the agent played on its own for more than 95% of the time. The authors report performance comparable to skilled players on several 3D game tasks and competitive scores on standard 2D game benchmarks.

The design choice matters beyond games. Using raw pixels and ordinary input devices, rather than a game's internal data, is the same bet that general computer-use agents such as ChatGPT Work and Claude in Chrome make. Gamer suggests that large amounts of plain keyboard and mouse logs are useful training data for that kind of agent.

The caveats are significant. This is a preprint and not peer reviewed. Most of the detailed evaluation is on miHoYo's own game, where the team has unusual access to data. "Comparable to skilled players" is the authors' own description, and the paper notes that reasoning remains a bottleneck in games that need deeper strategy. Serving costs also limit wider use for now.

Our take: One keyboard-and-mouse interface for every game is the right bet for general computer-use agents. The claim of skill comparable to good players comes from a game publisher testing largely on its own title, Honkai: Star Rail, so independent benchmarks matter.

Source: arXiv (Hugging Face Daily Papers)

More stories