What is BrowserGym?
BrowserGym wraps a Playwright-driven Chromium browser as a Gymnasium environment, the reset-and-step interface reinforcement learning code already speaks. Each step hands the agent the page's DOM, its accessibility tree, a screenshot, the open tabs and a chat history, and takes back an action written as a short function call, such as clicking an element by its id, filling a field or going back. Action sets can be restricted per benchmark, and a task is just a Python class, so new benchmarks plug in by subclassing one base class.
The project is Apache 2.0 licensed Python from ServiceNow Research, with about 1,400 GitHub stars and a paper in Transactions on Machine Learning Research (2025). Ten benchmarks are wired in: MiniWoB, WebArena, WebArenaVerified, VisualWebArena, WorkArena, AssistantBench, WebLINX, OpenApps, TimeWarp and Knows. The last PyPI release, 0.14.3, dates from January 2026, but the main branch is active: Knows, a Google Workspace benchmark, landed in October 2026, and a fix for silently skipped WebArena resets followed on 28 October. AgentLab, a companion repo, runs agents across these benchmarks in parallel and feeds a shared leaderboard.
Is BrowserGym worth it?
BrowserGym's main contribution is agreement. Before it, every web benchmark shipped its own browser harness, observation format and action syntax, so comparing an agent across WebArena and MiniWoB meant writing glue code twice. BrowserGym puts them behind one Gymnasium interface, and that is why it has become the default substrate: pypistats counts about 447,000 downloads of browsergym-core in the last month, most of them from evaluation pipelines rather than individual users.
The observation design is sensible. Agents get the accessibility tree with element ids as well as the raw DOM and a screenshot, so a text-only model and a vision model can run the same task. Actions are high-level calls on those ids, with coordinate clicks available for vision agents, and each benchmark can declare which actions it allows.
The catch: the library is the easy part. WebArena needs its shopping, forum, GitLab, map and Wikipedia sites self-hosted, with a URL per site in environment variables, and its fuzzy-match grader calls an OpenAI model, which adds to the bill. WorkArena needs ServiceNow instances obtained through a gated Hugging Face form, and Knows needs a Google Cloud service account. The README itself warns that this is research software, not a consumer product. PyPI has also lagged the repo since January, so recent fixes need a source install, and the open-ended task points a real browser at live websites under the agent's control.
Our take: if you are building a browser agent and want numbers anyone else will believe, run them through BrowserGym. Just plan the evaluation infrastructure as carefully as the agent.
Where it earns its place
An ML researcher training a web agent
Report results on WebArena and MiniWoB that reviewers will accept
Install browsergym and Playwright's Chromium, stand up the WebArena sites with the provided setup scripts, then loop reset and step over the registered task ids. AgentLab can run the tasks in parallel and log traces.
An enterprise automation team evaluating agents for ServiceNow workflows
Measure how often an agent completes forms, lists and catalogue orders in the real interface
Request WorkArena instance access, then run the atomic and compositional task sets through BrowserGym. Results show where agents fail before any pilot touches production.
A benchmark author
Publish a new web task suite without writing a browser harness
Subclass AbstractBrowserTask with setup and validation logic and register the tasks as Gymnasium environments; agents built for other BrowserGym benchmarks run on it unchanged.
Before you commit
- Setup
- `pip install browsergym` then `playwright install chromium`; the open-ended task runs in minutes, while WebArena, VisualWebArena and WorkArena each need their own server or account setup first.
- Integrations
- Gymnasium, Playwright, AgentLab, the BrowserGym leaderboard on Hugging Face and DoomArena for injecting attacks into pages; the demo agent uses OpenAI models.
- Data and privacy
- Runs locally; page content goes to whichever model your agent calls. Benchmark sites are self-hosted or run on accounts you control. Apache 2.0 licence.
- Cost at scale
- Free software; the costs are servers for the self-hosted sites, model calls for the agent and, on WebArena, OpenAI calls for fuzzy-match grading. Totals are not published.
- Limits
- Each benchmark has its own setup; WebArena needs periodic full resets of its sites; latest PyPI release 0.14.3 trails the main branch; about 26 open issues and 29 open pull requests at last check.
What’s good
- One Gymnasium API across ten web-agent benchmarks
- Accessibility tree, DOM and screenshots in every observation
- New benchmarks need only one subclass
- Backed by ServiceNow Research, with AgentLab and a shared leaderboard
What’s not
- WebArena and WorkArena need heavy, separate server setup
- WebArena grading calls a paid OpenAI model
- PyPI release has lagged the main branch since January 2026
- Research tool; not hardened for production browsing
Best for
Researchers and agent teams who need reproducible, comparable scores for browser agents across the standard web benchmarks.
BrowserGym alternatives
BrowserGym: common questions
- Is BrowserGym worth it?
- BrowserGym is the plumbing most web-agent research now runs on; adopt it for the shared interface and budget a day for each benchmark's servers. We rate it 4.2 out of 5.
- How much does BrowserGym cost?
- Free and open source (Apache 2.0); you pay for model calls and for hosting the benchmark sites.
- What are the downsides of BrowserGym?
- WebArena and WorkArena need heavy, separate server setup; webArena grading calls a paid OpenAI model; pyPI release has lagged the main branch since January 2026; research tool; not hardened for production browsing.
- Who is BrowserGym best for?
- Researchers and agent teams who need reproducible, comparable scores for browser agents across the standard web benchmarks.
- How much does BrowserGym cost for a team or in production?
- Free software; the costs are servers for the self-hosted sites, model calls for the agent and, on WebArena, OpenAI calls for fuzzy-match grading. Totals are not published.
- Is BrowserGym safe for company data?
- Runs locally; page content goes to whichever model your agent calls. Benchmark sites are self-hosted or run on accounts you control. Apache 2.0 licence.
- What are the best BrowserGym alternatives?
- The closest alternatives we have reviewed are Browser Use, Stagehand and Inspect.
Answers checked Nov 13, 2026.
Make BrowserGym? Show this rating on your site
