What is SketchVLM?
SketchVLM is the code for a paper from Auburn University and Adobe, accepted to the EMNLP 2026 main conference, on getting vision-language models to point and draw the way a person explains a picture. The method needs no training: it lays a coordinate grid over the image, asks the model to answer and to emit SVG strokes in grid coordinates, and renders those strokes as an overlay that leaves the original image untouched and can be edited afterwards.
The repository runs the paper's experiments with Claude, GPT, Gemini or any model on OpenRouter, in single-turn mode or a multi-turn mode where the model sees the canvas after each stroke. It includes the prompts, grid and rendering code, and links to three new benchmarks on Hugging Face (maze navigation, connect-the-dots and a ball-drop physics task) and to a separate web demo you can try in the browser.
The code is Python, mostly in Jupyter notebooks and scripts, with no licence file.
Is SketchVLM worth it?
SketchVLM's idea is simple enough to copy in an afternoon: put a grid on the image, ask a frontier model for strokes in grid coordinates, and draw them. The payoff is an answer a person can check at a glance, a traced path through a maze or a dot on every counted object, instead of a sentence that might be wrong.
The headline numbers need reading closely. The abstract reports gains of up to 28.5 percentage points in accuracy and up to 1.48 times better annotation quality, but those are measured against image-editing and fine-tuned sketching baselines such as ViLaSR and ThinkMorph, not against the same model answering in text. Against plain Gemini-3-Pro the paper's own table is mixed: physics prediction on VPCT rises from 89.3 to 96.0 and counting from 93.0 to 94.5, while ball drop falls from 83.8 to 79.7 and maze from 99.3 to 98.0. GPT-5 gains on most tasks. Drawing boxes as free strokes lowered detection scores for both models.
Other findings are useful for anyone building on it. Multi-turn drawing was never better than single-turn and took about 5.92 times as many turns. The grid helps GPT-5 but badly hurts Gemini on connect-the-dots. The method does not work on small models such as Qwen2.5-VL-7B. Annotation quality was scored by a Gemini judge whose agreement with humans was moderate, a kappa of about 0.51 against 0.84 between people.
As software, it is a research release: 172 stars, 19 forks, two contributors, the last commit on 17 September 2026 and no licence, which by default means you may read the code but not legally reuse it. The datasets and the web demo are the parts most teams will get value from.
Where it earns its place
A product designer at a photo-editing app
Show users where the AI is pointing when it gives editing advice
Try the hosted demo with your own images, then reimplement the grid-and-SVG prompt in your app; the paper's multi-turn example walks a user through removing a background. Cost is the API calls per image.
A machine learning researcher working on visual reasoning
Benchmark a new VLM on tasks that need drawing, not just text
Download the maze, connect-the-dots and ball-drop datasets from Hugging Face and run collab_sketch_with_label.py with --llm openrouter and your model; results land in results/ for comparison with the paper's tables.
An ML engineer at a manufacturing inspection firm
Check whether a frontier model can mark and count defects reliably
Run single-turn mode with --adaptive-grid on a sample of labelled images, using GPT-5 or Gemini; the paper suggests counting improves slightly while box-style localisation may get worse, so measure both.
Before you commit
- Setup
- pip install -r requirements.txt, put ANTHROPIC_API_KEY, OPENAI_API_KEY or GOOGLE_API_KEY in .env, then run collab_sketch_with_label.py with a provider, model and dataset folder.
- Integrations
- Anthropic, OpenAI, Google Gemini and OpenRouter APIs; datasets on Hugging Face; a separate TypeScript web demo in LoganBolton/SketchVLM_Demo.
- Data and privacy
- Images and prompts go to the model provider you choose; nothing is hosted by the authors except the public demo.
- Cost at scale
- Only API spend; multi-turn mode used about 5.92 times as many turns as single-turn in the paper, with no accuracy benefit.
- Limits
- No licence; no releases; small open models do not follow the drawing instructions; free-form strokes are weaker than boxes for object detection.
What’s good
- Training-free and works with any strong API model
- Overlay is non-destructive, editable SVG
- Three new benchmarks published on Hugging Face
- Paper reports failures and limits honestly
- Live web demo to try without setup
What’s not
- No licence file, so the code cannot safely be reused
- Gains over the plain base model are small or negative on several tasks
- Fails on small open models like Qwen2.5-VL-7B
- Research scripts, not a library or API
Best for
Researchers and product teams exploring how a vision model could show its reasoning on an image, who want a tested prompt-and-grid recipe to start from.
SketchVLM: common questions
- Is SketchVLM worth it?
- SketchVLM is a clever, cheap way to make a vision model show its working on the image, worth reading and borrowing ideas from, but it is paper code without a licence and its gains are narrower than the abstract suggests. We rate it 3.3 out of 5.
- How much does SketchVLM cost?
- Code is public on GitHub with no licence file, so reuse rights are unclear. Running it costs the API calls to GPT-5, Gemini, Claude or OpenRouter models.
- What are the downsides of SketchVLM?
- No licence file, so the code cannot safely be reused; gains over the plain base model are small or negative on several tasks; fails on small open models like Qwen2.5-VL-7B; research scripts, not a library or API.
- Who is SketchVLM best for?
- Researchers and product teams exploring how a vision model could show its reasoning on an image, who want a tested prompt-and-grid recipe to start from.
- How much does SketchVLM cost for a team or in production?
- Only API spend; multi-turn mode used about 5.92 times as many turns as single-turn in the paper, with no accuracy benefit.
- Is SketchVLM safe for company data?
- Images and prompts go to the model provider you choose; nothing is hosted by the authors except the public demo.
Answers checked Nov 3, 2026.