A Month with Midscene: Can AI Vision-Driven UI Automation Replace Playwright?

I once maintained a Playwright project with 200+ E2E test cases. Every time the product redesigned the UI — moved a button, added a modal — I’d spend half a day chasing broken selectors. The cumulative pain was real. Then I found Midscene, and for the first time I realized UI automation could just be… talking to it like a person.

Project Background

Midscene is an open-source AI-driven UI automation framework from ByteDance’s web-infra-dev team, sitting at 12.8k stars on GitHub. The core idea: replace selectors with natural language. Instead of page.click('#submit-button'), you tell it “click the submit button at the bottom of the form,” and a vision model figures out where to click.

It supports Web, Android, and iOS, written in TypeScript, last updated late April 2026. Under the hood it leverages vision-capable models like GPT-4V, Claude, and Gemini to “see” the page.

What Genuinely Impressed Me

1. Tests Read Like Conversation

Here’s a classic login test in Playwright:

await page.goto('https://example.com');
await page.locator('input[name="email"]').fill('[email protected]');
await page.locator('input[name="password"]').fill('pass123');
await page.locator('button[type="submit"]').click();
await expect(page.locator('.welcome')).toBeVisible();

The Midscene equivalent:

await agent.aiAction('Enter [email protected] in the email field, pass123 in the password field, then click login');
await agent.aiAssert('A welcome message is shown');

Readability skyrockets, and as long as the page is still visually recognizable, the script doesn’t need to change. This is what “lower test maintenance cost” actually looks like.

2. Visual Assertions Are Powerful

The aiAssert API is magical. I can write “the product list shows at least 5 items” or “the cart icon has a red number badge,” and it actually invokes the model to look at the page and judge.

Previously this kind of assertion required either heavy DOM querying or screenshot comparison. Now it’s a single line.

3. Built-In Visual Reports

After a test run, you get a beautiful HTML report with screenshots of every step, the AI’s “reasoning” (why it picked a particular element), and failure captures. Debugging is far more intuitive than reading trace files.

4. Plays Nice With Existing Tooling

Crucially: Midscene doesn’t reinvent the wheel. It’s built on top of Playwright, so existing Playwright projects can migrate gradually — keep traditional selectors where they work, swap in AI for volatile parts.

How to Get Started

npm install @midscene/web @midscene/playwright

Integrate into your Playwright tests:

import { test } from '@playwright/test';
import { PlaywrightAgent } from '@midscene/web/playwright';

test('shopping flow', async ({ page }) => {
  const agent = new PlaywrightAgent(page);

  await page.goto('https://shop.example.com');
  await agent.aiAction('Search for "iPhone 15"');
  await agent.aiAction('Add the first search result to the cart');
  await agent.aiAssert('Cart count shows 1');
});

Configure the model via env vars:

export OPENAI_API_KEY="sk-..."
export MIDSCENE_MODEL_NAME="gpt-4o"

If you’d rather not use OpenAI, Claude, Gemini, and Qwen-VL are all supported. Docs are clear on this.

Pain Points You Should Know

API costs aren’t trivial. Every AI action sends a screenshot to the vision model, and token consumption is heavier than you’d think. A full 50-test suite costs me about $1.50-$2.00 per run. If you run on every CI commit, monthly costs will hit $1k+.

It’s slower than pure Playwright. Each AI action waits for model inference — typically 3-8 seconds slower per step than a selector. A 2-minute test suite might balloon to 15 minutes with Midscene.

The AI occasionally fumbles. On complex pages or with ambiguous instructions (“click the top-right button” when there are two top-right buttons), the AI may pick wrong. The fix is either being more specific or falling back to selectors.

Vision models are timing-sensitive. Pages with heavy async loading, skeleton screens, or animations can trip up the screenshot timing. Midscene has wait logic internally, but you’ll occasionally catch a “loading” frame that confuses the assertion.

OpenAI access is hard from China. If you’re in mainland China, the OpenAI API isn’t directly accessible — you’ll need a proxy or to switch to Qwen-VL / GLM-4V. The latter work but with slightly worse accuracy.

How It Compares

Pure Playwright/Cypress: Fast, cheap, stable — but maintenance is brutal once your test count grows.

Selenium + traditional CV: Old-school, but OCR/template matching doesn’t approach the robustness of vision LLMs.

TestRigor, Mabl: Commercial AI testing platforms, feature-rich but expensive ($500-$5000/month). Midscene is the open-source alternative for budget-conscious teams.

Anthropic Computer Use API: Similar idea but desktop-focused and currently in beta. Midscene is more mature for web/mobile.

Who It’s For

• Teams maintaining mid-sized E2E test suites, especially with frequent UI changes.
• Startups and small teams without dedicated QA — PMs or developers can write tests in plain English.
• Teams wanting to lower the bar to writing tests — newcomers don’t have to learn selector syntax first.

If your project is performance-critical (running thousands of concurrent tests) or budget-tight, sticking with Playwright is more realistic.

Bottom Line

Midscene isn’t trying to replace Playwright — it’s adding an AI layer on top. The improvement in maintenance cost and accessibility is qualitative; the trade-off is execution speed and cost.

My current strategy: critical smoke tests in Midscene (high readability, low maintenance), regression tests in Playwright (fast, cheap). Combined, my team’s test maintenance time has dropped by roughly 60%.

12.8k stars, ByteDance backing, actively maintained — I expect this project will only grow.

GitHub: https://github.com/web-infra-dev/midscene

---

About the Author

Liudingyu is a full-stack developer and heavy GitHub user. With 900+ starred repos over the past 3 years, this site only covers tools I’ve actually used or deeply researched.

📧 Found a great tool to recommend? Email [email protected]