AI web scraping: Playwright plus LLM extraction, with working code

AI web scraping uses a language model to pull structured data out of web pages, so you describe the fields you want instead of writing brittle CSS selectors for every site. This guide builds a Playwright plus LLM pipeline in Python with schema-validated output, then covers reliability, cost control, robots.txt and the legal limits, and using coding agents to maintain scrapers.

What is AI web scraping?

AI web scraping means using a language model in a scraping pipeline, most often to turn a rendered web page into structured data. Instead of writing a CSS selector for the price, another for the title and another for stock status, you give the model the page text and a schema, and it returns the fields as JSON.

The term covers three different approaches:

This guide focuses on the first two, which are repeatable and cheap enough for production. Browser agents get their own guide: AI browser automation.

When should you use an LLM instead of CSS selectors?

Use LLM extraction when the structure varies and plain selectors when it doesn't. Selectors are free to run and fully predictable; LLM extraction costs tokens per page but survives layout changes and works across sites you have never seen.

The hybrid is often best: run selectors first, and when a required field comes back empty, send the page to the model and alert yourself that the selectors need updating.

How does a Playwright and LLM scraping pipeline work?

A Playwright and LLM pipeline has five stages, and each one is a place where you can save money or catch errors.

1. Check permission. Read the site's robots.txt and terms; skip disallowed paths.
2. Fetch and render. Playwright loads the page in a real browser, so JavaScript-rendered content appears.
3. Clean. Strip scripts, styles, navigation and footers. Send the main content as text or simplified HTML, not the raw page.
4. Extract. Call the model with the cleaned content and a schema; get validated JSON back.
5. Validate and store. Check types and ranges, flag nulls, store with the source URL and fetch time.

Install the pieces:

pip install playwright anthropic pydanticplaywright install chromium

The playwright install step downloads the browser binaries, as described in the Playwright Python docs.

Code: extract product data with Playwright and Claude

This script renders a product page, reduces it to visible text, and asks Claude for a validated Product object using structured outputs. The client.messages.parse method takes a Pydantic model, sends its schema to the API, and returns the parsed object.

from typing import Optionalfrom anthropic import Anthropicfrom playwright.sync_api import sync_playwrightfrom pydantic import BaseModelclass Product(BaseModel):    name: str    price: Optional[float]          # None if no price is shown    currency: Optional[str]         # ISO code such as "EUR"    in_stock: Optional[bool]    shipping_time: Optional[str]    # as written on the pagedef page_text(url: str) -> str:    with sync_playwright() as p:        browser = p.chromium.launch()        page = browser.new_page(user_agent="AcmePriceBot/1.0 (contact@acme.example)")        page.goto(url, wait_until="domcontentloaded")        main = page.locator("main") if page.locator("main").count() else page.locator("body")        text = main.inner_text()        browser.close()    return text[:20000]  # cap input size to control costclient = Anthropic()def extract(url: str) -> Product:    response = client.messages.parse(        model="claude-haiku-5-5",        max_tokens=1024,        system=(            "Extract product data from the page text. Use only what the page states. "            "If a field is not on the page, return null. Never guess prices."        ),        messages=[{"role": "user", "content": f"URL: {url}\n\n<page>\n{page_text(url)}\n</page>"}],        output_format=Product,    )    return response.parsed_outputif __name__ == "__main__":    print(extract("https://shop.example.com/products/trail-tent-2p"))

Three choices in this code matter. Optional fields with an explicit "return null" rule stop the model from inventing values. Sending inner_text() of the main element instead of full HTML cuts tokens dramatically. And a descriptive user agent with contact details lets site owners reach you instead of blocking you.

How do you handle dynamic pages, pagination and rate limits?

Most scraping failures come from page mechanics rather than extraction: content that loads after the first paint, results split across pages, and servers that slow down or block clients that request too fast. Playwright handles the first two well if you tell it what to wait for.

• Wait for the content, not a timer. Replace fixed sleeps with page.wait_for_selector(".product-card") or a locator, so the script continues as soon as the data is there and fails clearly when it never appears.
• Find the data source. Many pages load their data from a JSON endpoint. Open the browser's network tab; if the listing comes from a public JSON response, reading it is faster and cleaner than rendering HTML, under the same terms and robots.txt rules.
• Paginate deliberately. Follow "next" links or page parameters up to a hard limit, and stop when a page returns no new items, so a site change can't send you into an endless loop.
• Back off on errors. On HTTP 429 or 503, wait and retry with increasing delays, and stop the run after a few failures instead of hammering the server.
• Reuse one browser. Launching a browser per page is slow; open one, reuse pages, and close it at the end.

The same locator and waiting skills apply to testing; our guide to AI test automation with Playwright covers them from the testing side.

How do you make LLM extraction reliable?

Make extraction reliable by constraining the output, allowing "not found", validating values and checking a sample by hand. Schema-constrained output guarantees valid JSON; it does not guarantee correct values.

• Allow nulls. A required field with no data on the page invites a made-up value.
• Be explicit about formats. "Price as a number without currency symbol", "date as YYYY-MM-DD".
• Validate after extraction. Prices above zero, currencies from a known list, dates in a plausible range.
• Ask for evidence on risky fields. Add a price_text field holding the exact string from the page, and compare it to the parsed number.
• Spot-check. Review 20 random records against the live page for every new site or schema change.
• Log the source. Store URL, fetch time and a hash of the cleaned text so you can re-extract later without refetching.

Keep a small set of saved pages with known correct output, and rerun them whenever you change the prompt, schema or model. It is the scraping equivalent of a test suite.

How do you keep AI scraping costs down?

The biggest lever is input size: most of a raw HTML page is markup the model doesn't need. After that, model choice, batching and selective use of the LLM do the rest.

Prices are from Anthropic's pricing page at the time of writing (October 2026). A worked example: a page trimmed to 5,000 tokens with a 200-token answer costs about $0.0006 on Haiku 5.5, or about $0.60 per 1,000 pages, before batch discounts. Our guide to reducing LLM API costs covers caching and model routing in more depth.

Is AI web scraping legal and ethical?

Scraping legality depends on your jurisdiction, the site's terms, the kind of data and what you do with it; adding AI doesn't change the rules. Respect robots.txt, don't overload sites, prefer official APIs, and treat personal data as regulated data.

The Robots Exclusion Protocol (RFC 9309) standardizes robots.txt and states that its rules "are not a form of access authorization", meaning robots.txt is a request from the site owner, not a security control. Following it is still the baseline for responsible scraping. Python's standard library can check it:

from urllib.robotparser import RobotFileParserrp = RobotFileParser()rp.set_url("https://shop.example.com/robots.txt")rp.read()agent = "AcmePriceBot"url = "https://shop.example.com/products/trail-tent-2p"print(rp.can_fetch(agent, url))   # False means skip this URLprint(rp.crawl_delay(agent))      # seconds to wait between requests, if set

The robotparser docs list the other helpers, such as request_rate().

Other rules of thumb:

• Rate-limit. One request every few seconds per site is a reasonable default; honor any crawl delay.
• Read the terms of service. Some sites forbid automated access or commercial reuse.
• Avoid personal data. Names, emails and profiles of people fall under laws such as GDPR, which require a legal basis even when the data is public.
• Don't republish copyrighted content. Extracting facts such as prices differs from copying articles or images.
• Don't bypass logins, paywalls or CAPTCHAs. That moves you from gray area to likely violation.

For commercial projects, get legal advice for the countries you operate in.

How can a coding agent build and maintain scrapers?

A coding agent such as Claude Code is good at the tedious parts of scraping: inspecting a page, writing selectors, handling pagination, adding retries, and fixing a scraper when a site changes its layout. Give it saved sample pages and expected output so it can verify its own work.

Write a scraper for the product listing at https://shop.example.com/tentsusing Playwright in scrapers/shop_example.py, following the structure ofscrapers/base.py. Check robots.txt first and respect crawl-delay.Handle pagination, and output Product records (see models.py).Save two listing pages and one product page to tests/fixtures/ and writetests that parse them. Run the tests and show me the output.Use CSS selectors; fall back to the LLM extractor only if a required field is empty.

Scrapers break when sites change, often at inconvenient times. An agent with your fixtures and tests can repair a broken scraper and open a pull request for review. If you want that work to continue while your laptop is closed, Orkio runs coding agents on an always-on cloud box with your own Claude account.

Your first AI scraper checklist

Work through this list for your first project:

• Confirm there is no official API or data feed for what you need
• Check robots.txt and the terms of service; note crawl delays
• Define a schema with optional fields and explicit formats
• Render with Playwright and send only main-content text
• Extract with structured outputs on a small model; validate every record
• Save fixtures with known correct output and rerun them after each change
• Rate-limit, set a descriptive user agent, and log source URLs
• Keep personal data out unless you have a legal basis

Once the data is flowing, our guide to AI for data analysis covers turning it into charts and reports you can trust.

Frequently asked questions