Improving Crawling with an LLM (Without Replacing the Crawler)

August 8, 2026

I hate writing scrapers and crawlers. I hate the fact that, in 2026, crawlers still struggle with very, very basic things like filling out forms, logging into things, and avoiding tripping over their own extensions and tools. I wanted to try something a bit different.

Traditional crawlers are good at crawling. Give one a URL and it’ll fetch pages, parse links, process JavaScript, manage a queue, and keep going until it hits a limit or runs out of things to do.

However, traditional crawlers suck when we’re looking at websites that are applications or websites that are actually complicated.

1. Login. The login might be a normal form, a popup, an iframe, MFA, or a longer chain of redirects.
2. Safety. An application may expose logout, delete, revoke, rotate, reset, or purchase actions as ordinary links. To a crawler, /users/123/delete looks like another URL and leads to a bad time for everyone else.
3. Wizards. Some pages only exist after a specific sequence of actions.

Let’s be very clear that replacing the whole thing with AI is a horrible idea.

Ways to add an LLM to existing systems

Generally, we look at existing deterministic systems and talk about adding LLMs. Then we are talking about one of two options.

Fully autonomous crawling

Give an agent a browser and tell it to explore everything.

The appeal is obvious. In theory, the agent can deal with login, forms, JavaScript-heavy pages, and strange workflows. In practice, it can spend tokens rediscovering links a parser always handles. It may choose a different route on the next run.

There’s also the boring problem: this gets expensive in time and money. Using a model for work a parser can finish faster is a fancy way to wait longer and pay more.

AI-assisted crawling

Keep the normal crawler. Add a bounded LLM component that creates the missing inputs.

The crawl queue remains ordinary code. Parsing stays ordinary code too. Retries, concurrency, scope, and storage don’t need a model deciding what to do. The LLM handles authentication and safety-related browser choices, then passes artifacts to the deterministic system.

This way I only added nondeterministic activity where there are no alternatives today.

The architecture: Fuzzy layers

The basic flow looks like this:

1. Open a browser at the target or the supplied login URL.
2. Give the agent credentials, operator instructions, and a view of the current page.
3. Let it work through authentication with structured browser actions.
4. Require evidence that authenticated content is actually reachable.
5. Extract session state that the crawler can reuse.
6. Record blocked or destructive routes as exclusion rules.
7. Run the regular crawler with the recovered authentication and safety settings.

An operator can describe a login without writing a custom Playwright script for every site:

{
  "login_url": "https://example.com/login",
  "credentials": {
    "email": "{{env:APP_EMAIL}}",
    "password": "{{env:APP_PASSWORD}}"
  },
  "instructions": "Sign in and stop once the dashboard is visible."
}

Then submit the crawl with a limited discovery budget:

uv run tenzai-crawler create https://example.com \
  --auth-config-file ./auth.json \
  --discovery-max-rounds 2 \
  --discovery-max-actions 40 \
  --discovery-max-llm-pages 10

The current design is slightly more complicated. The current system has two bounded AI roles:

• Authentication action selection.
• Ambiguous workflow action selection during browser discovery.

How do we prove authentication?

One surprisingly fiddly problem is deciding if login worked.

New cookies or HTTP status codes are not useful, and doing regex searches on a website is bound to fail either with false positives or false negatives.

The agent has to show it can reach content that a clean browser context can’t. The verifier opens the same candidate URL in the authenticated browser context and a clean context. The authenticated side must reach a same-origin, non-login, non-denial state. Authentication is accepted only when the clean context is redirected or denied, shows a login challenge instead of authenticated controls, or receives 401/403 from an endpoint that succeeds in the authenticated context.

Safety cannot be static

The classic exclusion list for most crawlers has blocks for common words like logout, account deletion, reset, quit, and so forth. However, this is not enough, particularly in websites that include non-English language text and/or non-English language selectors. Furthermore, it doesn’t account for business logic specifically. Stuff like destroy, spoil, remove, etc., should not be clicked during crawling.

Ideally I would implement:

1. Crawl with the rules discovered by the auth agent.
2. Look at newly discovered pages and actions.
3. Use an LLM to add rules for dangerous behavior that were hidden earlier.
4. Continue from a known checkpoint.

An LLM helps here because the boundary is fuzzy. The word delete is easy to catch, however מחיקה, desconectar, quitter, abmelden are all potentially unsafe and most lists exclude those words (hah!). Worse when it involves application specific terms like “spoil”, “extract” and more.

The above sounds great, currently what is implemented is a single step of this loop.

Measuring if any of this helps

A demo can look great without telling us very much. The browser moves, text fills the terminal, and eventually somebody says “agentic.” Fun to watch, admittedly.

I measured four things based on test websites included in the repository along with crawler benchmarks mentioned later.

Coverage

How much of the application did the crawler reach?

On our test sites, the AI-assisted version found roughly 3x more coverage than the unauthenticated deterministic crawl.

Safety

Did the crawl stay away from actions we’d marked as blocked or destructive?

It avoided destructive URLs that only became visible after login.

Speed

How long did it take?

In evaluations included in the repository, the LLM version is 10x slower than crawling without the LLM.

Reliability

Would it work again?

That depends on the model, the page representation, the browser tools, and the instructions from the operator.

Depending on authentication sequence, I get occasional failed logins.

Still unsolved

Some obvious gaps.

Wizards and stateful workflows: Unlike authentication, workflows branch, require restarting from known pages (that have now changed) and more. Authentication has a clear finish line.

Safety: A ruleset built from one view can’t anticipate everything the crawler finds later. The ruleset needs to be extended during the run.

Existing ecosystem: No one needs yet another format. I need to support Burp Suite and Caido and other tools.

Try it out

The crawler is open source at github.com/TenzaiLabs/crawlee.

If you want to play with it, start with sites that were built for scraping. Please don’t begin with somebody else’s production application.

• Quotes to Scrape: JavaScript
• Quotes to Scrape: Login
• QScrape
• Scrape This Site

The actual lesson

Deterministic systems are still excellent at work with a clear answer. LLMs help when the input is messy and the interface expects a human to understand it.