VS Code extension

Apify Autoplans, an AI web scraping assistant

Apify Autoplans is an AI scraper code generator for VS Code. It drives a real browser on your machine to work out how a page is built — which selectors hold the data, how the next page is reached, which XHR call the page is actually reading from — and then writes the scraper from what the browser saw rather than from a guess about the markup.

The browsing half runs locally, so a site you are already logged into stays logged in. The model half runs on autoplans.dev, which holds the OpenRouter key and meters the tokens, so there is no provider key to paste into your editor.

Install from the Marketplace Read the docs

Install it

1
Install the extension
From the Marketplace, or from a terminal. It needs VS Code 1.95 or newer.
```
code --install-extension AutoPlansDev.apify-autoplans-vscode
```
2
Give it a key
Run "Apify Autoplans: Set API Key" from the Command Palette and paste a key from your dashboard; the browser tools read that one. The Actor Agent has its own sign-in, which opens autoplans.dev, mints a key there and hands it back to the editor. Both land in VS Code SecretStorage, not in settings.json.
```
Ctrl+Shift+P  Apify Autoplans: Set API Key
Ctrl+Shift+P  Apify Autoplans: Agent: Sign in with Browser
```
3
Let the Python environment build
The first browser tool call creates a virtual environment inside the extension and installs python/requirements.txt — browser-use, playwright, langchain-core, pydantic, python-dotenv. It needs python3 on PATH; setup.sh does the same by hand and also downloads Chromium, which the automatic path does not.
```
# inside the installed extension, e.g.
# ~/.vscode/extensions/autoplansdev.apify-autoplans-vscode-0.1.3/python
./setup.sh
```
4
Let the editor call the browser tools
The extension contributes five language model tools. In the Chat view you can reference one directly with # and its reference name, or leave the model to pick one while it works.
```
#analyzeHtml https://example.com/products
#learnFromWebsite https://example.com/products how are prices loaded
```
5
Open an Actor Agent session
The status bar carries an "Actor Agent" button; the session runs a coding agent in the folder you have open, carrying the Actor-building rules the extension ships. The agent runtime is a binary too large to ship inside an extension, so it is installed once from npm — "Apify Autoplans: Agent: Install the Coding Agent" runs exactly this line for you.
```
npm install -g autoplans-agent
```

What it does

A real browser, on your machine

Browser Use automation runs as a Python subprocess next to your editor: the browser-use library drives a local Chromium you can watch. Because it is local browser automation, a page behind a login is read from your own session, and cookies never leave the machine.

It watches the network, not only the DOM

Playwright request and response hooks record every xhr and fetch call the page makes, with headers, parameters and bodies; JSON responses under 10,000 characters are kept whole. That is how a hidden API endpoint turns up — the one worth scraping instead of the rendered table.

Five tools the editor can call

apify_scrape_website, apify_learn_from_website, apify_create_scraper, apify_analyze_html and apify_test_scraper are contributed as language model tools — the third analyses the site first and hands back what it found, the fourth identifies selectors, patterns and data extraction strategies. Reference one in a prompt as #scrapeWebsite, #learnFromWebsite, #createApifyScraper, #analyzeHtml or #testScraper.

@apify in the Chat view

A chat participant for the code half: it reads what you asked for — scrape, create, analyse, test — and streams back Actor code, a selector strategy or a review of the code you have selected in the editor. It asks the editor for a gpt-4o-family chat model and says so when none is available, so nothing on that path is billed to Autoplans.

Analysis in a fixed shape

The HTML analysis comes back under set headings every time: API endpoints and AJAX calls, input schema, navigation flow, target data selectors, scraping strategy, edge cases. It deliberately returns no code — the model writes the code from it, so you can read the evidence first.

Pagination and logins are detected, not assumed

The pagination pass answers a fixed schema: type (numbered, next and previous buttons, infinite scroll, load more), the next-page selector, the URL pattern for page numbers, items per page, and whether scrolling is what triggers new content. The login pass reports the username, password and submit selectors, whether a CSRF token is in play, and what else the form demands.

An agent that already knows the platform

Actor Agent sessions load always-on Actor-building rules, a documentation skill pointing at the live Apify and Crawlee references, and two specialists: an actor-builder that produces the Actor and a scraper-debugger that starts from a symptom — empty dataset, blocks, OOM, dead selectors — and works back to a cause.

Credentials in SecretStorage

Both keys are held in VS Code SecretStorage rather than in a settings file, and the agent runtime is started on 127.0.0.1 with a password generated at each start, so nothing is exposed to the webview or to other processes.

What Browser Use does here

browser-use is a Python framework for driving a browser with a language model. The extension ships it under python/, spawns it as a subprocess and talks to it over JSON on stdout, one command per run: analyze_html, learn_from_website, scrape_website, test_scraper, detect_pagination, detect_login_flow, login, smart_scroll, extract_product_schema, extract_table_data, ai_click, smart_wait, take_screenshot, monitor_network, save_session, load_session and run_task.

Before the agent starts, Playwright listeners are attached to the page so that xhr and fetch traffic is captured as it happens. Completed calls are summarised in batches of five against a schema that names each endpoint and its purpose, describes how data flows into the page, says whether authentication is required, and recommends calling the API or scraping the DOM. A final pass summarises the lot.

None of this is hidden behind the editor. The same script runs from a terminal, which is the fastest way to see what the analysis actually returns for a site you care about.

export AUTOPLANS_API_KEY=apk_live_...
./venv/bin/python browser_agent.py analyze_html '{"url": "https://example.com/products"}'

Where the model calls go, and where the key lives

The browser agent is pointed at autoplans.dev rather than at a provider: it builds an OpenAI-compatible client with base_url https://autoplans.dev/api/v1 and your Autoplans API key. That endpoint authenticates the key, checks it carries the ai:execute scope, counts the tokens against your account and forwards the call to OpenRouter with the server's key. Requests are limited to an allowlist of models, so a caller cannot name an expensive one; the browser agent asks for google/gemini-2.5-flash-lite by default.

The practical result is that your machine never holds an OpenRouter key. It holds one Autoplans key, in SecretStorage, and you can revoke it from the dashboard. If you would rather point the tools somewhere else, DEFAULT_MODEL and API_BASE_URL are read from python/.env, and the extension's own proxy endpoint is a setting.

# python/.env
DEFAULT_MODEL=google/gemini-2.5-flash-lite
API_BASE_URL=https://autoplans.dev/api/v1

What generating an Apify actor produces

Apify is the platform these scrapers run on: an Actor is a program packaged as a Docker image that takes JSON input and writes rows to a dataset. There is no fixed template in the extension — the files come out of the agent following the rules it carries, and those rules are specific about what a finished Actor contains.

The crawler is Crawlee, and Cheerio is the default: an HTTP crawler is an order of magnitude cheaper than a browser one, so the rules forbid reaching for PlaywrightCrawler until a fetch of the page proves that JavaScript rendering is unavoidable. Requests go through the request queue rather than an array, so a run deduplicates and resumes after a migration; list and detail pages are labelled with userData.label and split across routed handlers, which is where pagination stops being a source of quiet bugs. Results are pushed with Actor.pushData() as they are found, not accumulated in memory.

Deployment stays yours. The extension never asks for an Apify token and never touches your account — you run apify run locally and apify push when you are satisfied.

.actor/actor.json         name, title, version, memory, timeout
.actor/input_schema.json  startUrls, maxItems, proxyConfiguration
Dockerfile
README.md                 the Store listing: sample output, input table, limits

Anti-blocking and pagination, as implemented

Anti-blocking here is a rule the agent follows when it writes the Actor, not a proxy network the extension runs. Every generated Actor takes its proxy configuration from Apify rather than hardcoding one, and exposes it in the input schema so that whoever runs it can switch to residential addresses when a datacentre address starts collecting 403s. Fingerprinting and the session pool are Crawlee's, not a hand-written user-agent string; retries and concurrency are Crawlee's autoscaled pool rather than a loop. The debugging rules read a 403 or 429 storm as missing proxy configuration or a disabled session pool, which is usually what it is.

Pagination is handled twice over. Before any code is written, the browser reports what the page actually does — numbered links, a next button, a load-more button or an infinite scroll, with the selector or URL pattern that drives it, and a smart scroll pass can load a set number of items to confirm. After that it is structural: the list handler enqueues, the detail handler extracts, and the queue keeps the two honest.

const proxyConfiguration = await Actor.createProxyConfiguration();

Settings, logs and what runs where

Five settings are contributed. proxyEndpoint is the backend the extension talks to; agentEndpoint is the autoplans.dev endpoint the Actor Agent signs in against; agentRuntimePath overrides the lookup for the agent binary when it is installed somewhere unusual; agentOpenIn decides whether a session opens in its own window or an editor tab; enableDebugLogs turns on the verbose log.

There are two output channels to read when something goes wrong. "Apify Autoplans" carries the extension side, including what the Python browser prints to stderr when a command runs it, and "Autoplans Agent" carries the runtime — "Apify Autoplans: Agent: Show Log" opens it. If a session hangs, "Agent: Restart" stops the runtime and the next session starts a fresh one.

{
  "apifyAutoplans.proxyEndpoint": "https://autoplans.dev",
  "apifyAutoplans.agentEndpoint": "https://autoplans.dev/api/v1/mcp",
  "apifyAutoplans.agentOpenIn": "window",
  "apifyAutoplans.enableDebugLogs": false
}

Questions

Can I scrape a website without writing selectors by hand?

You can get them without opening DevTools yourself: the browser visits the page, and the analysis comes back with a selector per field, the container that groups them, and whether the value is text, an attribute or an href. You should still read what it found before shipping it — a selector that matched once on a page you were logged into is not yet a scraper that runs unattended.

Do I need GitHub Copilot for this?

For two of the three surfaces, yes. Chat participants and language model tools are VS Code chat features: the participant asks the editor for a chat model and streams the answer, and tools are invoked by agent mode or referenced with #, which means a chat extension that provides models has to be installed. The Actor Agent needs none of that, since it runs its own agent runtime locally, and neither do the commands, which drive the Python browser directly.

Is this a Playwright scraper generator, or does it write Puppeteer code?

Neither by default. The rules the agent carries default to Crawlee's CheerioCrawler over plain HTTP and only move to PlaywrightCrawler when a fetch proves the page needs a browser to render. Crawlee also ships a PuppeteerCrawler; these rules do not use it, since Playwright covers the same ground. Playwright does run on your machine, though: it is what browser-use drives while it is learning the site.

How do I generate an Apify actor with AI here?

Analyse the target first, so the agent is working from what the page really contains, then open an Actor Agent session and describe what you want scraped. The agent writes the Actor into the folder you have open — actor.json, input schema, Dockerfile, README and the crawler — and is told to verify with a local run and to report what it actually checked rather than declaring it done.

Where does the OpenRouter key live?

On the server. Model calls go to autoplans.dev, which forwards them to OpenRouter with its own key, so there is nothing to paste into VS Code beyond an Autoplans API key, and that key sits in SecretStorage. It has to carry the ai:execute scope; the browser sign-in mints one that does.

Does it need my Apify account?

No. Nothing in the extension reads an Apify token or calls the Apify API — the output is source code in your workspace. Deploying it is a separate, deliberate step you take with the Apify CLI, against whichever account you choose.

How is this different from the other Autoplans extension for VS Code?

Autoplans publishes two. The one at /vscode is for projects, tasks and the task map inside your editor. This is the web scraping assistant VS Code extension: the same coding agent, pointed at Actor building, with a local browser and the Apify rules attached. Installing both is fine; they sign in the same way and hold separate keys.

What if Python is missing, or the first run is slow?

The browser tools need python3 on PATH — 3.11 or newer is what setup.sh expects — and the first call spends a while building the virtual environment and installing the requirements before anything is scraped. If it then fails for want of a browser, run setup.sh once: that is the step which downloads Chromium. After that it is the browsing itself that takes the time, since the agent waits for pages like a person does. Turn on apifyAutoplans.enableDebugLogs and read the "Apify Autoplans" output channel when a run ends without data.

Autoplans for VS Code Documentation Pricing

Start with one project idea.

Describe it, get a backlog, and point your agent at it. Free, no card.

Start free