Browser automation (Selenium, Puppeteer, Playwright, Scrapy)

Working examples for the four major automation tools. Tabs sync across the page: your library choice carries through every section.

Basic setup

Playwright
Puppeteer
Selenium
Scrapy

Playwright has the cleanest proxy API.

const { chromium } = require('playwright');

const browser = await chromium.launch({
  proxy: {
    server: 'http://HOST:PORT',
    username: 'USER',
    password: 'PASS',
  },
});
const page = await browser.newPage();
await page.goto('https://api.ipify.org');
console.log(await page.textContent('body'));
await browser.close();

Per-context proxy (multiple proxies in one browser):

const context = await browser.newContext({
  proxy: { server: 'http://HOST:PORT', username: 'U', password: 'P' },
});

const puppeteer = require('puppeteer');

const browser = await puppeteer.launch({
  args: ['--proxy-server=http://HOST:PORT'],
});
const page = await browser.newPage();
await page.authenticate({ username: 'USER', password: 'PASS' });
await page.goto('https://api.ipify.org');
console.log(await page.content());
await browser.close();

Stealth: significantly reduces bot detection:

npm install puppeteer-extra puppeteer-extra-plugin-stealth

const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
puppeteer.use(StealthPlugin());

Chrome and Firefox don’t accept user:pass in the --proxy-server flag. Easiest option: selenium-wire.

pip install selenium-wire

from seleniumwire import webdriver

options = {
    "proxy": {
        "http": "http://USER:PASS@HOST:PORT",
        "https": "http://USER:PASS@HOST:PORT",
        "no_proxy": "localhost,127.0.0.1",
    }
}

driver = webdriver.Chrome(seleniumwire_options=options)
driver.get("https://api.ipify.org")

Without selenium-wire (IP-whitelist auth only):

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

opts = Options()
opts.add_argument(f"--proxy-server=http://HOST:PORT")
driver = webdriver.Chrome(options=opts)

Add your machine’s IP to the dashboard whitelist first.undetected-chromedriver for sites that detect Selenium:

pip install undetected-chromedriver

import undetected_chromedriver as uc

options = uc.ChromeOptions()
options.add_argument(f"--proxy-server=http://HOST:PORT")
driver = uc.Chrome(options=options)

For credentialed proxies with uc, layer it with selenium-wire.

Per-request:

def start_requests(self):
    yield scrapy.Request(
        url="https://api.ipify.org",
        meta={"proxy": "http://USER:PASS@HOST:PORT"},
    )

Default proxy via middleware in settings.py:

DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 110,
    "myproject.middlewares.ProxyMiddleware": 100,
}

class ProxyMiddleware:
    def process_request(self, request, spider):
        request.meta["proxy"] = "http://USER:PASS@HOST:PORT"

Rotating per request:

import uuid

class RotatingProxyMiddleware:
    def process_request(self, request, spider):
        sid = uuid.uuid4().hex[:8]
        request.meta["proxy"] = (
            f"http://USER-session-{sid}-lifetime-1:PASS"
            f"@HOST:STICKY_PORT"
        )

Tune settings:

CONCURRENT_REQUESTS = 32           # match your thread plan
DOWNLOAD_DELAY = 0
RETRY_TIMES = 3
DOWNLOAD_TIMEOUT = 30

Common issues

Site detects automation

Use stealth plugins (puppeteer-extra-plugin-stealth, playwright-extra) or undetected-chromedriver for Selenium.

WebRTC reveals my real IP

Use stealth plugins or set browser flags to disable WebRTC.

Multiple Chrome instances conflict

Use unique user-data-dir per instance.

Same IP every request despite rotating

Connection reuse. Force a new session ID per request or use a fresh browser context.

​Basic setup

​Common issues

Basic setup

Common issues