> ## Documentation Index
> Fetch the complete documentation index at: https://docs.eclipseproxy.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Browser automation (Selenium, Puppeteer, Playwright, Scrapy)

> Using EclipseProxy with headless browsers and scraping frameworks

Working examples for the four major automation tools. Tabs sync across the page: your library choice carries through every section.

## Basic setup

<Tabs>
  <Tab title="Playwright">
    Playwright has the cleanest proxy API.

    <CodeGroup>
      ```javascript Node theme={"dark"}
      const { chromium } = require('playwright');

      const browser = await chromium.launch({
        proxy: {
          server: 'http://HOST:PORT',
          username: 'USER',
          password: 'PASS',
        },
      });
      const page = await browser.newPage();
      await page.goto('https://api.ipify.org');
      console.log(await page.textContent('body'));
      await browser.close();
      ```

      ```python Python theme={"dark"}
      from playwright.sync_api import sync_playwright

      with sync_playwright() as p:
          browser = p.chromium.launch(proxy={
              "server": "http://HOST:PORT",
              "username": "USER",
              "password": "PASS",
          })
          page = browser.new_page()
          page.goto("https://api.ipify.org")
          print(page.content())
          browser.close()
      ```
    </CodeGroup>

    **Per-context proxy** (multiple proxies in one browser):

    ```javascript theme={"dark"}
    const context = await browser.newContext({
      proxy: { server: 'http://HOST:PORT', username: 'U', password: 'P' },
    });
    ```
  </Tab>

  <Tab title="Puppeteer">
    ```javascript theme={"dark"}
    const puppeteer = require('puppeteer');

    const browser = await puppeteer.launch({
      args: ['--proxy-server=http://HOST:PORT'],
    });
    const page = await browser.newPage();
    await page.authenticate({ username: 'USER', password: 'PASS' });
    await page.goto('https://api.ipify.org');
    console.log(await page.content());
    await browser.close();
    ```

    **Stealth**: significantly reduces bot detection:

    ```bash theme={"dark"}
    npm install puppeteer-extra puppeteer-extra-plugin-stealth
    ```

    ```javascript theme={"dark"}
    const puppeteer = require('puppeteer-extra');
    const StealthPlugin = require('puppeteer-extra-plugin-stealth');
    puppeteer.use(StealthPlugin());
    ```
  </Tab>

  <Tab title="Selenium">
    Chrome and Firefox don't accept user:pass in the `--proxy-server` flag. Easiest option: `selenium-wire`.

    ```bash theme={"dark"}
    pip install selenium-wire
    ```

    ```python theme={"dark"}
    from seleniumwire import webdriver

    options = {
        "proxy": {
            "http": "http://USER:PASS@HOST:PORT",
            "https": "http://USER:PASS@HOST:PORT",
            "no_proxy": "localhost,127.0.0.1",
        }
    }

    driver = webdriver.Chrome(seleniumwire_options=options)
    driver.get("https://api.ipify.org")
    ```

    **Without selenium-wire** (IP-whitelist auth only):

    ```python theme={"dark"}
    from selenium import webdriver
    from selenium.webdriver.chrome.options import Options

    opts = Options()
    opts.add_argument(f"--proxy-server=http://HOST:PORT")
    driver = webdriver.Chrome(options=opts)
    ```

    Add your machine's IP to the dashboard whitelist first.

    **undetected-chromedriver** for sites that detect Selenium:

    ```bash theme={"dark"}
    pip install undetected-chromedriver
    ```

    ```python theme={"dark"}
    import undetected_chromedriver as uc

    options = uc.ChromeOptions()
    options.add_argument(f"--proxy-server=http://HOST:PORT")
    driver = uc.Chrome(options=options)
    ```

    For credentialed proxies with `uc`, layer it with selenium-wire.
  </Tab>

  <Tab title="Scrapy">
    **Per-request:**

    ```python theme={"dark"}
    def start_requests(self):
        yield scrapy.Request(
            url="https://api.ipify.org",
            meta={"proxy": "http://USER:PASS@HOST:PORT"},
        )
    ```

    **Default proxy via middleware** in `settings.py`:

    ```python theme={"dark"}
    DOWNLOADER_MIDDLEWARES = {
        "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 110,
        "myproject.middlewares.ProxyMiddleware": 100,
    }
    ```

    ```python theme={"dark"}
    class ProxyMiddleware:
        def process_request(self, request, spider):
            request.meta["proxy"] = "http://USER:PASS@HOST:PORT"
    ```

    **Rotating per request:**

    ```python theme={"dark"}
    import uuid

    class RotatingProxyMiddleware:
        def process_request(self, request, spider):
            sid = uuid.uuid4().hex[:8]
            request.meta["proxy"] = (
                f"http://USER-session-{sid}-lifetime-1:PASS"
                f"@HOST:STICKY_PORT"
            )
    ```

    **Tune settings:**

    ```python theme={"dark"}
    CONCURRENT_REQUESTS = 32           # match your thread plan
    DOWNLOAD_DELAY = 0
    RETRY_TIMES = 3
    DOWNLOAD_TIMEOUT = 30
    ```
  </Tab>
</Tabs>

## Common issues

<Accordion title="Site detects automation">
  Use stealth plugins (puppeteer-extra-plugin-stealth, playwright-extra) or undetected-chromedriver for Selenium.
</Accordion>

<Accordion title="WebRTC reveals my real IP">
  Use stealth plugins or set browser flags to disable WebRTC.
</Accordion>

<Accordion title="Multiple Chrome instances conflict">
  Use unique `user-data-dir` per instance.
</Accordion>

<Accordion title="Same IP every request despite rotating">
  Connection reuse. Force a new session ID per request or use a fresh browser context.
</Accordion>