Why am I still getting banned or 403 with a proxy configured?

A proxy fixes IP reputation. It does not fix everything else a target checks. Common causes people mistake for "the proxy is broken": your own IP leaking through an X-Forwarded-For or Via header on a misconfigured proxy chain, a thin or reused IP pool that was already flagged before you started, a User-Agent that does not match the rest of your header set, missing Accept-Language or Referer headers a real browser always sends, or a target with strong bot-detection (a WAF-protected real estate or ticketing site, for example) that fingerprints beyond IP alone - TLS handshake, header order, timing. BirdProxies removes IP-reputation as a cause. It does not by itself defeat every anti-bot system on the other end, and nobody selling proxies honestly should tell you it does.

Should I use rotating residential or a static ISP proxy for a Scrapy spider?

Rotating for anything stateless - a spider that hits many independent pages and does not carry a session between requests. Static ISP for anything stateful - logged-in crawls, multi-page checkout-style flows, or a target that fingerprints an IP changing mid-session. Mixing them per-spider (not per-project) is normal: one CrawlSpider on rotating residential, one login-bound spider on a static ISP IP.

Does Scrapy work with Splash or Playwright for JS-rendered pages through BirdProxies?

Yes, but the proxy has to be set on whichever component actually makes the request. For scrapy-splash, pass the proxy through Splash's own args (splash_args={"proxy": "..."}) since Splash renders the page itself and Scrapy's HttpProxyMiddleware never touches that request. For scrapy-playwright, set the proxy on the Playwright browser context, not on the Scrapy request. Setting it in only one place is the most common reason people think the proxy "isn't working" against a JS-rendered target - it is working, just not on the request that matters.

How many concurrent requests can I run through BirdProxies from Scrapy?

BirdProxies does not impose a fixed concurrent-connection cap at the proxy layer. In practice your real ceiling is Scrapy's own CONCURRENT_REQUESTS / CONCURRENT_REQUESTS_PER_DOMAIN settings and, more importantly, whatever rate the target site tolerates before it starts serving 429s or CAPTCHAs regardless of proxy. Turning on AUTOTHROTTLE_ENABLED is still worth doing - a clean IP pool does not mean a target's own rate limiting stops applying to you.

Can I use SOCKS5 proxies with Scrapy?

Scrapy's built-in HttpProxyMiddleware only understands http:// and https:// proxy URLs, not socks5:// - there is no native SOCKS5 support in core Scrapy. If a spider needs SOCKS5 specifically, you'll need a third-party download handler or a local SOCKS-to-HTTP bridge. BirdProxies serves proxies over HTTP directly, so for a plain Scrapy project HTTP is the simpler path unless something else downstream in your pipeline specifically requires SOCKS5.

How do I geo-target a Scrapy crawl?

Append a country tag to the proxy username before it goes into request.meta['proxy'] - for example YOUR_USER-country-de routes that request through a German residential IP. BirdProxies covers 195+ countries. Set it per-request in your rotating middleware if different spiders need different countries in the same run.

4.9Trustpilot

Proxies for Scrapy Spiders - We Handle 403 Errors|

Wire BirdProxies into DOWNLOADER_MIDDLEWARES. Rotating residential for stateless crawls, static ISP for session-bound spiders, and an honest look at what a proxy fixes when a spider still gets blocked.

Works with the tools your spiders already use

Built for Scrapy Spiders

Rotating residential for wide crawls, static ISP for anything session-bound.

Large-Scale Spider Crawls
A CrawlSpider following pagination and category links across thousands of pages puts all of that volume through one IP unless you rotate it. Rotating residential gives each request (or each batch) a different address instead of building up a request history on one IP.
Scrapy + Splash or Playwright for JS-Rendered Pages
Plain Scrapy only sees the initial HTML response. For targets that render content client-side, pair Scrapy with Splash or scrapy-playwright and route that rendering layer through the same proxy so the rendered request and the proxy IP match, instead of rendering from one address and scraping from another.
E-Commerce and Price-Monitoring Spiders
Scheduled spiders that re-check the same product or listing pages on a recurring interval are exactly the pattern anti-bot systems are built to catch. A rotating pool spreads that recurring traffic instead of hammering the target from a single static address every run.
Geo-Targeted Crawls
Append a country tag to the proxy username to crawl a target the way a visitor from that country would see it - localized pricing, region-locked listings, or country-specific search results.

What a Scrapy spider needs

HTTP proxy support that drops straight into DOWNLOADER_MIDDLEWARES, a fresh IP per request when the crawl calls for it, and enough bandwidth headroom that a 36-page crawl does not quietly turn into a 3,600-page one without you noticing the bill.

Set Up in 5 Minutes

Wire BirdProxies into Scrapy's DOWNLOADER_MIDDLEWARES in four steps.

1
Install Scrapy
Standard Scrapy install. No extra package is needed for basic HTTP proxy support - Scrapy ships HttpProxyMiddleware in the core.
```
pip install scrapy
```
2
Wire the Proxy into DOWNLOADER_MIDDLEWARES
Scrapy already has a proxy hook: HttpProxyMiddleware reads request.meta['proxy'] (or the http_proxy/https_proxy env vars) and turns embedded credentials into a Proxy-Authorization header for you. Point it at BirdProxies and it works with zero extra middleware for a single static IP.
```
# settings.py
BOT_NAME = "my_spider"

DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 100,
}

# Retry on the status codes that usually mean "blocked", not just "down"
RETRY_HTTP_CODES = [403, 429, 500, 502, 503, 504, 522, 524, 408]
RETRY_TIMES = 3
```

Rotate a Fresh IP Per Request

For a crawl where each request should look like a different visitor, set the proxy in a custom middleware instead of a single fixed meta value, so every request builds its own session-tagged username against BirdProxies rotating residential.

# middlewares.py
import os
import uuid

class BirdProxiesRotatingMiddleware:
    def process_request(self, request, spider):
        session = uuid.uuid4().hex[:10]
        user = f"{os.environ['BIRD_PROXY_USER']}-session-{session}"
        request.meta["proxy"] = (
            f"http://{user}:{os.environ['BIRD_PROXY_PASS']}"
            f"@residential.birdproxies.com:7777"
        )

    def process_response(self, request, response, spider):
        # A block page often still returns 200 - check for it explicitly,
        # a status-code check alone will miss this
        if response.status == 200 and b"captcha" in response.body.lower():
            spider.logger.warning(f"Block page on {request.url}, retrying with a new IP")
            new_request = request.copy()
            new_request.dont_filter = True
            del new_request.meta["proxy"]  # forces process_request to assign a fresh one
            return new_request
        return response

Switch to a Static IP for Session-Bound Spiders

A spider that logs in, then paginates behind that login, needs the same exit IP for the whole run - rotating mid-session on an authenticated crawl looks like a session hijack to the target. Use a BirdProxies static ISP proxy for that spider instead of the rotating one.

# settings.py - separate credentials for the login-bound spider
ISP_PROXY = "http://YOUR_ISP_USER:[email protected]:7777"

class LoginBoundSpider(scrapy.Spider):
    name = "account_crawl"

    def start_requests(self):
        yield scrapy.Request(
            "https://example.com/login",
            meta={"proxy": ISP_PROXY},
            callback=self.parse_login,
        )

Всегда ВКЛЮЧЕНО, всегда побеждаем

Преимущества, которые держат вас впереди

Безопасные и защищенные прокси

Наслаждайтесь частными, неограниченными и молниеносными соединениями без проблем.

Молниеносные скорости

Слишком медленно? Слишком поздно? Не с нами. Наслаждайтесь ультрабыстрыми соединениями каждый раз!

Мой IP пометили в Instagram