Proxies for Scrapy Spiders - We Handle 403 Errors|
Wire BirdProxies into DOWNLOADER_MIDDLEWARES. Rotating residential for stateless crawls, static ISP for session-bound spiders, and an honest look at what a proxy fixes when a spider still gets blocked.
Works with the tools your spiders already use
Built for Scrapy Spiders
Rotating residential for wide crawls, static ISP for anything session-bound.
Large-Scale Spider Crawls
A CrawlSpider following pagination and category links across thousands of pages puts all of that volume through one IP unless you rotate it. Rotating residential gives each request (or each batch) a different address instead of building up a request history on one IP.
Scrapy + Splash or Playwright for JS-Rendered Pages
Plain Scrapy only sees the initial HTML response. For targets that render content client-side, pair Scrapy with Splash or scrapy-playwright and route that rendering layer through the same proxy so the rendered request and the proxy IP match, instead of rendering from one address and scraping from another.
E-Commerce and Price-Monitoring Spiders
Scheduled spiders that re-check the same product or listing pages on a recurring interval are exactly the pattern anti-bot systems are built to catch. A rotating pool spreads that recurring traffic instead of hammering the target from a single static address every run.
Geo-Targeted Crawls
Append a country tag to the proxy username to crawl a target the way a visitor from that country would see it - localized pricing, region-locked listings, or country-specific search results.
What a Scrapy spider needs
HTTP proxy support that drops straight into DOWNLOADER_MIDDLEWARES, a fresh IP per request when the crawl calls for it, and enough bandwidth headroom that a 36-page crawl does not quietly turn into a 3,600-page one without you noticing the bill.
Set Up in 5 Minutes
Wire BirdProxies into Scrapy's DOWNLOADER_MIDDLEWARES in four steps.
Install Scrapy
Standard Scrapy install. No extra package is needed for basic HTTP proxy support - Scrapy ships HttpProxyMiddleware in the core.
pip install scrapyWire the Proxy into DOWNLOADER_MIDDLEWARES
Scrapy already has a proxy hook: HttpProxyMiddleware reads request.meta['proxy'] (or the http_proxy/https_proxy env vars) and turns embedded credentials into a Proxy-Authorization header for you. Point it at BirdProxies and it works with zero extra middleware for a single static IP.
# settings.py BOT_NAME = "my_spider" DOWNLOADER_MIDDLEWARES = { "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 100, } # Retry on the status codes that usually mean "blocked", not just "down" RETRY_HTTP_CODES = [403, 429, 500, 502, 503, 504, 522, 524, 408] RETRY_TIMES = 3Rotate a Fresh IP Per Request
For a crawl where each request should look like a different visitor, set the proxy in a custom middleware instead of a single fixed meta value, so every request builds its own session-tagged username against BirdProxies rotating residential.
# middlewares.py import os import uuid class BirdProxiesRotatingMiddleware: def process_request(self, request, spider): session = uuid.uuid4().hex[:10] user = f"{os.environ['BIRD_PROXY_USER']}-session-{session}" request.meta["proxy"] = ( f"http://{user}:{os.environ['BIRD_PROXY_PASS']}" f"@residential.birdproxies.com:7777" ) def process_response(self, request, response, spider): # A block page often still returns 200 - check for it explicitly, # a status-code check alone will miss this if response.status == 200 and b"captcha" in response.body.lower(): spider.logger.warning(f"Block page on {request.url}, retrying with a new IP") new_request = request.copy() new_request.dont_filter = True del new_request.meta["proxy"] # forces process_request to assign a fresh one return new_request return responseSwitch to a Static IP for Session-Bound Spiders
A spider that logs in, then paginates behind that login, needs the same exit IP for the whole run - rotating mid-session on an authenticated crawl looks like a session hijack to the target. Use a BirdProxies static ISP proxy for that spider instead of the rotating one.
# settings.py - separate credentials for the login-bound spider ISP_PROXY = "http://YOUR_ISP_USER:[email protected]:7777" class LoginBoundSpider(scrapy.Spider): name = "account_crawl" def start_requests(self): yield scrapy.Request( "https://example.com/login", meta={"proxy": ISP_PROXY}, callback=self.parse_login, )
Sempre Ligado, Sempre Vencendo
Benefícios Que Mantêm Você na Frente
Proxies Seguros & Protegidos
Aproveite conexões privadas, sem restrições e ultra-rápidas sem complicações.
Velocidades Ultra-Rápidas
Muito lento? Muito tarde? Não com a gente. Aproveite conexões ultra-rápidas sempre!
Suporte 24/7
Dia ou noite, estamos aqui para ajudar você 24/7.
Contorne Restrições
403 Proibido! Não mais. Acesse qualquer site, em qualquer lugar com facilidade.
Scrapy already has a proxy hook built in (HttpProxyMiddleware). The part that actually matters is what you point it at, and being honest about what a proxy does and does not fix when a spider still gets blocked.
Scrapy is a Python framework for building spiders that crawl and extract structured data at scale, distinct from browser-automation tools like Selenium or Playwright and from lighter fetch libraries. It ships HttpProxyMiddleware in its core, which reads a proxy URL from request.meta['proxy'] (or the http_proxy/https_proxy environment variables) and turns embedded credentials into a Proxy-Authorization header automatically. BirdProxies plugs into that exact hook: rotating residential IPs for stateless crawls where each request should look like a different visitor, and static ISP IPs for spiders that carry a login or session across requests, where a rotating IP would look like a session hijack instead of normal browsing. Neither pool guarantees a spider gets past every anti-bot system on its own. Proxies fix IP reputation; they do not fix a leaking X-Forwarded-For header, a mismatched User-Agent, missing headers a real browser would send, or a target that fingerprints beyond IP address entirely. The setup steps on this page include a custom rotating middleware and a retry-on-block-page pattern, because 'the proxy isn't working' is very often something else in the request stack, not the IP.
What HttpProxyMiddleware Actually Does
Scrapy already reads request.meta['proxy'] and turns embedded credentials into a Proxy-Authorization header - no third-party package needed for HTTP.
Enable scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware in DOWNLOADER_MIDDLEWARES and set request.meta['proxy'] = 'http://user:pass@host:port'. That is the entire built-in integration. Everything past that - rotation per request, sticky sessions, retry-on-block logic - is a custom middleware you write on top of it, shown in the setup steps on this page.
Rotating vs Static, By Spider Not By Project
A stateless crawl wants a fresh IP per request. A login-bound spider wants the same IP for the whole run.
Mixing proxy types per spider is normal, not an edge case: a CrawlSpider pulling category and pagination links wants BirdProxies rotating residential so no single IP accumulates a long request history. A spider that logs in first and then paginates behind that session wants a BirdProxies static ISP IP, because a rotating address that changes mid-session reads to the target as a session hijack, not normal browsing.
Why a Configured Proxy Still Sees 403
A proxy fixes IP reputation. It does not fix a leaking header, a mismatched User-Agent, or a target that fingerprints beyond IP.
Reddit's r/scrapy has recurring threads of people running a paid proxy service against a strong anti-bot target and still getting banned, or getting 403 through a working proxy with a valid User-Agent. The usual causes: your real IP leaking through an X-Forwarded-For or Via header on a misconfigured proxy chain, an IP pool that was already flagged before you started using it, headers that do not match what a real browser sends, or a target that fingerprints TLS handshake and header order on top of IP. BirdProxies removes the IP-reputation cause. It does not by itself defeat every anti-bot system on the other end.
Splash and Playwright Need the Proxy Set On the Right Component
The proxy has to be set on whichever piece actually renders the request - Splash's own args, or the Playwright browser context, not just Scrapy's request.meta.
For scrapy-splash, pass the proxy through splash_args since Splash renders the page itself and Scrapy's HttpProxyMiddleware never touches that request. For scrapy-playwright, the proxy goes on the Playwright browser context. Setting it only in Scrapy's own request.meta while a rendering layer makes the actual request is the most common reason a proxy looks broken against a JS-heavy target when it is not.
Scrapy Proxy Setup Comparison
What a Scrapy project actually needs from a proxy provider: HTTP proxy support that drops into HttpProxyMiddleware with no extra package, and pricing that scales with a crawl instead of per-IP.
| Feature | BirdProxies | Bright Data | Webshare |
|---|---|---|---|
| HTTP proxy for HttpProxyMiddleware | Yes, no extra package | Yes, no extra package | Yes, no extra package |
| Residential at scale | €3.75/GB | $10.50/GB | $0.65/GB, smaller pool |
| Static ISP for session-bound spiders | Yes (€1.10-1.40/IP) | Yes ($2.10/IP) | Yes ($0.30/IP, smaller) |
| Concurrent request cap at proxy layer | None | None | Limited on lower tiers |
| Crypto payment | SOL/USDC/USDT/LTC | No | No |
| No KYC | Yes | No (enterprise) | Yes |
| Self-serve checkout | Yes | Partial | Yes |
Proxies Built for Scrapy
Starter
Test rotating residential with one spider
€5.00
/ GB
€5.00
Recarregue quando quiser
Production
Recurring crawls at moderate volume
€4.75
/ GB
€23.75
Recarregue quando quiser
Scale
High-volume multi-spider crawling
€4.50
/ GB
€45.00
Recarregue quando quiser
Enterprise
Multi-TB scraping operations
€3.75
/ GB
€375.00
Recarregue quando quiser
StarterTest rotating residential with one spider €5.00 / GB €5.00 Recarregue quando quiser | ProductionRecurring crawls at moderate volume €4.75 / GB €23.75 Recarregue quando quiser | ScaleHigh-volume multi-spider crawling €4.50 / GB €45.00 Recarregue quando quiser | EnterpriseMulti-TB scraping operations €3.75 / GB €375.00 Recarregue quando quiser | |
|---|---|---|---|---|
| Dados | 1 GB | 5 GB | 10 GB | 100 GB |
| Tamanho do Pool | 72M+ IPs | 72M+ IPs | 72M+ IPs | 72M+ IPs |
| Países | 170+ | 170+ | 170+ | 170+ |
| Tipo de Sessão | Rotativo/Fixo | Rotativo/Fixo | Rotativo/Fixo | Rotativo/Fixo |
| Desbloqueado | Garantido | Garantido | Garantido | Garantido |
What Actually Fixed the Blocks
Scoped, honest results from
Scrapy Proxy FAQs
Everything you need to know about using proxies with Scrapy
Scrapy already ships HttpProxyMiddleware. Enable it in DOWNLOADER_MIDDLEWARES and set request.meta['proxy'] = 'http://user:pass@host:port' (either directly, or from a custom middleware's process_request). No third-party package is required for a basic HTTP proxy - the credentials-in-URL format is handled for you and turned into a Proxy-Authorization header.
Let's start our journey with a personal gift for you ❤️
SCRAPY15