Scrapy Proxies That Beat 403 Errors|
Wire BirdProxies into DOWNLOADER_MIDDLEWARES. Rotating residential for stateless crawls, static ISP for session-bound spiders, and an honest look at what a proxy fixes when a spider still gets blocked.
Works with the tools your spiders already use
Built for Scrapy Spiders
What a downloader middleware needs from the proxy underneath it
Scrapy Proxies
Works with HttpProxyMiddleware
Scrapy already ships a proxy hook, so pointing request.meta at the gateway is enough for a single static address with zero extra packages.
A middleware for fresh IPs
A custom downloader middleware assigns a new session-tagged username per request, so a CrawlSpider fanning out across pages is not on one address.
Retry codes that mean blocked
Wire 403, 429 and 503 into RETRY_HTTP_CODES so a blocked request gets a fresh address on retry instead of failing the whole crawl.
A static IP for logged-in spiders
Route a login-bound spider through a static ISP address instead of the rotating pool, so an authenticated crawl does not look hijacked.
Made for Production Crawl Pipelines
What keeps a Scrapy spider crawling instead of stuck retrying blocks
One address per session-bound spider
A login-bound spider keeps one static IP for its whole run, so it never looks like a hijacked session mid-crawl.
Fast enough for high concurrency
Concurrent requests stay quick, so Scrapy's own concurrency settings decide the pace, not the proxy underneath it.
24/7 Support
A crawl stuck retrying blocks does not wait until Monday. Someone answers when the middleware needs a second look.
Past per-IP crawl limits
A fresh session-tagged address per request keeps one flagged IP from stalling the rest of the crawl.
Proxies Built for Scrapy
Starter
Test rotating residential with one spider
€5.00
/ GB
€5.00
Top up anytime
Production
Recurring crawls at moderate volume
€4.75
/ GB
€23.75
Top up anytime
Scale
High-volume multi-spider crawling
€4.50
/ GB
€45.00
Top up anytime
Enterprise
Multi-TB scraping operations
€3.75
/ GB
€375.00
Top up anytime
StarterTest rotating residential with one spider €5.00 / GB €5.00 Top up anytime | ProductionRecurring crawls at moderate volume €4.75 / GB €23.75 Top up anytime | ScaleHigh-volume multi-spider crawling €4.50 / GB €45.00 Top up anytime | EnterpriseMulti-TB scraping operations €3.75 / GB €375.00 Top up anytime | |
|---|---|---|---|---|
| Data | 1 GB | 5 GB | 10 GB | 100 GB |
| Pool Size | 72M+ IPs | 72M+ IPs | 72M+ IPs | 72M+ IPs |
| Countries | 170+ | 170+ | 170+ | 170+ |
| Session Type | Rotating/Sticky | Rotating/Sticky | Rotating/Sticky | Rotating/Sticky |
| Unbanned | Guaranteed | Guaranteed | Guaranteed | Guaranteed |
What Actually Fixed the Blocks
Scoped, honest results from developers running Scrapy in production
Scrapy Proxy FAQs
Everything you need to know about using proxies with Scrapy
Scrapy already ships HttpProxyMiddleware. Enable it in DOWNLOADER_MIDDLEWARES and set request.meta['proxy'] = 'http://user:pass@host:port' (either directly, or from a custom middleware's process_request). No third-party package is required for a basic HTTP proxy - the credentials-in-URL format is handled for you and turned into a Proxy-Authorization header.
Let's start our journey with a personal gift for you
SCRAPY15