On 2026-10-24 a poster on r/webscraping described a scraper that had run green on GitHub Actions until 2026-10-22 and then failed on every run: "The site returns 200 from any non-GitHub IP, including with a plain python-requests user agent. From GitHub runners, it's 403 on every attempt. ... So it's not flaky, it's a rule against the runner IP range."
That diagnosis is right, and the method behind it is worth copying. They changed one variable (where the request came from), kept everything else fixed, and the 403 followed the location. Most "works here, 403 there" threads on r/scrapy change two things at once, the place and the client, and then blame whichever one they already suspected.
This post is about telling those two apart. If your scraper used to work and then stopped, read your scraper worked for weeks and now it is blocked instead.
Two different problems produce the same 403
A 403 that depends on where the code runs is almost always about the IP, and a 403 that depends on which library sends the request, from the same machine, almost never is.
| What changed | Same public IP? | Most likely cause |
|---|---|---|
| Laptop to GitHub Actions runner | No | The runner's cloud IP range is refused |
| Laptop to Docker on a cloud server | No | Hosting IP range, same as above |
| Laptop to Docker on the same laptop | Yes | A different TLS library in the image, or the proxy setting never reached the container |
| requests to Scrapy, same box, same proxy | Yes | TLS handshake, header set and order, User-Agent, cookies, request rate |
| Any IP to an IP in one country | No | A country lock |
Why GitHub Actions runners get 403 when your laptop does not
GitHub-hosted runners send traffic from cloud data center addresses, and GitHub publishes those ranges, so a site can refuse all of them with one rule.
The list sits at https://api.github.com/meta under the actions key. On 2026-10-31 it held 8,350 CIDR blocks. A firewall can load that list, or block the cloud provider's ASN outright, and every runner gets the same answer. Backoff does nothing against a rule like that. The poster's three attempts over roughly ten seconds came back "all 403". Waiting might work eventually; they noted the site "has blocked scrapers for months at a time before".
They were weighing a serverless forward proxy that keeps the job on Actions and only changes the egress IP. That moves the request to another cloud range, and cloud ranges tend to sit on the same blocklists. GitHub's larger runners can get static IPs on paid plans. Those are cloud addresses too.
What fixes it is sending only the scraping step through an address outside the hosting ranges. A static ISP proxy suits a CI job: one IP, identical on every run, announced under an internet provider's network instead of a cloud host's. Scope it to the step so checkout and dependency installs skip it:
- name: Run scraper
env:
HTTPS_PROXY: ${{ secrets.SCRAPER_PROXY }}
HTTP_PROXY: ${{ secrets.SCRAPER_PROXY }}
run: python scrape.py
requests honours these variables because trust_env is on by default. Scrapy's HttpProxyMiddleware reads them through Python's getproxies() at startup. Store the full http://user:pass@host:port string as the secret.
One caveat. A site that blocks every GitHub range has decided it doesn't want automated traffic from there, so check that what you collect, and how often, fits its terms.
Docker: a different box, or a different TLS library
When a spider works in PyCharm and gets 403 in a container, first check whether the container leaves from the same IP at all.
A container on your own laptop normally exits through the laptop's NAT, so the target sees your usual public IP. On 2024-12-26 a poster on r/scrapy wrote: "From IDE (PyCharm) code works perfectly. From Docker Container responses are 403. I checked headers ... requests are identical (except IP)". They were using custom proxies. With a proxy actually in use, httpbin reports the proxy's IP from both places, so a different IP points one of two ways. Either the container runs on another host, like a cloud server, and you're back in the previous section. Or the proxy setting never reached the container, because shell environment variables don't pass into docker run without -e HTTPS_PROXY=....
If the IP really matches, look at the image. A python:3.x-slim image ships its own OpenSSL build, which can differ from your desktop Python's. A different build can mean a different TLS ClientHello, and some anti-bot systems score that before a single header arrives.
And httpbin can't settle it. Its /anything endpoint echoes parsed headers as a JSON object. It says nothing about header order or the TLS handshake.
Same machine, same proxy: requests gets 200, Scrapy gets 403
When requests and Scrapy run on one machine through one proxy, the target sees the same IP from both, so the difference has to be somewhere else.
Three r/scrapy threads describe exactly this. On 2024-11-02 one poster had requests.get return 200 while Scrapy, with "the same proxy (a rotating residential proxy) and different user agent", got 403. On 2023-03-07 another wrote "I'm blocked with Scrapy but not with python's Requests module, even if I send the same request", and Scrapy landed on a captcha page. On 2024-02-03 a third found that "Even with the same headers and no proxy, the script using scrapy always without fail gets detected."
We tested it. On a Linux box with Scrapy 2.19 and current requests, we sent the headers dict from the 2023-03-07 thread to a local listener and captured what each library put on the wire:
requests Scrapy
GET / HTTP/1.1 GET / HTTP/1.1
Host: 127.0.0.1:8088 User-Agent: Mozilla/5.0 (X11; ...
User-Agent: Mozilla/5.0 (X11; ... Accept: */*
Accept-Encoding: gzip Accept-Encoding: gzip
Accept: */* Accept-Language: en
Connection: keep-alive Host: 127.0.0.1:8088
Scrapy put Host last, dropped Connection: keep-alive, and added Accept-Language: en from its DEFAULT_REQUEST_HEADERS. The TLS ClientHello differed too. requests offered 31 cipher suites and an ALPN extension. Scrapy offered 30, a different extension list, and no ALPN. That follows from the stacks: requests runs on urllib3 and Python's ssl module, Scrapy on Twisted and pyOpenSSL. Neither looks like Chrome, and both speak HTTP/1.1 where a browser would negotiate HTTP/2 with most large sites.
A site that fingerprints any of these layers can tell the two clients apart on the first request. A proxy changes none of it. Through an HTTP CONNECT tunnel your TLS handshake reaches the site untouched.
So fix the layer that differs. Override DEFAULT_REQUEST_HEADERS with what a real browser sends, including a sensible Accept-Language. For the handshake, swap the download handler: scrapy-impersonate is a Scrapy download handler built on curl_cffi that presents a browser's TLS and HTTP/2 fingerprint, and scrapy-playwright drives a real browser if the site also needs JavaScript. Then repeat the capture. A settings file proves nothing until the wire agrees.
Why a real Chrome User-Agent can do worse than "Scrapy/2.11.1"
A browser User-Agent on a client that doesn't behave like that browser is a contradiction, and some sites punish a contradiction harder than an honest bot name.
On 2024-04-16 a poster ran a private squid proxy that worked in Firefox and returned "403 access denied" in Scrapy, until "the requests started to work only after I disabled the USER_AGENT parameter". Scrapy fell back to its default, Scrapy/2.11.1 (+https://scrapy.org), and got through. Their custom UA claimed Chrome 123 on Windows while the handshake and headers said Python on Twisted.
Either make the claim true, with impersonation or a real browser, or stop making it. Rotating a list of browser UAs over the same non-browser handshake just repeats the contradiction with more variety.
The 403 or 429 that shows up after a few minutes
A block that starts partway through a run is about rate or cookies, and rotating IPs won't help when the site keys its limit on something other than the IP.
On 2022-11-06 a poster scraping rental listings through a residential proxy with rotating User-Agents got 403 "after a few minutes of scraping" and never recovered. They asked whether cookies were to blame. On 2024-07-31 another got "response 429 most of the time" despite "proxy rotation, limited concurrent requests and delayed download_delay".
Scrapy's framework defaults are CONCURRENT_REQUESTS = 16, CONCURRENT_REQUESTS_PER_DOMAIN = 8 and DOWNLOAD_DELAY = 0, with AutoThrottle off. Projects generated by recent versions override that in settings.py with CONCURRENT_REQUESTS_PER_DOMAIN = 1 and DOWNLOAD_DELAY = 1. Older projects don't. The retry middleware retries 429 and 5xx twice by default, ignores 403, and doesn't read Retry-After. Every 429 can become three requests.
Cookies are the quieter half. COOKIES_ENABLED is on, and by default every request shares one jar. Rotate the IP per request and the site sees one session cookie arriving from many addresses. Give each sticky session its own jar instead:
import uuid
import scrapy
def sticky_proxy(sid):
return f"http://USER-session-{sid}:PASS@HOST:PORT" # session syntax per your provider
class ListingSpider(scrapy.Spider):
name = "listings"
custom_settings = {
"CONCURRENT_REQUESTS_PER_DOMAIN": 2,
"DOWNLOAD_DELAY": 1.5,
"AUTOTHROTTLE_ENABLED": True,
}
async def start(self):
for url in self.start_urls:
sid = uuid.uuid4().hex[:8]
yield scrapy.Request(url, meta={"proxy": sticky_proxy(sid), "cookiejar": sid})
Follow-up requests from a page should carry the same cookiejar and proxy values. request.meta['proxy'] takes credentials in the URL, and Scrapy builds the Proxy-Authorization header itself.
One more trap. New projects set ROBOTSTXT_OBEY = True, and a request robots.txt disallows never leaves your machine. The log says "Forbidden by robots.txt". Read the log before buying anything.
Country locks are an IP problem with one answer
A site that answers only one country's visitors can only be reached from an IP in that country.
On 2023-01-13 a poster found a property site refused every proxy and VPN "except if it is from Chile". No header change gets around that. You need an exit that geolocates to Chile and isn't in a hosting range, since a Chilean data center IP can fail the same way a runner does. Rotating residential with a country filter is the usual tool. Confirm the country is in your provider's location list first.
Do you need a proxy at all?
If your origin IP isn't what the site refuses, a proxy adds cost and a failure point and fixes nothing.
On 2026-10-30 a poster on r/LangChain wanted markdown extraction with the constraint "I don't want to own a browser. I don't want a proxy provider", for "around 500 pages a week". At that volume plenty of sites serve a home connection without complaint. Test from home first.
Which proxy for which of these
Pick the proxy type by the job, and know what it won't do.
- CI runner or cloud box refused by range: a static ISP proxy, one stable IP across runs.
- Volume spread across many pages: rotating residential, with concurrency still capped.
- Country-locked site: residential with a country filter.
- Logged-in or multi-step session: a sticky residential session or a static ISP IP, one cookie jar per IP.
- requests works and Scrapy fails on the same box: no proxy fixes this. Headers, TLS and rate first.
BirdProxies sells rotating and sticky residential, static ISP and mobile, with user and password auth over HTTP or SOCKS5. Core Scrapy only speaks HTTP proxies, so use the HTTP endpoint there. The Scrapy proxy page has the middleware wiring.
Run the tests in this order
Change one variable per step and stop at the first step where the result flips.
- Same box, same URL, three clients. If curl and requests get 200 and Scrapy gets 403, it's the client.
URL="https://example.com/page"
curl -s -o /dev/null -w "%{http_code}\n" "$URL"
python3 -c "import requests,sys; print(requests.get(sys.argv[1]).status_code)" "$URL"
scrapy shell --nolog -c "response.status" "$URL"
- See what each client really sends. Point both at a local listener (
nc -l 8088, ornc -l -p 8088on older netcat) and compare order and extra headers. - Confirm the exit IP everywhere you run. Run
curl -s https://api.ipify.orgon the host, inside the container (docker run --rm IMAGE curl -s https://api.ipify.org) and in a CI step, with and without the proxy variables. With the proxy on, all three should print the proxy's IP. - Same code, three origins: home, a container on a cloud host, a CI runner. If only the cloud and the runner fail, the origin is refused and a proxy is the right fix.
- Failing origin plus proxy. If it passes, you're done. If not, go back to steps 1 and 2.
FAQ
Why does my scraper work locally but get 403 on GitHub Actions?
GitHub-hosted runners use cloud data center IPs, published under the actions key at api.github.com/meta, and sites can block the whole list. Route the scraping step through a static ISP or residential proxy with HTTPS_PROXY set from a secret.
Why does requests return 200 but Scrapy returns 403 with the same headers?
The wire traffic differs. Scrapy reorders headers, adds Accept-Language: en, and uses Twisted and pyOpenSSL for TLS where requests uses urllib3 and Python's ssl. The IP is identical, so a proxy won't change the result.
Why does my Scrapy spider work in PyCharm but not in Docker?
Check the exit IP from inside the container. Either it runs on a host with a hosting IP, or your proxy variables weren't passed in with -e. If the IP matches, compare the image's OpenSSL build with your desktop Python's.
Why did removing my Chrome User-Agent fix the 403?
The fake Chrome UA contradicted Scrapy's real TLS and header fingerprint. Some sites treat that worse than an honest Scrapy/x.y (+https://scrapy.org) string.
Does rotating proxies stop 429 errors in Scrapy?
Only if the site limits per IP. If it limits per session cookie or account, rotation won't help, and one shared cookie jar can make it worse. Lower CONCURRENT_REQUESTS_PER_DOMAIN, add DOWNLOAD_DELAY or AutoThrottle, and give each proxy session its own cookiejar.
How do I scrape a site that only allows IPs from one country?
Use an exit IP in that country outside the hosting ranges, usually residential with a country filter. Headers and fingerprints can't get around a country lock.
Do I need a proxy to scrape a few hundred pages a week?
Often not. Test from a home or office connection first and add a proxy only if the origin IP is what gets refused.
Can Scrapy use SOCKS5 proxies?
Not in core. HttpProxyMiddleware handles http:// and https:// proxy URLs, so use an HTTP endpoint or add a third-party handler.