Through a proxy you can see errors from three places: the gateway, the network between, and the target. Tell them apart before you retry, because retrying the wrong one wastes money and can make a block worse.
From the gateway
- 407: credentials not accepted. Retrying will not help; fix the credentials or the encoding.
- 502 or 504 from the gateway: the route to the target failed. Retry a small number of times with a delay.
- Connection reset or timeout before any response: often the network or an unreachable exit. Retry, and count it.
From the target
- 429: you are going too fast. Slow down and honour a Retry-After header if present.
- 403 on the first request: the target refuses what you appear to be. More retries from the same client will not change that; look at headers, fingerprint and address type.
- 200 with a challenge or empty body: a soft block. It is a failure; detect it with a content check.
- 404 and 410: the page is gone. Do not retry.
A retry pattern
Wait longer after each failure, add randomness so workers do not retry in step, cap the attempts, and record the attempt count with the result so retry amplification is visible in your cost.
import random
import time
import requests
RETRYABLE = {429, 500, 502, 503, 504}
def fetch(url, proxies, attempts=4, base=1.0, cap=30.0):
for attempt in range(1, attempts + 1):
try:
r = requests.get(url, proxies=proxies, timeout=30)
if r.status_code not in RETRYABLE:
return r, attempt
except (requests.ConnectionError, requests.Timeout):
pass
time.sleep(random.uniform(0, min(cap, base * 2 ** attempt)))
return None, attemptsPair this with the checks in the method page, so that success means a page that passed a content check, not only a status code.