A 403 and a 429 look similar in a log and need different responses. Changing proxy type for a rate-limit problem wastes money, and slowing down for a reputation problem wastes time.
429: you were too fast
Too Many Requests means a limit was reached for an identifier, usually an address or an account. Read the Retry-After header if present and honour it. Spread requests across more addresses or lower your rate. A different proxy type is rarely the fix.
403: you were identified
Forbidden is a decision about the request. Work through the causes in order of cheapness:
- Headers: a missing or implausible User-Agent, Accept-Language or Referer. Fix this first.
- Geography: the page is not served to that country. Retry from the intended market.
- Network reputation: datacenter ranges are blocked by this target. Test a residential or ISP route on a small sample.
- Fingerprint: the target inspects TLS or browser behaviour. A real browser or an unblocking API is the proportionate answer.
Soft blocks
The most expensive failure returns 200 and a challenge page. Put a content check in your pipeline, because a status check alone will count it as success.
def is_real(response, must_contain):
return response.status_code == 200 and must_contain in response.text