Integration · Python

Scrapy

Per-request proxy through request meta, with retry settings that respect cost.

Install

terminal
pip install scrapy

Example

spider.py
import scrapy


class IpSpider(scrapy.Spider):
    name = "ip"
    custom_settings = {"RETRY_TIMES": 2, "DOWNLOAD_TIMEOUT": 30}

    async def start(self):
        yield scrapy.Request(
            "https://httpbin.org/ip",
            meta={"proxy": "http://USERNAME:PASSWORD@GATEWAY_HOST:PORT"},
        )

    def parse(self, response):
        yield response.json()

Replace USERNAME, PASSWORD, GATEWAY_HOST and PORT with the values in your dashboard. Never commit them.

Notes

  • On request-priced APIs, a retry is a billable request unless the product states otherwise. Cap RETRY_TIMES deliberately.

Common errors

407 Proxy Authentication RequiredCause. Credentials missing, mistyped, or not URL-encoded.
Fix. Percent-encode special characters in the username and password, or pass credentials separately from the URL where the client allows it.
Timeouts on first requestCause. Wrong gateway host or port, or an outbound firewall rule on your side.
Fix. Copy host and port from the dashboard entry for your product and test with curl first.

Next: measure it on your real target.