The :rate_limiter plugin is a convenience plugin for automatically handling server-side rate limiting, an HTTP-based protocol for a server to communicate to a client that a server is currently overloaded, and asking it to try again later.
session = HTTPX.plugin(:rate_limiter)
response = session.get("https://hackernews.org/")
# HEADLINE: "GET / HTTP/1.1"
# ...
# -> HEADLINE: 429 HTTP/1.1
# -> HEADER: retry-after: 5
# ...
# failed to get response, 2 tries to go...
# retrying after 5 secs...
# retrying!!
# HEADLINE: "GET / HTTP/1.1"
# ...
# -> HEADLINE: 200 HTTP/1.1
# ...
Important! The :rate_limiter plugin builds on top of the :retries plugin, and sets some of its options, so, besides the :max_retries, you are suggested not to tweak any of the options of the :retries plugin.
The :rate_limiter plugin implements the “retry after” policy by leveraging scheduler timers. At the time of writing this page, the only supported scheduler is internal, so HTTPX will await until it gets a reply from the server (or the maximum number of retries is exceeded, or an unrecoverable error happens, etc etc).
The :rate_limiter plugin is also not ideal in a scenario, where a request is performed inside a background job, as you’d rather deal with a rate-limiting response by reenqueing the job and schedule it to run after the retry period is over. In such a scenario, you’re best left with using the :retries plugin, and the retry_on callback to signal whether you want to retry the request immediately (or let it fail and reschedule the job).
Next: AWS Sigv4