Rate Limiter
Note: introduced in 0.10.0
.
The :rate_limiter
plugin is a convenience plugin for automatically handling server-side rate limiting, an HTTP-based protocol for a server to communicate to a client that a server is currently overloaded, and asking it to try again later.
session = HTTPX.plugin(:rate_limiter)
response = session.get("https://hackernews.org/")
# HEADLINE: "GET / HTTP/1.1"
# ...
# -> HEADLINE: 429 HTTP/1.1
# -> HEADER: retry-after: 5
# ...
# failed to get response, 2 tries to go...
# retrying after 2 secs...
# retrying!!
# HEADLINE: "GET / HTTP/1.1"
# ...
# -> HEADLINE: 200 HTTP/1.1
# ...
Important! The :rate_limiter
plugin builds on top of the :retries
plugin, and reuses the available options, so, besides the :max_retries
, you are suggested not to change any of the other options from the :retries
plugin.
The :rate_limiter
plugin implements the “retry after” policy by leveraging scheduler timers. At the time of writing this page, the only supported scheduler is internal, so HTTPX
will await until it gets a reply from the server (or the maximum number of retries is exceeded, or an unrecoverable error happens, etc etc).
The :rate_limiter
plugin is also not ideal in a scenario, where a request is performed inside a background job, because you’d best want to deal with a rate-limiting response by reenqueing the job in the queue and schedule it for when the retry period is over. In such a scenario, you’re best left with using the :retries
plugin, and the retry_on
callback to signal whether you want to retry the request immediately (or let it fail and reschedule the job).
Next: AWS Sigv4