Home Wiki

Connections

HTTPX abstract all the network/application connection level gory details from the user, so that, from the user’s perspective, it’s all HTTP. It’s a design decision, therefore a “feature not a bug”.

This document explains what’s the default behaviour, how to enable variants of it (and what are the implications), and alternative usages.

Default

HTTPX is optimized for concurrent request handling, which means, it will try to reuse resources as much as possible (sockets, TLS sessions) for the maximum number of requests, and will interleave them as much as the underlying protocol allows.

Network

You can just perform requests using IPs in the URLs:

HTTPX.get("http://[::1]/get")
HTTPX.get("http://127.0.0.1/get")

This will use the IPs as the authority. Sometimes, you want to set the authority, but bypass name resolving (because you know the IP of the server or you have an SSH tunnel on a local IP, or DNS is down). You can achieve that by using the :addresses option:

google_ip = "163.164.165.166"
HTTPX.get("https://google.com", addresses: [google_ip])

:addresses must be an array of IPs, and in case connecting to the first fails, it’ll try on the next one and so on.

Transport

httpx uses TCP sockets. The sockets will be reused:

for concurrent calls coalescing on the same origin (HTTPX.get("https://example.com/1", "https://example.com/2");

for the duration of the HTTPX::Session.wrap block:

HTTPX.wrap do |http|
r1 = http.get("https://example.com/1")
r2 = http.get("https://example.com/2")
end

if you’re using the :persistent plugin (sockets will be kept open):

http = HTTPX.plugin(:persistent)
r1 = http.get("https://example.com/1")
r2 = http.get("https://example.com/2")

You can also open the tcp socket and pass it yourself, via the :io option:

google_sock = TCPSocket.new("163.164.165.166")
HTTPX.get("https://google.com", io: google_sock)

# if socket was created externally, httpx won't close it either!
google_sock.closed? #=> false

# or if you're connecting to multiple domains:
google_sock = TCPSocket.new("163.164.165.166")
amazon_sock = TCPSocket.new("184.24.16.9")
HTTPX.get(
  "https://google.com", "https://www.amazon.com",
  io: {
    "google.com" => google_sock,
    "www.amazon.com" => amazon_sock
  }
)
google_sock.closed? #=> false
amazon_sock.closed? #=> false

UNIX Sockets

You can use unix sockets by using the :transport option and setting it to “unix”.

Providing the UNIX socket location can be done using the :addresses optionn:

server_socket = "/path/to/socket"

HTTPX.get("https://myapp", transport: "unix", addresses: [server_socket])

However, just like in the section above, you can just initialize the socket yourself, and pass it to the httpx session:

io = UNIXSocket.new("/path/to/socket")

HTTPX.get("https://myapp", transport: "unix", io: { "myapp" => io })

Session

If the request is done using an “https://” URI, TLS will be used, and ALPN negotiation will privilege “h2” if the server supports it. If the server doesn’t support “h2” or ALPN negotiation at all, it will fallback to HTTP/1.1 over TLS.

If the request is done using an “http://” URI, it will use “HTTP/1.1” by default.

If the server supports “cleartext HTTP/2”, you can set the :fallback_protocol option to "h2":

http = HTTPX.get(fallback_protocol: "h2")
http.get("http://nghttp2.org") #=> will use HTTP/2

If you use the h2c or upgrade/h2 plugins, and the server supports upgrading a connection to HTTP/2, it’ll be done using HTTP/2.

Presentation

If the selected protocol is HTTP/2, requests will be interleaved in the same connection (and socket). Requests will use the same connection if:

They are sent to the same origin;
They are sent to different origins, but both resolve to the same IP and the server TLS certificate matches against both origins (wilcard or hostnames lists, if TLS is being used);

(This technique is called “unsharding”, which I picked up from cURL’s blog).

If the selected protocol is HTTP/1, HTTPX will try to use HTTP pipelining to send multiple requests to the server (the number of requests is controlled by the :max_concurrent_requests option). If the server doesn’t support pipelining, it will fallback to sending one request at a time in the same connection; if the peer closes the connection however, it falls back again, this time to one request per connection.

Caveat: Some servers/proxies may not fail gracefully when receiving the pipelined requests and may abruptly abort the connection. There is no recovery strategy implemented for that. However, you can disable pipelining by setting :max_concurrent_requests to 1.

External Event Loop integration

TODO: fill this up when this is possible

Next: Thread Safety

honeyryder