What is logging?

(skip this section if you’re above being lectured about what’s logging again).

Logging is one of those fundamental features of any type of program you use. At a high level, it keeps a record of what a program is and has been doing, be it error messages, or general information, that can be used for audit trail, debugging issues, or generally just figuring out what the hell is happening with a process.

Because this is a feature as old as time, a lot of energy has been spent trying to standardize it. The generally most accepted standard (in UNIX corners at least) has been the Syslog standard, which separates the program generating the message (ex: logging library interface writing to stdout, or a file, or a socket, or all at the same time) from the program managing its storage (ex: logrotate, logstash…) and the program reporting/analysing it (ex: kibana, or plain tail and grep).

(Even) more standards have existed for the message format, which may depend of the type of program you’re using (an example being the common log format for server logs). Some general rules are agreed upon though, such as: there is a log entry per line, a log entry should identify its severity level (examples: “debug”, “info”, “error”, “warn”, “alert”, …), and contain a timestamp, besides the actual log message.

Logging in ruby

The ruby gateway to logging is the logger standard library. In a nutshell, users log by using Logger objects, which know where to write them (internally called “log device”), and how to write them (“formatter”):

require "logger"

# logger which writes messages to standard out
logger = Logger.new(STDOUT)

# writes debug message with the default message format:
#=> $<capital letter for severity level>, [$<timestamp ruby to_s> #$<process id>] $<severity full again, why, we know it already> -- : $<log message>
logger.debug "foo"
#=> D, [2025-11-05T12:10:08.282220 #72227] DEBUG -- : foo

# only writes messages with INFO level or higher
logger.info!
logger.info "foo"
#=> I, [2025-11-05T12:10:54.862196 #72227]  INFO -- : foo
logger.debug "foo"
#=>
# use block notation to avoid allocation the message string
logger.debug { "foo" }
#=>

class MyCustomFormatter
  # formatters must at least implement this method
  def call(severity, time, progname, msg)
    "my format -> #{msg}"
  end
end

# swap formatter
logger.formatter = MyCustomFormatter.new
logger.info { "foo" }
#=> "my format -> foo"

# enable daily log rotation
daily_logger = Logger.new("my.log", :daily)
daily_logger.info "foo" #=> writes log entry into my.log
# sleep for one day...
daily_logger.info "foo" #=> will rename my.log to my.log.1 and write new message to brand new my.log file

logger is a mixed bag. The default formatter is certainly unusual (although it feels like every programming language has its own default logging format, so perhaps an historical artifact?), and considering ruby’s historical UNIX friendliness, I’m always surprised that default messages do not include the system user. Swapping the formatter is easy though.

The Log device interface feels a bit more limiting. While writing to stdout/stderr or a file is easy, writing to a socket (like a syslog server) is much harder than it needs to be (you have to write your own Logger::LogDevice subclass). It also works a bit counter to the Syslog standard described above, as, being a utility to “streamline the generation of messages”, it shouldn’t really care about storing details (such as log rotation), and doesn’t support the ability to write to multiple locations at once.

Still, it’s rather straightforward to use, as long as none of the limitations mentioned above matter to you.

Logging in rack

One of the main uses of ruby in the industry has been web applications. Most of them are wrapped inside rack containers and deployed using application servers like webrick or puma. rack ships with a common logger middleware, which emits a log entry per request using the apache common logging format:

# example of a web request log:
# client ip, user or "-", datetime, method, path, http version, status code, response body size in bytes, processing-to-send time
#
127.0.0.1 - [01/May/2025:07:20:10 +0000] "GET /index.html HTTP/1.1" 200 9481 10

you can use it in your rack application by adding it to your config.ru file:

# config.ru
use Rack::CommonLogger

run MyApp

The above isn’t common though, as the framework you may be using to build your application may do it for you, or ship with its own logger middleware implementation. For instance, both roda and sinatra ship or recommend its own extension plugin, for different reasons, such as performance or configurability.

Logging in rails

In rails applications, most interact with logging via the Rails.logger singleton object. While mostly API compatible with the standard logger library counterpart, it bundles its own (rails) conventions on top of it.

Like a true schroedinger’s cat, Rails.logger is and is not a logger at the same time: the documentation says it’s an instance of ActiveSupport::Logger (a subclass of stdlib’s Logger), but if you inspect it in the console, it’s actually something else:

Rails.logger #=> instance of ActiveSupport::BroadcastLogger

Rails documents that one can change the logger in application config (a common use case is to write test logs to /dev/null by setting config.logger = Logger.new("/dev/null")) in config/environments/test.rb), but in the end, the singleton instance is an instance of ActiveSupport::BroadcastLogger, a proxy object which can register multiple log devices and forward message calls to them. From the official docs:

stdout_logger = Logger.new(STDOUT)
file_logger   = Logger.new("development.log")
broadcast = BroadcastLogger.new(stdout_logger, file_logger)

broadcast.info("Hello world!") # Writes the log to STDOUT and the development.log file.

It seems that the broadcast logger was rails internal solution to the lack of support for multiple log devices per Logger instance in the logger standard library.

The rails logger also ships with its own formatter, which does the simplest possible thing:

Rails.logger "foo" #=> "foo"

Alternatively to ActiveSupport::Logger, rails has ActiveSupport::TaggedLogging. This adds the capability to add “context tags” to a scope, where all log messages within it will be formatted with it:

logger = ActiveSupport::TaggedLogging.new(Logger.new(STDOUT))
logger.tagged("FOO") { logger.info "Stuff" } #=> Logs "[FOO] Stuff"
logger.tagged("BAR") do
  logger.info "Stuff" #=> Logs "[BAR] Stuff"
end
logger.tagged("FOO", "BAR") { logger.info "Stuff" } #=> Logs "[FOO] [BAR] Stuff"

Structured logging

All those standards and message formats are nice and all, but in 2025, everyone and their mothers want structured logging. The most common format, at least in the corners I work in, is JSON. It probably has to do with it, in spite of its deficiencies, being a quite simple serialization format and widely adopted, which guarantees virtually universal support. As a counterpart to the log management stack for syslog-type systems, new stacks started popping up, such as the fluentd/logstash/elasticsearch/kibana OS stack, alongside SaaS solutions like Splunk or Datadog.

There was renewed interest in re-standardizing log message “envelopes”, one of the emerging standards being the logstash event format.

# logstash event format
'{"message":"foo","tags":["tag1"],"source":"127.0.0.1","@version":"1","@timestamp"}'

That being said though, the ecosystem hasn’t really consolidated on formats yet, so it’s common to see different standards in use across different systems. What’s common across all of them though, is the need to logically structure the log message separately from its associated metadata, or context.

Nowadays, structured logging fills a complementary role in the larger picture of observability.

The new world of observability

Monitoring the health of a system isn’t a new requirement. As mentioned above, logging is quite an old OS telemetry feature. Back in the “old days” of server/system administration, it was common to set up software like Nagios to collect OS-level telemetry data and visualize i.e. memory consumption, CPU usage, instance connectivity, among other data points. in user-friendly web GUIs.

Since the explosion of Cloud Computing and the Google SRE playbook, and trends such as microservices or lambda functions, observability took a center stage and grew until it incorporated several concepts which used to be thought of as apart from each other. Nowadays the buzzwords are RUM, Open Telemetry, APM, RED metrics, error tracking, among others, and these are all driven by system and application-emitted metrics, logs, and its new more recent friend, traces, which are a way to visualize execution flows which incorporate related execution flows (usually callend “spans”) within it, as horizontal bars correlating timelines.

That center stage translated into big business, and companies like Datadog, Sentry or Honeycomb became almost as critical to a client’s success as the features that client provides. Observing, measuring, monitoring the health / performance / volume of our applications has never been as easy (and as expensive).

ruby logging in 2025

Sadly, the ruby logger library APIs didn’t keep up with the times, and are quite limited for this new paradigm. While nothing stops anyone from swapping the default formatter with a JSON capable counterpart, the Logger::Formatter API, which relies on implementation of call with a fixed set of positional arguments, makes it impossible to support metadata other than what the function already expects:

class MyJSONFormatter
  # formatters must at least implement this method
  def call(severity, time, progname, msg)
    # can't receive i.e. user data, just the 4 levels above:
    { severity: time: progname:, message: msg }.to_json
  end
end

This diminishes its reusability, and as a result, every other logger library in the ecosystem which logs JSON (and other formats) does not use the logger library as its foundation layer, and ends up reinventing the Formatter API to its needs.

But don’t take my word for it. Looking at the most used logging libraries in ruby toolbox which support structured JSON format, log4r has its own base formatter class which defined #format(String event) as the overridable method; lograge also has its own base formatter class which defines #call(Hash data) as its own, while semantic logger also has one, this time defining #call(SemanticLogger::Log log, SemanticLogger::Formatters::Base logger), and so does logstash-logger have its own base formatter, which funnily enough supports… the same call API as ruby logger formatters!

This is official xkcd territory.

(Practically all of the above also solve the problem of writing to multiple log devices, in most cases naming this feature “log appenders”. But this is not the feature I’m writing the post about).

rails logging in 2025

Given that ActiveSupport::Logger is a subclass of Logger, it also inherits (OO-pun intended) its problems, therefore by the transitive property, rails logger does not support structured logging (and JSON in particular). So if your rails application emits JSON logs, you’re either using one of the alternatives above, or an in-house library made out of spare parts of everything mentioned so far, or worse (gulp) a parser (like grok) regex-matching your string entry and spitting a JSON from it.

The most stable, and to my knowledge, more widely adopted logging libraries, are lograge and (rails) semantic logger.

In both cases, the Rails.logger singleton instance broadcasts to a custom logger implementation provided by the library, and the main log-related subscriptions for default notifications in-and-around business operations (like processing web requests) are swapped by custom (to each library) subscriptions, which make use of the logger API and allow adding extra context to each of these log messages.

lograge

lograge documents a custom_options callback, which receives a hash and returns another hash. The received hash is the event hash which gets passed to request-level event notifications, and can be augmented in controllers by re-defining the controller append_info_to_payload callback. The returned hash gets passed “as is” to the eventual JSON log entry (which also contains a readable “message”), giving almost full control of the JSON message format.

It has several drawbacks though, one of them being, it only subscribes to action-controller-level events, so active jobs will keep being logged by “standard” rails logger. Also, it’s not possible to share or add different context to other logger calls when using Rails.logger.info and friends.

If you’re using the rails framework for anything other than web requests, I wouldn’t recommend it.

(It also subscribes to action cable events, but I suspect very few applications running in production use it).

semantic logger

In turn, (rails) semantic logger subscribes not only to action controller events, but active job events as well (and active record events, and active view, and action mailer… if that can be subscribed, it will be subscribed!), which makes it more compelling to use. It also ships with interesting features which allow to not only add context to direct logging calls, but setting context to a given scope as well:

logger.info("hi", payload: {"foo" => "bar"})
#=> '{"message":"hi","payload":{"foo":"bar"}....'
logger.info("hi")
#=> '{"message":"hi",....'
SemanticLogger.tagged("foo" => "bar") do
  logger.info("hi")
  #=> '{"message":"hi","payload":{"foo":"bar"}....'
end
logger.info("hi")
#=> '{"message":"hi",....'

Still, while having this feature, semantic logger still disappoints by recommending a similar type of integration as lograge does for requests (log_tags callback + append_info_to_payload), which limit the scope of request-level payload to the single logger call happening within log subscribers. It feels like a lost opportunity, considering that it’d be great to share that context with all user-defined logger calls happening within the scope of the request processing (including calls happening from within the controller action), and other rails-level business transactions (such as active job #perform calls) do not have an append_info_to_payload counterpart (perhaps someone should suggest that feature to rails?).

The resulting JSON format (all non-standard context under "payload", some things under "named_tags" when using some obscure API) isn’t the friendliest either, and in most cases, ends up being rewritten by a pre-processing step before log ingestion happens.

Still, despite all its flaws and somewhat clunky API, it showcases the potential of, for lack of a better name, a logger context API.

Context API

Imagine if, during the scope of request processing, several context scopes could be interleaved, each one with its context, tearing down each sub-context when exiting blocks; this context could then be used in the log analysis engine to aggregate groups of messages tags from each particular context, allowing more fine-grained filtering.

If you’re using any type of tracing integration, you don’t need to imagine, because this is how the tracing API works! For example, if you are using the datadog SDK:

# from the datadog sdk docs:
def index
  # Get the active span and set customer_id -> 254889
  Datadog::Tracing.active_span&.set_tag('customer.id', params.permit([:customer_id]))

  # create child span, add tags to it
  Datadog::Tracing.trace('web.request') do |span|
    span.set_tag('http.url', request.path)
    span.set_tag('<TAG_KEY>', '<TAG_VALUE>')
    # execute something here ...
  end
end

Something like this, using plain loggers, should be possible too:

def index
  logger.add_context(customer_id: params.permit([:customer_id]))
  # logger.info calls will include the "customer_id" field
  logger.with_context(http_url: request.path, tag_key: "tag_value") do
    # logger.info calls will include the "customer_id", "http_url" and "tag_key" fields
  end
  # logger.info calls will only include the "customer_id" field
end

And that’s why, to somewhat stitch the inconsistencies described above together, I’m proposing such an API to the logger standard library.

Feature Request

For a more detailed description, you can read the issue and PR description/comments. In a nutshell, two ways are introduced of adding context: per block (via Logger#with_context) and per call (keyword argument in Logger#info, Logger.error and friends):

# per block
logger.with_context(a: 1) do
  logger.info("foo") #=> I, [a=1] [2025-08-13T15:00:03.830782 #5374]  INFO -- : foo
end
logger.with_context(a: 1) do
  logger.with_context(b: 2) do
    logger.info("foo") #=> I, [a=1] [b=2] [2025-08-13T15:00:03.830782 #5374]  INFO -- : foo
  end
end

# per call
logger.info("foo", context: {user_id: 1}) #=> I, [user_id=1] [2025-08-13T15:00:03.830782 #5374]  INFO -- : foo
logger.info(context: {user_id: 1}) { "foo" } #=> I, [user_id=1] [2025-08-13T15:00:03.830782 #5374]  INFO -- : foo

The proposal tries to retrofit context into the current default message format, and does not aim at proposing a JSON message formatter. At least until this is done.

That’s it!

There’s a lot of devil in the details though, and if you’ll read through the PR discussions, there were many meaningful points raised:

• how/where to manage contexts?
  • ruby should manage contexts per thread AND per fiber, which raises some questions around context sharing across parent-child fibers, what the runtime supports OOTB, as well as certain core APIs which spawn fibers under the hood.
• should context be managed in formatters rather than logger instances?
  • I’m leaning on the latter, but it’ll depend on future developments in logger. For example, will it ever support multiple log devices per instance? And if so, will each log device have its own formatter? In such a case, should context be shared across formatters?
• what’s the bare minimym feature set
  • do we need per-call context? can it get away with with_context only?

Logging context in rack

Unlocking per-request logging context becomes as simple as including this middleware in your rack application:

class LoggingContext
  def initialize(app, logger = nil)
    @app = app
    @logger = logger
  end

  def call(env)
    @logger.with_context { @app.call(env) }
  end
end

# then in config.ru
use LoggingContext

run MyApp

You could then make use of this API in your application, knowing that context will be correctly tore down at the end of the request lifecycle:

# This is just an example of how to add request info as logging context, it is NOT supposed to be a recommendation about how to log
# authentication info.

# roda (with rodauth) endpoint
class MyApp < Roda
  plugin :common_logger
  plugin :rodauth

  # ...

  route do |r|
    logger = @logger || request.get_header(RACK_ERRORS)
    r.rodauth

    get 'index' do
      @user = DB[:accounts].where(:id=>rodauth.session_value).get(:email)

      logger.with_context(user: { id: @user.id }) do
        view 'index'
      end
    end
  end
end

# rails controller action
class MyController
  before_action :require_user
  around_context :add_logging_context

  # ...

  def index
    Rails.logger.info "about to index" # will log user.id in context
  end

  private

  def add_logging_context
    Rails.logger.with_context(user: { id: @user.id }) { yield }
  end
end

Logging context in background jobs

Similar approaches can be applied for your preferred background job framework. For brevity, I’ll just show below how you could use the same callback/middleware strategy for Sidekiq and Active Job:

# 1. Sidekiq
class LoggingContext
  include Sidekiq::ServerMiddleware
  def initialize(logger)
    @logger = logger
  end

  def call(job, payload, queue)
    @logger.with_context(job: { queue: queue, id: job["jid"] }) { yield }
  end
end

# when initializing...
Sidekiq.configure_server do |config|
  config.server_middleware do |chain|
    # if you're using rails, replace bellow with Rails.logger
    chain.add MyMiddleware::Server::ErrorLogger, logger: LOGGER
  end
end

# then in job...
class MyJob
  include Sidekiq::Job

  def perform(arg1, arg2)
    LOGGER.info "performing" # will include job.queue and job.id in context
  end
end

# 2. Active Job
class ApplicationJob < ActiveJob::Base
  around_perform do |job, block|
    Rails.logger.with_context(job: { queue: job.queue_name, id: job.id }) do
      block.call
    end
  end
end

# then in job...
class MyJob < ApplicationJob
  def perform(arg1, arg2)
    Rails.logger.info "performing" # will include job.queue and job.id in context
  end
end

Logging context in other languages

Another angle of this discussion is looking at how other ecosystems solve this problem. I’ll just mention a few examples, as my purpose is not to be exhaustive, so apologies in advance if I skipped your second-preferred language.

Java

While core Java Logger APIs do not seem to support this, most applications use the log4j library, which supports a feature called Thread Context, which is very similar to the one described above:

ThreadContext.put("ipAddress", request.getRemoteAddr());
ThreadContext.put("hostName", request.getServerName());
ThreadContext.put("loginId", session.getAttribute("loginId"));

void performWork() {
  // explicitly add context for this function, which copies all context until then
  ThreadContext.push("performWork()");
  LOGGER.debug("Performing work"); // will include ipAddress, etc...
  // do work
  ThreadContext.pop();
}

// or with auto-closing enabled
try (CloseableThreadContext.Instance ignored = CloseableThreadContext
        .put("ipAddress", request.getRemoteAddr())
        .push("performWork()")) {

    LOGGER.debug("Performing work");
    // do work
}

Verbose (it’s Java), but it works!

Java 21 released Virtual Threads, which are somewhat like coroutines which coordinate execution across a number of OS threads. It’s not clear to me whether log4j thread contexts support them OOTB.

go

One of go’s main features is the wide array of functionality provided by its standard library, and logging context is no exception.

The standard library logging package is called slog, which supports, in the usual go way, using context.Context objects to pass structured context, but also extending logger instances themselves, via the .With call, with per instance context:

(slog also ships with a JSON formatter.)

import (
	"context"
	"log/slog"
	"os"
)

func main() {
	logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
		Level: slog.LevelInfo,
	}))
  // Add default attributes to all log entries
	baseLogger := logger.With(
		"app", "example",
		"env", "production",
	)
  slog.SetDefault(logger)

  http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
    // Extract or generate a request ID for tracing
		requestID := r.Header.Get("X-Request-ID")
		if requestID == "" {
			requestID = "default-id"
		}

    // Attach the request ID to context
		ctx := context.WithValue(r.Context(), "request_id", requestID)

    // Create request-scoped logger
		reqLogger := logger.With(
			"request_id", requestID,
			"path", r.URL.Path,
			"method", r.Method,
		)

    handleRequest(ctx, reqLogger, w, r)
  }

  http.ListenAndServe(":8080", nil)
}

func handleRequest(ctx context.Context, logger *slog.Logger, w http.ResponseWriter, r *http.Request) {
	logger.InfoContext(ctx, "Handling request") // includes request_id, path, metho
	w.Write([]byte("Request handled"))
	logger.InfoContext(ctx, "Request processed") // includes request_id, path, metho
}

While it takes some getting used to both ways of doing the same thing, it’s still interesting to see how the usage of explicit context forwarding permeates across the ecosystem, including in logging.

python

As usual with all things python, it’s all a bit of a mess, and in accordance with the “there’s always one obvious way to do something” reality, there are at least 2 ways of doing it.

BFirst, when using the standard logging package, per-call context is supported via the extra keyword argument:

logger = logging.getLogger()
logger.info("msg", extra={"foo": "bar"})

Internally, logging message calls will generate Log records, an object which contains multiple attributes, including this .extra; these records then get passed to formatters, which will access this extra context when formatting the message.

Now that we got that out of the way…

The logging package avoids extra API to support contexts, instead providing ways for an introspection-based approach, such as the logging.LoggerAdapter interface.

import logging
from flask import g

class UserAdapter(logging.LoggerAdapter):
  def process(self, msg, kwargs):
    extra = kwargs.get("extra", {})
    extra['user_id'] = g.user_id
    kwargs['extra'] = extra
    return msg, kwargs

logger = logging.getLogger(__name__)
adapter = UserAdapter(logger)

The adapter above relies on importing external context store APIs, which tend to be framework-specific; for once, the example above will only work with flask, so you may have troubles reusing this outside of it, such as, p. ex. a background task execution lifecycle (something like celery, for example). If the background task framework supports a similar imported context store API based approach, in order to reuse the adapter you’ll still have to play a game of “which execution context am I in?”. All in all, you’ll have a hard time if you want to use that local variable as context transparently on multiple log calls.

Some of these limitations can be circumvented by using the contextvars package.

Another recommendation to add contextual info is to using logging.Filter:

import logging
from flask import g

class UserFilter(logging.Filter):
  def filter(self, record):
    record.user_id = g.user_id
    return True

# later, you'll have to explicitly add the filter to the logger
logger = logging.getLogger(__name__)
f = UserFilter()
logger.addFilter(f)

Adding this to all (or a subset of) endpoints of a web application will involve a similar middleware such as what loggerAdapter provides, while having the same limitations, so I’m not sure what this abstraction buys one, besides making it a bit more explicit in some cases.

All in all, python’s approach(es) does not feel at all ergonomic, requiring boilerplate to get things done. It is truly the most low-level of high-level languages.

Beyond logging

If the feature gets accepted, most of the inconsistencies described above can be dealt with. For once, all base formatters from the libraries described above can base off the standard library Logger::Formatter, thereby standardizing on a single API and enabling reusable extensions. Adding a simpler json formatter variant will be much easier (who knows, perhaps the standard library can ship with one). rack could ship with a logging context middleware.

It also opens up quite a few opportunities for context coalescing.

For instance, logs/traces/metrics context sharing. Imagine tools like the datadog SDK, or its OTel counterpart. what if, instead of adding tags to traces only, one could add it automatically to the context of a known logger instance?

Datadog.active_logger = Rails.logger

# add as context to current active trace and log
Datadog.active_trace.set_tags("foo", "bar")
# instead of the current version, which only adds to active trace
Datadog::Tracing.active_trace.set_tags("foo", "bar")

The datadog dashboard already links traces with logs which contain a corresponding “trace_id” field. Now imagine not having to deal with the mental burden of knowing which tags are searchable in APM trace search, which ones are searchable for logs, which ones are common which ones are similar… there’d be a single context to deal with! (Now, if only datadog could listen to their users and import user-defined trace tags to trace-generated metrics…).

This could be the rug that ties the whole room together.

Rails 8 new event subscription API

If you mostly use ruby through the lens of rails, you may have looked at the recent 8.1 announcement and read about Structured Event Reporting, and may be thinking “that solves it, right?”.

Sorta, kinda, and no.

It sorta solves the problem around sending context into events. Above I complained about the append_info_to_payload being the only way to arbitrarily inject data into the event object, and this only working for the web request case. So this is a +1.

It kinda makes it work for “rails logs”, as event subscription is how rails default request/view/activerecord logs are emitted. This is probably why most of the API around Rails.event mimics some of the Rails.logger API (#tagged being the most obvious one), and hint at it being the main motivating factor behind the feature (it was developed by a Shopify employee, so you’d have to confirm with someone who works there).

But ultimately, it does not solve the main issue around logging context. Rails.logger is public API. As application users, we are encouraged to use it as the gateway to write our own logs. Event subscription is nice, but I’m not going to pivot to “emit events so I can write logs”. So while nice, it looks a bit like a rails solution to a rails problem.

What now?

This does not solve the lack of support for multiple log devices. Nor support for non-file log devices. Those are its own battles. If you feel strongly about any of them though, don’t hesitate, go ahead and propose a solution.

Origin story

The http-2 gem, is a (quote) pure ruby implementation of the HTTP/2 protocol and HPACK header compression. It’s “transport agnostic”, as in, it does not mess directly with sockets, instead accepting byte strings (via conn << bytes), and allowing callbacks to be registered, in order to be called at key moments of an HTTP/2 connection management lifecycle.

# from the README
require 'http/2'

socket = YourTransport.new

conn = HTTP2::Client.new
conn.on(:frame) {|bytes| socket << bytes }

while bytes = socket.read
 conn << bytes
end

Internally, it handles the head-scratching details of the HTTP/2 specs, such as binary frame encoding, stream multiplexing, header compression, and so on, so that, to the end-user, it almost feels like using an HTTP/1 parser. And it does all that, using approachable pure ruby code. It’s been around since 2014 (long before I planned maintaining an HTTP library), and I’d go as far as calling it the reference implementation of HTTP/2 in ruby.

So when I started toying around with building an HTTP application server, and ultimately came up with an HTTP client (httpx, no less), it was a no-brainer decision to pick http-2 for the HTTP/2 parts of it. Over time, I also became a contributor, authoring several patches, and ultimately gettinng to learn the head scratching details of the HTTP/2 protocol, which the gem initially abstracted for me.

A fork in the road

git forks serve the best spaghetti code

As httpx usage by the community picked up, so did the bug reports, some of them related to http-2. Being sort of involved in its development, I could see some cracks which weren’t evident in the beginning, namely spec compliance, and some performance issues here and there. http-2 being critical to my “HTTP library that could”, I set myself to solve the ones I was able to, and propose the patches upstream, in one pull request.

http-2 had a single maintainer at the time, Ilya Grigorik, which was also the author. I could see that, over time, he took more time to answer issues or review pull requests in github, sometimes months. Which can mean a lot of things, but if one could reduce it to common characteristics, it usually means that people are just busy with life and/or overwhelmed with “dayjob” responsibilities, and have very little, if any time left for interesting-but-ultimately-unpaid work.

The format (one single PR) in which the changes were proposed certainly presented a challenge, given the scope, even if each change was contextually in its own commit (I guess github pull request review flows aren’t optimized for that use-case yet). There were requests to break them down in shorter pull requests, but this was easier said than done (latter changes often depended on earlier changes), and ultimately demanded that I spent even more of my personal time in work that wasn’t receiving much of it from everyone else involved. This left the pull request stuck in a social deadlock, where the reviewer didn’t have time nor the motivation to review the full scope of changes, the requester didn’t have time nor the energy to adjust the scope of the changes, and the community didn’t have neither the time nor the context to help the requester nor the reviewer. The tool certainly didn’t help, but time was certainly the essence of the problem here.

This standstill was only worsened by having to regularly rebase changes and resolve the resulting conflicts from upstream, and a growing frustration from not being able to solve the production issues I ultimately needed to fix. I felt that, in order to progress with httpx, I needed to solve the problem of not owning its critical dependencies, so I needed to do something drastic.

So I forked http-2, and http-2-next was born. And httpx has been using since version 0.6.0, released around November 2019.

Good times

Fred from the metaphorical shackles of collaboration, I was finally able to improve on what was missing, and then some: compliance tests became a first-class continuous integration citizen; benchmarks were run regularly; new, more performant, ruby APIs were being used, while the gem public API remained backwards-compatible. All this contributed to improved httpx performance when benchmarked against other HTTP clients.

On the other hand, the parent was receiving very little activity (less than 10 commits since the fork).

Overall, the decision to fork was an overwhelming net-positive, for httpx, despite some hiccups along the way.

But the main drawback of the decision was, nobody was watching.

Bad times

The http-2 gem was quite popular by the time the fork happened: it’s still over 800 stars even today, and still relied upon: 711 github repositories reference it, and is a dependency from some noteworthy gems, such as the ruby AWS SDK.

There have been other “forks” as well: async-http, the HTTP workhorse of the async ecosystem, used to have it as a dependency, having been replaced meanwhile by protocol-http2, which although not officially a fork, it certainly used it as reference; tipi, a fiber-based HTTP application server, still declares it as a dependency, but its author has since forked http-2 under a new name, probably with the intent of releasing it as a separate gem.

Whether these forks happened for the same reasons as mine did is irrelevant, as the outcome should be evident: duplication effort and community fragmentation. All these forks have to solve the same issues of the original implementation (spec compliance above all), while not talking to and collaborating with each other. The ecosystems using these “forks” also ultimately determine their popularity, usage, and consequently, the conditions under which a certain category of bugs is found and reported; and when reporting them, httpx gem users will use the http-2-next repo, while users of async gems will report bugs under the protocol-http2 repo.

Only 3 bug reports have been filled overall for http-2-next (almost 2 million downloads). 4 for protocol-http2 (over 5 million). Since 2019, http-2 has had 8 bug reports (over 17 million downloads overall).

The numbers above are to be taken with a grain of salt. Bugs may have been reported in the repo of the parent gem depending on them. Nevertheless, are the low bug reports correlated with higher quality / less bugs, or lower usage? There’s not a definitive answer.

What I do know is that, despite full API compatibility with the parent gem, no other gem besides httpx declares http-2-next as a direct dependency (the same happens for protocol-http2 and async-http, but there’s no API parity there). They’ve been around for at least 5 years, so why is that? Why hasn’t the community migrated to a better alternative? Are they blind?

It turns out that such a thing rarely, if ever, happens.

You got to have a “carrot”. It can be a certification. In real life, ain’t nobody got the time to validate whether your fork improves compliance legit. There may be multiple forks around claiming the same. Who’s the regulated authority ensuring specifications are held up? What, there’s no “HTTP/2 certified seal of approval”? What, you said specs run in your CI? Sure, I’ll take your word for it…

It can be convincing prominent gems using the parent gem to switch to yours. Depending on who you’re asking it from, guarantees will be asked for. And without a certification, all that is left is trust in the fork maintainer (reliance on social capital), or usage metrics, such as github repository stats (which can be inflated by maintainer popularity, proglang userbase volume, or well-timed devrel in HN) or number of gem downloads (which can be inflated by misconfigured CIs and internet bots). Now, I hate taking decisions on dependencies based on github stars as much as the next guy, but I also work and have worked in places where convincing managers to take your side in decision logs often involves looking at a table comparing options where “measure X is bigger for option 1 than option 2” where no one really understands X, but it’s important to take decisions based on data (and in some cases yes, X was github stars, and I felt dirty).

Awareness to your fork can also be achieved in other ways. You can present it at a conference. You can write a few blog posts about it (hello there!). Ultimately that requires investing more of your time, which you may not have, and ay have ultimately been the main reason for forkig (as per above, it was the case for http-2).

And even if you do all of the above, the path of least resistance will keep most on the parent gem. Despite all of its known flaws. Despite being somewhat inactive. It’s the devil they know. It’ll fail in unexpected ways, may or may not get reported back, and the fork maintainer will have no other option but to monitor the changes from the parent repo.

To sum up, while the decision to fork was an overwhelming net-positive for httpx, that’s certainly debatable for the maintainers, and the community as a whole.

A light that never goes out

Recently, a ruby AWS SDK maintainer became a committer, and started picking up outstanding issues in the http-2 repository. It eventually stumbled in my at-the-time-still-open pull request. He promptly asked me whether I wanted to resume the work. I gave him a very short version of the history described above, and suggested using http-2-next, which was turned down as being “too difficult” (probably not technically, as per what I wrote in the previous section). He was nonetheless interested in helping remove the obstacles preventing it from having been merged in the past. So I found myself considering whether it was worth doing it.

It’s been 5 years. A lot of things were against it: http-2-next source code is primarily hosted in gitlab, and integrated with gitlab CI (readers of this blog should already know I’m a gitlab fanboy. I had since adapted code style and linting rules to my own personal preferences (for instance, I prefer having double quote strings everywhere and avoid the ambiguity of dealing with both; I know, controversial). Unexisting things like RBS type signatures. The scope of changes was therefore much greater than before, which would make reviewing it even harder than before; accomplishing it would not be possible by just cherry-picking commits from one side to the other, as both main repo and fork had moved forward, and the potential for conflicts was just too high.

On the other side of the coin, there was a lot going for it. For example, there was no breaking public API change, so it’s not like a wildly different gem being merged into another, which would have held adoption back. http-2 still has a lot more community watching the repo or reporting bugs, and that would help validate the performance and compliance benefits committed to the fork even more.

So we all sat together (virtually), and came to an agreement. http-2-next was to be ported “as-is” into the “main” branch of http-2, in one giant pull request. Once reviewed, this would become the repository HEAD. Once that was done, I’d become co-maintainer, with gem push rights.

There were compromises made: one giant commit instead of multiple smaller commits meant both that http-2 maintenanceship had to accept extra changes they perhaps would not agree to otherwise (different linting rules, for example), and http-2-next maintenanceship would lose the commit history of each change from the fork (the old repo will always be there for consultation purposes though), all in the name of reducing the overhead of getting the changes upstream and publish a release. It also meant I had to say goodbye to gitlab CI and just learn how to bake the same cake with Github Actions, although some things were lost along the way; for instance, I was able to publish coverage docs in gitlab and link to them on the coverage badge, and I still don’t know how to generate coverage badges in Github Actions, nor how to make coverage docs publicly available (if someone knows how to do it, I’ll wait for your pull request:) ).

It took what it had to take, but we did it! http-2 1.0.0 was released in June 2024, and, 5 years after, httpx 1.4.0 became the first version since 0.6.0 to declare http-2 as a dependency.

Conclusion

I wrote this post as a celebration of a fork successfully being merged back into the mothership. This is not just about me, the ruby community, or my own particular gem drop in the rubygems ocean. Generally, this type of event is the exception, not the rule. In the FOSS world, forks are allowed, and encouraged. And for many good reasons. It’s empowering. It’s liberating. It can help breed innovation. But sometimes, they’re unnecessary fragmentation. Of contributors, and users. They generate effort duplication. They may lead to competting efforts in an environment where there may ultimately be no trophy at the end of the line, rather an inbox full of angry users and bug reports, or complete silence, and ultimately burnout. And when you realize it, it’s too late, or costly, to go back.

Back then, I was so obsessed with the idea of “killing” my dependencies, that I couldn’t see the bigger picture. In hindsight, if I could do things differently, I would have tried to contact Ilya in order to figure out whether I could help with reducing his burden, perhaps not being fearful of suggesting becoming a maintainer and getting a no for an answer. Essentially, just try to solve the social collaboration problem first, before jumping into implementing a technical solution.

Raise your glass to all forks, old and new, dead and gone, alive and well! May they all find their way back to the Source!

Every year, a few articles come out with a title similar to “the best ruby http clients of the year of our lord 20xx”. Most of the community dismisses them as clickbait, either because of the reputation of the content owner website, companies pushing their developers to write meaningless content in their company tech blog for marketing purposes, or AI bots trained on similar articles from the previous decade and serving you the same contet over and over.

And they’re right. Most of the times, these articles are hollow, devoid of meaningful examples or discussions about relevant features, trade-offs or performance characteristics, and mostly rely on shallow popularity metrics such as total downloads, number of stars on GitHub, or number of twitter followers from the core maintainer, to justify selections. They’ll repeat what you know already for years: faraday is downloaded 20 million times a year, httparty parties hard, no one likes net-http, and there are too many http clients in the ruby community.

These articles very rarely mention newcomers. Being the developer of httpx, a relatively recent (created in 2017) HTTP client, and having extensively researched the competition, I can’t help but feel that there’s a lot that hasn’t been mentioned yet. So, given the context I gathered all over these years, I believe I can myself do the article I’d like someone else to have done already about the topic but didn’t.

Alas, this is yet another “the state of ruby HTTP clients in 2023”. There are many like it, but this one is mine. And while you’ll find it hardly surprising that I recommend you to use httpx nowadays (I’m the maintainer after all), I’ll try to make the analysis as unbiased as possible, and play the devil’s advocate here and there.

Population

As of the time of writing this article, there are 33 http client gems listed in ruby toolbox. It takes a book to cover them all! How can I limit the sample to relevant gems only? What classifies as “relevant” anyway?

While the ruby toolbox ranking suffers from the “social” factor as well (github and number of stars are an important metric in their score calculation after all), it does collect data around maintenance health, which is a variable to take into account.

Categorization is not very precise either; for instance, some of the listed gems are hardly HTTP “clients”, rather a layer built on top of other HTTP clients instead. For instance, flexirest or restfulie are DSLs around “RESTful API” concepts; hyperclient is a DSL to build HAL JSON API clients; json_api_client does the same for APIs following the JSON API Spec; all of them are using net-http, ruby’s own standard library include HTTP client, under the hood though. So one can dismiss them as not really HTTP clients.

Some of the listed gems can’t event perform HTTP requests. For instance, multipart-post, the second best-ranked by project score index, is essentially a group of components to be used with net-http to enable generation of multipart requests. You still have to use net-http directly though! There are other gems of this kind (I’ll address them later) which aren’t part of this list either.

Filtering by these two metrics alone, we come to a much shorter list of candidates, which most rubyists should be familiar with:

• faraday
• excon
• rest-client
• httparty
• httpclient
• typhoeus
• HTTPrb
• mechanize
• httpi
• curb
• em-http-request
• httpx
• net-http

But we can go even further.

Active maintenance

While I don’t personally measure gems by the change rate of the source code, as I believe that there’s a thing such as considering a piece of software as “feature complete”, one can’t apply that line of thought to gems having frequent complaints and bug reports, with barely a response from any maintainer. And there are entries in our remaining list of candidates which, although very popular based on number of downloads and GitHub stars, haven’t been very (if at all) responsive to user feedback in the last couple of years.

Take rest-client for example: one of the oldest and most downloaded gems of the list, its last release was in 2019, with several unanswered bug reports and open pull requests since then.

httpclient, even older that rest-client, is in an even worse condition: last released in 2016(!), several unanswered issues, including this one which is particularly concerning, and should render the gem unusable.

For another example, there’s also typhoeus, last released in 2020, with several open issues as well.

While maintainers shouldn’t be criticized for exercising the freedom of leaving their maintenance duties behind, I find it concerning nonetheless that articles keep popping up recommending their orphaned gems. Consider as well that these gems are still reverse dependencies of thousands of other gems. As an example, typhoeus is the default HTTP client library in openapi-generator, which automates the generation of API client SDKs in several programming languages (including ruby).

So while I’ll probably mention some of them here and there, I won’t further analyse any of the alternatives which are de facto unmaintained.

Wrappers, wrappers everywhere

When it comes to HTTP clients in ruby, there are 3 main groups:

• Those which wrap net-http
• Those which wrap curl
• Everything else

On top of these, you’ll find the “general wrappers” which integrate with as many HTTP “backends” as possible, and aim at providing common interfaces and functionality on top. This group includes faraday, the best-ranked gem by project score in Ruby Toolbox, and httpi, which is a transitive dependency of savon, the most popular ruby SOAP client. This means that, for most of the purposes of this article’s research, they’re irrelevant, although I’ll still include faraday due to its popularity.

Faraday

faraday provides a common HTTP API, and an integration layer every client can integrate with, and distributes common functionality around. In a nutshell, it aims at doing what rack did for application servers: provide a “common middleware” and enable switching the “engine”. Its mirroring of rack’s stragegy goes beyond that, as it even copies some of its quirks, such as the rack env, all the way to “status - headers - body” interface, and the concept of middlewares.

Its approach has had undeniable success: not only the most downloaded, it’s also the HTTP client gem with the most reverse dependencies. Nevertheless, it’s far from the “one true way” of putting HTTP requests in front of people.

For once, it does not guarantee full feature coverage for all supported backends: while one can argue whether this can be made feasible or not, maintenance of the integration layer requires decent knowledge of both faraday and the underlying HTTP client, for each of the supported clients, and there isn’t enough skill around with the time and motivation to do it. So just assume that there’s always something which will be missing for a given integration, some feature which was recently added, some feature which only exist in that particular backend, and so on. Which makes the advantage of possibly switching backends heavily constrained by how deeply the faraday featureset is used.

Moreover, the features it offers (usually via middlewares) often repeat functionality already provided by some of the backends, and sometimes incomplete in comparison. For instance, faraday provides HTTP auth, json encoding, or multipart encoding, as features; however, it only supports Basic HTTP auth (some backends support other schemes authentication schemes, such as Digest HTTP auth). Also, some of the backends already deal with multipart requests (in some cases in a more complete manner, we’ll get to that later), and dealing with JSON may arguably not be a “hard” problem worth having a middleware for (the json standard library makes that already quite easy). Some of the value of these middlewares is therefore a bit dilluted, at least when not dealing with more involved features (like dealing with retries, for instance).

Moreover, by basing itself on the rack protocol, it also inherits its problems. rack API, although simple, ain’t easy. Consider the lowest common denominator:

def call(env)
  [200, {}, ["Hello World"]]
end

That env variable isn’t self-explanatory; it’s a bucket of key-value junk. And while the rack spec does a reasonable job of specifying which keys must or should be there and what they should point to, faraday does not provide a specification. So env ends up being an undefined “object which is an hash?”, where you can call things such as env.request, env.ssl, env.body, env[:method]or env[:parallel_manager], and the only way to know which is which, is by reading the code of existing adapters and hope/test you’re using the right thing. All of that for the convenience of having something similar to rack, because it makes things… simple? 🤷

Building features on top of middleware was also a mistake inherited from rack in hindsight. Order matters.

To sum up, although faraday treats the backends it integrates with as dump pipes, they’re rarely dumb. Its choices in integration path also make it rather limiting when building adapters for it, and “spread ownership” from having adapters as its own separate gems (a decision of faraday maintainers) results in adapters covering a “low common denominator” subset of features - which makes it hard to switch adapters - so gems integrating with faraday usually settle with just one. Its user-facing API is reasonably ok (if you forget about parallel requests of multipart support); however, most third-party SDK/gems based on faraday just treat it as an implementation detail, and end up not exposing faraday connections to end users to “augment with middlewares” or even changing backend. And they’ll have to deal with its other quirks. The stripe gem decided not to wait any longer for that upside.

So if you want an HTTP client to implement an SDK on top of, do your research and pick up your own HTTP client, instead of faraday.

Wrapping curl

curl is the most widely used HTTP client in all of software. It’s probably top 10 in most used software in general. It’s used even in Mars. This is synonym to “battle-tested”, “fully-featured”, and “performant”. Being written in C, it’s no wonder that, for a multitude of runtimes with any sort of C ABI interoperability, there are a lot of wrappers for it. And ruby is no exception: typhoeus, curb and patron at least, are all libcurl wrappers, interfacing with it either via libffi or C extensions.

This is no free lunch either. For once, HTTP is only one of the many protocols supported by curl for transfers. The integration will therefore have to make sure that no other protocol can be abused (and, for example, some vulnerable FTP code path is accidentally called), only possible by custom-building curl with support for only HTTP; however, in most cases, integrations will often target the system-installed libcurl, which is open-ended in that regard.

This, on the other hand, makes deployments and dependency tracking harder: now you’ll have to follow changes and security announcements related both to the ruby HTTP library and libcurl. Otherwise, how will you know that a bugfix has been released, or worse, a security fix? (Did I already mention that libcurl is written in C? Here’s a recent reminder.) You’ll also need to ensure that the version of libcurl you want to compile against is installed in your production servers, which makes server setups (containers or not) more cumbersome to maintain: installing curl, or libcurl, is usually something left for the system package manager to handle (aptget, yum, brew…), but these tend to take years to adopt the “latest greatest” version of libcurl, in this case the one containing that security fix you so desperately need. So you’ll have to do the work of downloading, unpacking and installing it as a pre-compiled system package (don’t forget to do the same with the several libcurl dependencies, like libidn2, or nghttp2, etc…). To mitigate some of the pain associated with this, it’s usually best practice that the ruby interface ends up supporting multiple versions of libcurl which may be installed, at the cost of increased risk and maintenance overhead for the gem maintainers.

Alternatively, you can include it as an on-the-fly-compiled vendored C dependency from the gem. That will come with its own can of worms though. Even FFI-based integrations aren’t free of system-related problems. This is the type of overhead that a pure ruby package does not incur.

Usability of the gem API is also a problem. However good libcurl API is, it is idiomatic C, not idiomatic ruby. And for all its efforts in hiding the details of libcurl API, these tend to leak into the surface of end user ruby code:

# using typhoeus
case response.code
when 200
  # success
when 0
  # special curl code for when something is wrong

# using curb
# curl_easy and curl_multi are C-level libcurl interfaces
# curb exposes them to ruby code almost "as is"
c = Curl::Easy.new("https://http2.akamai.com")
# this is the C-way how conn options are set (this one enables HTTP/2). So one line for each...
c.set(:HTTP_VERSION, Curl::HTTP_2_0)

This could probably be worth it if there’d be a huge feature gap, or the performance was much greater than the non-curl based alternatives, but this is not the case either (more about this later).

So from the standpoint of coding in ruby, I don’t see many advantages which justify the downsides of choosing a library wrapping libcurl.

Wrapping net-http

net-http is the standard library HTTP client. Because it ships with ruby, it’s probably (because I don’t have numbers to back it up, but still, high degree of certainty) the most widely used ruby HTTP client. A significant portion of that usage is indirect though, given how many gems out there wrap it (httparty and rest-client most notably; faraday default adapter is also for net-http).

And that’s because nobody likes writing net-http code. And it’s easy to see why, just look at this cheatsheet: its API is convoluted, verbose, needlessly OO-heavy (why does one need an exception for every HTTP error status code…), it just does not enact joy. Worse, there’s no fix for that: because it’s standard library, and its clunky API is relied up almost as much as ruby core syntax, it’s resistant to change, so its clunkiness is relied upon in a lot of legacy code; any change to address the mentioned points risks having a wide “blast radius” and breaking a significant portion of ruby production deployments.

For this reason, and for a while already, (httparty first release is from 2008!), several libraries have been released with the expressed goal of exposing a user-friendlier DSL for doing HTTP requests, while abstracting the difficulty of dealing with net-http API internally. Off this wave, the “one that parties hard” and rest-client have been the most popular ones. The improvements are perceived by many to offset the drawbacks of the using net-http, while still retaining the whole “engine” intact. This creates a whole new set of problems though.

One is “feature parity drift”. net-http has many features AND lacks key features, but still receives active development, sometimes addresses the latter. For a wrapper, this means that, there’s always going to be a subset of recent functionality which hasn’t been properly wrapped yet. httparty took years to include configuration to cover all possible net-http options: just in 2018, I remember ranting about not being able to enable net-http’s debug output from its API, an option supported in net-http at least since the ruby 1.8.7. days; and somewhere, someone’s still waiting for max_retries support to be added to rest-client.

Another is “implementation multiplication”. net-http lacks some basic core functionality one would expect from an HTTP client, like support for multipart request or digest auth; so faraday has to fill in the gaps, just like faraday, or rest-client, and this despite known patches to net-http itself being developed by the community, all of which is a massive repetition of effort, where certain edge-case bugs may be present in some but not in others, clearly not the most efficient use of a community time and energy.

And meanwhile, new features arrive in net-http every year; it being in standard library, there’s always someone pushing for new features to be added, which reflects in “continuous overhead” for wrapper maintainers, which are required to perpetually shim the new functionality. If the wrappers are maintained at all, that is (rest-client hasn’t since a release in 3 years, so as good as “unmaintained”).

So while I agree with the overall sentiment that net-http is not code I like reading or maintaining, and that its existence only reflects badly on ruby itself (no one will take a “ruby is beautiful” statement seriously by looking at its stdlib HTTP-related code), on the other hand, given the situation I just described, and economy of dependencies trumps freedom of solution choice, using net-http straight up is a better option than sticking with one of its wrappers.

Evaluation

So far, one can see that, although there seems to be plenty of choice, there’s actually a short list one can reasonably hold on to:

• faraday
• excon
• rest-client (no release in the last 3 years, high number of unanswered issues)
• httparty
• httpclient (no release in the last 3 years, high number of unanswered issues)
• typhoeus (no release in the last 3 years, high number of unanswered issues)
• HTTPrb
• mechanize
• httpi (fringe HTTP client wrapper, no release in almost 2 years)
• curb
• em-http-request
• httpx
• net-http

I’m also removing em-http-request and mechanize from this list. About em-http-request, despite its low-but-existing activity rate, its adoption hangs on it being used via an async framework, eventmachine, which itself hasn’t seen much activity lately, and has fallen out of use and popularity due to its API and runtime incompatibility with “standard” ruby network code. About mechanize, despite it technically being an HTTP client, it’s mostly a “web scraping” tool which interacts with webpages (fill up forms, click links, etc…), impersonating the role of a browser (which is also technically an HTTP client).

So now that we have a defined sample for the analysis, let’s begin.

UX / Developer ergonomics

Response

The most basic feature required from an HTTP client library is performing GET requests (for example, to download a webpage). And that’s a feature that any library mentioned in this article so far (and all the others that haven’t, most probably), is able to easily perform. In fact, it’s so easy, that you can achieve it using similar API for all them:

# please download google front page
uri = "https://www.google.com"
response = HTTPX.get(uri) # httpx
response = Excon.get(uri) # Excon
response = Faraday.get(uri) # faraday
response = HTTP.get(uri) # HTTPrb
response = HTTParty.get(uri) # httparty
response = Curl.get(uri) # curb
response = Net::HTTP.get_response(URI(uri))  # even net-http manages to inline

The response object that each of these calls returns will be a bit “different but similar” in most situations: some will return the response status code via a .status method, while others call it .code:

response.status #=> 200, for httpx, excon, faraday
response.code #=> 200, for HTTPrb, httparty, curb
response.code #=> "200", why, net-http…

The response object will also allow access to the response HTTP headers, in most of cases via a .headers method. The returned object is not always the same, although in most cases is, at the very least, something which allows [key] based lookups, and which can be turned into a Hash:

# httpx
response.headers #=> a custom class, which implements basic [] and []=, responds to .to_h
# excon
response.headers #=> instance of a custom class inheriting from Hash
# faraday
response.headers #=> instance of a custom class inheriting from Hash
# HTTPrb
response.headers #=> a custom class, which implements basic [] and []=, responds to .to_h
# httparty
response.headers #=> a custom SimpleDelegator (to a Hash) class
# curb
response.headers #=> a Hash
# net-http
response.header #=> a custom class, which is HTTPSuccess when 200, something else otherwise….

# all support case-insensitive lookup
response.headers["content-type"] #=> "text/html; charset=ISO-8859-1"
response.headers["Content-Type"] #=> "text/html; charset=ISO-8859-1"

# only httpx provides access to multi-value header
response.headers["set-cookie"] #=> "SOCS=CA…; AEC=AUEFqZe…; __Secure-ENID=12.SE=A8"
response.headers.get("set-cookie") #=> ["SOCS=CA…", "AEC=AUEFqZe…", "__Secure-ENID=12.SE=A8"] , accesses each "set-cookie" response header individually

Finally, the response object allows retrieving the response body, usually via a .body method. As with the example above, the returned object is not always the same, but at the very least can be turned into a String, and in some cases, can be handled as a “file”, i.e. can be read in chunks, which is ideal when dealing with chonky payloads. In some cases, there is custom API for decoding well known encoding formats into plain ruby objects:

# httpx
response.body #=> a custom class
response.to_s #=> a ruby string
response.form #=> if "application/x-www-form-urlencoded" content-type, returns the ruby Hash
response.json #=> if "application/json" content-type, returns the ruby Hash
# excon
response.body #=> a ruby string
# and that's it, no shortcut for decoding
# faraday
response.body #=> a ruby string
# HTTPrb
response.body #=> a custom class, which implements .to_s and .readpartial
# httparty
response.body #=> a ruby string
#faraday
conn = Faraday.new('https://httpbin.org') do |f|
  # json decoder supported via faraday middleware
  f.response :json
end
json = conn.get("/get").body # already a ruby Hash
# curb
response.body #=> a ruby string
# net-http
response.body #=> a ruby string.

# --------

big_file_url = 'https://some-cdn.com/path/to/file'

# httpx and HTTPrb support chunked response streaming via implementations of .read
# or .readpartial, so this is possible with both:

response = HTTPX.get(big_file_url) # httpx
response = HTTP.get(big_file_url) # HTTPrb

IO.write("/path/to/file", response.body)
# HTTPX has an API just for this:
response.body.copy_to("/path/to/file")
# both also implement .each, which yield chunks
response.body.each { |chunk| handle_chunk(chunk) }

# other options have their own bespoke "read in chunks" callback

# excon
File.open("/path/to/file", "wb") do |f|
  streamer = lambda do |chunk, remaining_bytes, total_bytes|
    f << chunk
  end
  Excon.get(big_file_url, :response_block => streamer)
end

# faraday
File.open("/path/to/file", "wb") do |f|
  Faraday.get(big_file_url) do |req|
    req.options.on_data do |chunk, overall_received_bytes, env|
      f << chunk
    end
  end
end

# httparty
File.open("/path/to/file", "wb") do |f|
  HTTParty(big_file_url, stream_body: true) do |fragment|
    if fragment.code == 200 # yup, you gotta test fragments….
      f << fragment
    end
  end
end

# curb
File.open("/path/to/file", "wb") do |f|
  c = Curl::Easy.new(big_file_url)
  c.on_body {|data| f << data}
  c.perform
end

# net-http
File.open("/path/to/file", "wb") do |f|
  u = URI(big_file_url)
  Net::HTTP.start(u.host, u.port) do |http|
  request = Net::HTTP::Get.new(u)
  http.request(request) do |response|
    response.read_body do |chunk|
       f << chunk
    end
  end
end

And this is where the first usability differences are noticeable: 1) httpx and httprb both make the task of dealing with response body chunking a bit more intuitive than the rest, which rely on “same but different” blocks; 2) httpx provides a few shortcuts to parse well-known mime-types into ruby objects (faraday does the same for JSON via some middleware boilerplate); 3) ruby stdlib mitigates some of the shortcomings of other libraries by supporting decoding of common mime types natively (JSON.parse(response.body) for strings works well enough).

Request

Another common feature that all HTTP clients support is requests with other HTTP verbs, such as POST requests. This usually requires support for passing the request body, as well as the setting headers (a feature which is also useful for GET requests btw) in a user-friendly manner.

In order to use another HTTP verb, most libraries will rely on a same-named downcased method, while relying on more or less verbose options to pass extra parameters:

# use-cases:
# 1. GET with the "x-api-token: SECRET" header
# 2. GET with the "?foo=bar" query param in the request URL
# 3. POST the "data" string
# 4. POST the "foo&bar" urlencoded form data
# 5. POST the '{"foo":"bar"}' JSON payload
# 6. POST the '{"foo":"bar"}' JSON payload with the "x-api-token: SECRET" header
get_uri = "https://httpbin.org/get"
post_uri = "https://httpbin.org/post"

# httpx
# 1.
response = HTTPX.get(get_uri, headers: { "x-api-token" => "SECRET" })
# 2.
response = HTTPX.get(get_uri, params: { "foo" => "bar" })
# 3.
response = HTTPX.post(post_uri, body: "data") # defaults to "application/octet-stream" content-type
# 4.
response = HTTPX.post(post_uri, form: { "foo" => "bar" })
# 5.
response = HTTPX.post(post_uri, json: { "foo" => "bar" })
# 6.
response = HTTPX.post(post_uri, headers: { "x-api-token" => "SECRET" }, json: { "foo" => "bar" })

# excon
# 1.
response = Excon.get(get_uri, headers: { "x-api-token" => "SECRET" })
# 2.
response = Excon.get(get_uri, query: { "foo" => "bar" })
# 3.
response = Excon.post(post_uri, body: "data") # does not specify content type

# excon does not provide shortcuts for encoding the request body
# in well known encoding formats, so DIY.
# 4.
response = Excon.post(post_uri, :body => URI.encode_www_form('foo' => 'bar'), :headers => { "Content-Type" => "application/x-www-form-urlencoded" })
# 5.
response = Excon.post(post_uri, :body => JSON.dump('foo' => 'bar'), :headers => { "Content-Type" => "application/json" })
# 6.
response = Excon.post(post_uri, :body => JSON.dump('foo' => 'bar'), :headers => { "Content-Type" => "application/json", "x-api-token" => "SECRET" })

# faraday
# 1.
# starting on the wrong foot, here's a 2nd argument that needs to be nil...
response = Faraday.get(get_uri, nil, { "x-api-token" => "SECRET" })
# 2.
# depending on whether GET or POST, the 3rd argument is either transformed
# into a URL query string or POST form body
response = Faraday.get(get_uri, { "foo" => "bar" }, { "x-api-token" => "SECRET" })
# 3.
response = Faraday.post(post_uri, "data") # defaults to application/x-www-form-urlencoded content-type
# 4.
response = Faraday.post(post_uri, {"foo" => "bar"}) # can encode ruby objects to default
# 5.
conn = Faraday.new('https://httpbin.org') do |f|
  # json encoder supported, again via more middleware boilerplate
  f.request :json
end
response = conn.post("/post", {"foo" => "bar"})
# 6.
response = conn.post("/post", {"foo" => "bar"}, { "x-api-token" => "SECRET" })

# HTTPrb
# 1.
response = HTTP.headers("x-api-token" => "SECRET").get(get_uri)
# 2.
response = HTTP.get(get_uri, params: { "foo" => "bar" })
# 3.
response = HTTP.post(post_uri, body: "data") # does not specify content type...
# 4.
response = HTTP.post(post_uri, form: {"foo" => "bar"})
# 5.
response = HTTP.post(post_uri, json: {"foo" => "bar"})
# 6.
response = HTTP.headers("x-api-token" => "SECRET").post(post_uri, json: {"foo" => "bar"})

# httparty
# 1.
response = HTTParty.get(get_uri, headers: { "x-api-token" => "SECRET" })
# 2.
response = HTTParty.get(get_uri, query: { "foo" => "bar" })
# 3.
response = HTTParty.post(post_uri, body: "data") # defaults to application/x-www-form-urlencoded content-type
# 4.
response = HTTParty.post(post_uri, body: {"foo" => "bar"}) # can encode ruby objects to default as well
# 5.
# no shortcut provided for json, DIY
response = HTTParty.post(post_uri, body: JSON.dump({"foo" => "bar"}), headers: {"content-type" => "application/json"})
# 6.
response = HTTParty.post(post_uri, body: JSON.dump({"foo" => "bar"}), headers: {"x-api-token" => "SECRET", "content-type" => "application/json"})

# curb
# 1.
response = Curl.get(get_uri) do |http|
  http.headers['x-api-token'] = 'x-api-token'
end
# 2.
response = Curl.get(Curl.urlalize(get_uri, {"foo" => "bar"}))
# 3.
response = Curl.post(post_uri, "data") # defaults to application/x-www-form-urlencoded content-type, like curl
# 4.
response = Curl.post(post_uri, {"foo" => "bar"})
# 5.
# needs block-mode to add headers...
response = Curl.post(post_uri, JSON.dump({"foo" => "bar"})) do |http|
  http.headers["content-type"] = "application/json"
end
# 6.
response = Curl.post(post_uri, JSON.dump({"foo" => "bar"})) do |http|
  # one of these for each new header you'll need to add...
  http.headers["content-type"] = "application/json"
  http.headers["x-api-token"] = "SECRET"
end

# net-http
get_uri = URI(get_uri)

# 1. and 2.
# net-http does not provide query params API, you have to use URI for that
get_uri.query = URI.www_encode_form({"foo" => "bar"})
# and now you can do the request...

http = Net::HTTP.new(get_uri.host, get_uri.port)
request = Net::HTTP::Get.new(get_uri.request_uri)
request["x-api-token"] = "SECRET"
response = http.request(request)

# 3.
post_uri = URI(post_uri)
response = Net::HTTP.post(post_uri, "data")  # defaults to application/x-www-form-urlencoded content-type

# 4.
response = Net::HTTP.post_form(post_uri, {"foo" => "bar"})

# 5.
http = Net::HTTP.new(post_uri.host, post_uri.port)
request = Net::HTTP::Post.new(post_uri.request_uri)
request["content-type"] = "application/json"
request.body = JSON.dump({"foo" => "bar"}
response = http.request(request)

# and let's forget the last, I'm tired of writing net-http examples. you get the picture from the above

This is not exhaustive, but it does tell one a few things: 1) net-http starts showing how verbose can it get; 2) For most options, API shortcuts for encoding the request body are quite limited beyond “x-www-form-urlencoded”; 3) some clients get a bit too creative with the usage of blocks; 4) faraday positional arguments make it a bit confusing to do simple requests. 5) httpx and httprb manage to achieve all examples in concise one-liners; 6) As in the previous section, ruby has quite a lot of stdlib support to circumvent some of these shortcomings (via uri or json bundled gems).

Multipart

Another common and widely supported encoding format for upload files is multipart/form-data, aka Multipart. While a common and old standard, even supported by browsers for form submission, it’s surprising to find that some HTTP clients either don’t implement, require a separate dependency for it, or implement it partially. Let’s demonstrate:

# please:
# 1. POST a "document.jpeg" file
# 2. POST a "selfie.mp4" file
# 3. POST a "document.jpeg" file and a "selfie.mp4" file
# 4. POST a "document.jpeg" file, a "selfie.mp4" file, and a "name=Joe" text field
# 5. POST a "document.jpeg" file, a "selfie.mp4" file, and a "{"name": "Joe", "age": 20}" JSON "data" field
post_uri = "https://httpbin.org/post"
doc_path = "/path/to/document.jpeg"
selfie_path = "/path/to/selfie.mp4"

# httpx
# 1.
HTTPX.post(post_uri, form: { document: File.open(doc_path) })
# multipart payload
# single part with name="document", filename="document.jpg" and content-type=image/jpeg

# 2.
HTTPX.post(post_uri, form: { selfie: Pathname.new(selfie_path) }) # also supports pathnames
# multipart payload
# single part with name="selfie", filename="selfie.mp4" and content-type=video/mp4

# 3.
HTTPX.post(post_uri, form: { document: File.open(doc_path), selfie: File.open(selfie_path) })
# multipart payload
# first part with name="document", filename="document.jpg" and content-type=image/jpeg
# second part with name="selfie", filename="selfie.mp4" and content-type=video/mp4

# 4.
HTTPX.post(post_uri, form: { document: File.open(doc_path), selfie: File.open(selfie_path), name: "Joe" })
# first part with name="document", filename="document.jpg" and content-type=image/jpeg
# second part with name="selfie", filename="selfie.mp4" and content-type=video/mp4
# third part with name="name", content-type=text/plain

# 5.
HTTPX.post(post_uri, form: { document: File.open(doc_path), selfie: File.open(selfie_path), data: { content_type: "application/json", body: JSON.dump({name: "Joe", age: 20}) }})
# first part with name="document", filename="document.jpg" and content-type=image/jpeg
# second part with name="selfie", filename="selfie.mp4" and content-type=video/mp4
# third part with name="data", content-type=application/json


# excon
# does not support multipart requests

# faraday
# does not support multipart requests OOTB
# requires separate faraday-multipart extension gem for that: https://github.com/lostisland/faraday-multipart
require 'faraday'
require 'faraday/multipart'

conn = Faraday.new do |f|
  f.request :multipart
end
# 1.
conn.post(post_uri, {document: Faraday::Multipart::FilePart.new(File.open(doc_path), 'image/jpeg') })
# requires using a specific faraday-multipart class for file parts
# mime types need to be known ahead of time!

# 2.
conn.post(post_uri, {selfie: Faraday::Multipart::FilePart.new(File.open(selfie), 'video/mp4') })

# 3.
conn.post(post_uri, {
  document: Faraday::Multipart::FilePart.new(File.open(doc_path), 'image/jpeg'),
  selfie: Faraday::Multipart::FilePart.new(File.open(selfie), 'video/mp4')
})

# 4.
conn.post(post_uri, {
  document: Faraday::Multipart::FilePart.new(File.open(doc_path), 'image/jpeg'),
  selfie: Faraday::Multipart::FilePart.new(File.open(selfie), 'video/mp4'),
  name: "Joe"
})
# when it comes to text/plain, you can just pass a string

# 5.
conn.post(post_uri, {
  document: Faraday::Multipart::FilePart.new(File.open(doc_path), 'image/jpeg'),
  selfie: Faraday::Multipart::FilePart.new(File.open(selfie), 'video/mp4'),
  data: Faraday::Multipart::ParamPart.new(
    JSON.dump({name: "Joe", age: 20}),
    'application/json'
  )
})
# separate custom part class for other encodings!

# HTTPrb
# does not support multipart OOTB
# requires separate "http/form_data" gem: https://github.com/httprb/form_data
# 1.
HTTP.post(post_uri, form: { document: HTTP::FormData::File.new(doc_path, content_type: "image/jpeg") })
# requires using a specific http/form_data class for file parts
# mime types need to be known ahead of time!

# 2.
HTTP.post(post_uri, form: { selfie: HTTP::FormData::File.new(selfie_path, content_type: "video/mp4") })

# 3.
HTTP.post(post_uri, form: {
  document: HTTP::FormData::File.new(doc_path, content_type: "image/jpeg"),
  selfie: HTTP::FormData::File.new(selfie_path, content_type: "video/mp4")
})

# 4.
HTTP.post(post_uri, form: {
  document: HTTP::FormData::File.new(doc_path, content_type: "image/jpeg"),
  selfie: HTTP::FormData::File.new(selfie_path, content_type: "video/mp4"),
  name: "Joe"
})
# encodes strings as text/plain

# 5.
HTTP.post(post_uri, form: {
  document: HTTP::FormData::File.new(doc_path, content_type: "image/jpeg"),
  selfie: HTTP::FormData::File.new(selfie_path, content_type: "video/mp4"),
  name: HTTP::FormData::Part.new(JSON.dump({name: "Joe", age: 20}), content_type: 'application/json')
})
# separate custom part class for other encodings!


# httparty
# some built-in multipart capabilities in place

# 1.
HTTParty.post(post_uri, body: { document: File.open(doc_path) })
# multipart payload
# single part with name="document", filename="document.jpg" and content-type=image/jpeg

# 2.
HTTParty.post(post_uri, body: { selfie: File.new(selfie_path) })
# multipart payload
# single part with name="selfie", filename="selfie.mp4" and content-type=application/mp4
# The content-type is wrong!

# 3.
HTTParty.post(post_uri, body: {
  document: File.open(doc_path),
  selfie: File.open(selfie_path)
})
# multipart payload
# first part with name="document", filename="document.jpg" and content-type=image/jpeg
# second part with name="selfie", filename="selfie.mp4" and content-type=application/mp4 (Wrong!)

# 4.
HTTParty.post(post_uri, body: {
  document: File.open(doc_path),
  selfie: File.open(selfie_path),
  name: "Joe"
})
# first part with name="document", filename="document.jpg" and content-type=image/jpeg
# second part with name="selfie", filename="selfie.mp4" and content-type=application/mp4 (Wrong!)
# third part with name="name", content-type=text/plain

# 5.
# passing a custom json part is not supported!

# curb
# requires more calls to set it up
# 1.
c = Curl::Easy.new(post_uri)
c.multipart_form_post = true
c.http_post(Curl::PostField.file('document', doc_path))
# multipart payload
# single part with name="document", filename="document.jpg" and content-type=image/jpeg

# 2.
c = Curl::Easy.new(post_uri)
c.multipart_form_post = true
c.http_post(Curl::PostField.file('selfie', selfie_path))
# multipart payload
# single part with name="selfie", filename="selfie.mp4" and content-type=application/octet-stream
# this mime-type is wrong!

# 3.
c = Curl::Easy.new(post_uri)
c.multipart_form_post = true
c.http_post(
  Curl::PostField.file('document', doc_path),
  Curl::PostField.file('selfie', selfie_path))
# multipart payload
# first part with name="document", filename="document.jpg" and content-type=image/jpeg
# second part with name="selfie", filename="selfie.mp4" and content-type=application/octet-stream (Wrong!)

# 4.
c = Curl::Easy.new(post_uri)
c.multipart_form_post = true
c.http_post(
  Curl::PostField.file('document', doc_path),
  Curl::PostField.file('selfie', selfie_path),
  Curl::PostField.content('name', "Joe"))
# first part with name="document", filename="document.jpg" and content-type=image/jpeg
# second part with name="selfie", filename="selfie.mp4" and content-type=application/octet-stream (Wrong!)
# third part with name="name", content-type=text/plain

# 5.
c = Curl::Easy.new(post_uri)
c.multipart_form_post = true
c.http_post(
  Curl::PostField.file('document', doc_path),
  Curl::PostField.file('selfie', selfie_path),
  Curl::PostField.content('data', JSON.dump({name: "Joe", age: 20}), "application/json"))
# first part with name="document", filename="document.jpg" and content-type=image/jpeg
# second part with name="selfie", filename="selfie.mp4" and content-type=application/octet-stream (Wrong!)
# third part with name="data", content-type=application/json

# net-http
# does not support multipart requests
# you can use the previously mentioned multipart-post gem
# https://github.com/socketry/multipart-post
require "net/http"
require 'net/http/post/multipart'

url = URI.parse(post_uri)


# 1.
File.open(doc_path) do |file|
  req = Net::HTTP::Post::Multipart.new(
    url.path,
    "document" => UploadIO.new(file, "image/jpeg")
  )
  res = Net::HTTP.start(url.host, url.port, use_ssl: true) do |http|
    http.request(req)
  end
end
# uses multipart-post provided class to build part
# mime type needs to be known ahead of time!


# 2.
File.open(selfie_path) do |file|
  req = Net::HTTP::Post::Multipart.new(
    url.path,
    "selfie" => UploadIO.new(, "video/mp4")
  )
  res = Net::HTTP.start(url.host, url.port, use_ssl: true) do |http|
    http.request(req)
  end
end

# 3.
File.open(doc_path) do |doc_file|
  File.open(selfie_path) do |selfie_file|
    req = Net::HTTP::Post::Multipart.new(
      url.path,
      "document" => UploadIO.new(doc_file, "image/jpeg"),
      "selfie" => UploadIO.new(selfie_file, "video/mp4")
    )
    res = Net::HTTP.start(url.host, url.port, use_ssl: true) do |http|
      http.request(req)
    end
  end
end

# 4.
File.open(doc_path) do |doc_file|
  File.open(selfie_path) do |selfie_file|
    req = Net::HTTP::Post::Multipart.new(
      url.path,
      "document" => UploadIO.new(doc_file, "image/jpeg"),
      "selfie" => UploadIO.new(selfie_file, "video/mp4"),
      "name" => "Joe"
    )
    res = Net::HTTP.start(url.host, url.port, use_ssl: true) do |http|
      http.request(req)
    end
  end
end
# text inputs will be encoded as text/plain

# 5.
File.open(doc_path) do |doc_file|
  File.open(selfie_path) do |selfie_file|
    req = Net::HTTP::Post::Multipart.new(
      url.path,
      "document" => UploadIO.new(doc_file, "image/jpeg"),
      "selfie" => UploadIO.new(selfie_file, "video/mp4"),
      "data" => UploadIO.new(StringIO.new(JSON.dump({name: "Joe", age: 20})), "application/json")
    )
    res = Net::HTTP.start(url.host, url.port, use_ssl: true) do |http|
      http.request(req)
    end
  end
end
# kinda works....
# first part with name="document", filename="document.jpg" and content-type=image/jpeg
# second part with name="selfie", filename="selfie.mp4" and content-type=application/octet-stream (Wrong!)
# third part with name="data", content-type=application/json...
# but also filename=local.path, which is wrong!!!

As mentioned earlier, multipart encoding support across our researched HTTP clients is quite… non-standardized. excon, faraday, httprb and net-http do not support it “out-of-the-box”, although in the case of the last 3, there are at least well known “extension gems” adding support for it. In some of these cases, the “parts” need to be passed as instances from a custom class (Faraday::Multipart::FilePart for faraday, HTTP::FormData::File for httprb, Curl::PostField for curb, UploadIO for net-http), which make orchestrating these requests needlessly cumbersome, as the ruby File object abstraction they wrap should give them all they need (the ones which require a wrapper class for “non-file” parts are puzzling). Still, by either accepting or wrapping File objects, it indicates that, at best, they probably stream the multipart request payload in chunks (at worst, they may buffer the payload in a file; I didn’t research them that thoroughly).

The feature that is “broken” in most cases is mime type detection; faraday, httprb and net-http extensions pass the “burden” of identifying it to the caller, which now has to figure out how to do it, and orchestrate the whole thing themselves; in other cases (httparty, curb, httpx), this job is outsourced to a separate module or library, but the devil is in the details here: httparty outsources this concern to mini_mime, a “lighter” version of the mime-types gem, which keeps a registry of “file extension to mime types” relations, and as we’ve seen in the snippet above, isn’t accurate for mp4; I don’t know what internally curb uses, but it’s not accurate either for mp4 (perhaps, like typhoeus it integrates with mime-types?).

httpx works by using one of an array of known ruby gems which detect a file’s mime type by inspecting its magic bytes (the most accurate way to figure it out), and if none is available, it’ll use the file command, which requires a shell call, but uses the same approach to detect mime types, and is widely supported and installed. Besides that, it directly supports “low common denominator” interfaces, such as File, Pathname or Tempfile objects, as “parts” (core and stdlib classes), and therefore requires no custom external class to deal with multipart payloads.

Networking

When deploying HTTP clients in production setups, you’ll often find yourself trying to minimize the impact of HTTP requests in your business operations. For instance, you’ll want to make sure that you’re reusing connections when possible, in order to minimize the impact of TCP slow starts, or that very slow peers won’t hog you beyond what you consider reasonable. In short, we’re looking at support for persistent connections, and timeouts.

Most of the bunch support persistent connections (via HTTP/1.1 keep-alive), to some extent, in most of cases using ruby blocks to enable “persistent” contexts to users, and in some cases enabling persistent connection support via a client flag. Some clients will only allow persistent connections to be set on only one peer per block, whether others will enable persistence for all requests within a block. Some will not only allow connection re-use, they’ll also support sending multiple requests at the same time, by leveraging HTTP/1.1 features such as pipelining, or by using HTTP/2 multiplexing.

# please download hackernews first 2 pages
uris = %w[https://news.ycombinator.com/news https://news.ycombinator.com/news?p=2]

# httpx
# using HTTP/2 multiplexing or HTTP/1.1 pipelining, depends of peer server support
responses = HTTPX.get(*uris)
# will make requests concurrently when targetting different peers
responses = HTTPX.get("https://www.google.com", *uris)
# also supports persistent blocks
HTTPX.wrap do |http|
  # if you need to do sequential requests and want to reuse the connection
  r1 = http.get(uris[0])
  r2 = http.get(uris[1])
end
# explicitly setting the client to persistent by default
# will auto-reconnect when peer server disconnects due to inactivity
# will perform TLS session resumption when possible
http = HTTPX.plugin(:persistent) # also sets retries
responses1 = http.get(*uris) # conns open
responses2 = http.get(*uris) #conns still open
http.close # in order to explicitly close connections

# Excon
# persistent connection set for a single peer
connection = Excon.new("https://news.ycombinator.com", :persistent => true)
# sequential connections
connection.get(path: "/news")
connection.get(path: "/news?page=2")
# or send them at once using HTTP/1.1 pipelining (if peer supports)
connection.requests({path: "/news" }, {path: "/news?page=2"})
connection.reset # don't forget to close them when you don't need them anymore

# faraday by itself does not support persistent connections, so you'll have to pick
# adapters which actually support that
conn = Faraday.new(:url => "https://news.ycombinator.com") do |f|
  # the net-http-persistenta dapter suports it
  f.adapter :net_http_persistent, pool_size: 5
  # the httpx adapter does too
  f.adapter :httpx, persistent: true
end
# and now you can re-use
response = conn.get("/news")
response = conn.get("/news?page=2")
# faraday also supports a weird parallel api, which only the httpx and typhoeus adapters support, AFAIK
conn = Faraday.new(:url => "https://news.ycombinator.com") do |faraday|
  faraday.adapter :httpx
  # or
  faraday.adapter :typhoeus
end
conn.in_parallel do
  response1 = conn.get("/news") # does not block
  response2 = conn.get("/news?page=2") # does not block
end # waits until requests are done
response1.body.to_s #=> the response as a ruby String
response2.body.to_s #=> the response as a ruby String

# HTTPrb
# supports persistent connections on a single peer via block:
HTTP.persistent("https://news.ycombinator.com") do |http|
  r1 = http.get("/news").to_s
  # BIG CAVEAT: because httprb delays consuming the response payload,
  # you have to eager-consume it within the block before the next request
  # is sent (hence the #to_s calls)
  r2 = http.get("/news?page=2").to_s
end
# or initializes the client, and it's up to you when to close
http = HTTP.persistent("https://news.ycombinator.com")
r1 = http.get("/news").to_s # remember to eager load!
r2 = http.get("/news?page=2") # remember to eager load!
http.close # you forgot to eager load! payloads may have been lost!

# httparty does not support persistent connections!

# curb
# supports persistent and parallel requests, also via HTTP/2,
# via the curl multi api ruby shim, which feels like writing C, if you ask me
m = Curl::Multi.new
# add a few easy handles
uris.each do |url|
  responses[url] = ""
  c = Curl::Easy.new(url) do|curl|
    curl.follow_location = true
    curl.on_body{|data| responses[url] << data; data.size }
    curl.on_success {|easy| puts "success, add more easy handles" }
  end
  m.add(c)
end
m.perform

# net-http
# supports persistent connection on a single peer via block
Net::HTTP.start("news.ycombinator.com", 443, use_ssl: true) do |http|
  # sequential requests only
  responses = uris.map do |uri|
    req = Net::HTTP::Get.new(URI(uri).request_uri)
    http.request(req)
  end
end

This example shows httpx versatility in terms of options on how to make persistent, and even concurrent usage of connections, obvious, convenient and flexible. It also starts showing the limitations of the alternatives: the ones that actually support persistent connections, only support it on one peer per connection/session object; while all of them support plain sequential keep-alive requests, only httpx and curb support concurrent requests via HTTP/2 multiplexing and HTTP/1.1 pipelining (excon only supports the latter); while faraday itself does not provide the low level networking features, it does build quite the convoluted API on top of them to support persistent connections and parallel requests; while curb provides access to the low-level features we all expect curl to support, the API to use them feels almost like a verbatim translation from its C API, which is far from “idiomatic ruby”, and does not look like the easiest code to maintain; and oh well, net-http keeps looking verbose and limited (although not as limited as httparty in that regard).

The ability to set timeouts is the other key feature required to mitigate service delivery against service throttling, or network congestion. ruby being so adopted in the startup world, where one sometimes needs to run before it can walk, such matters are usually brushed aside during early product delivery, until production incidents happen. Perhaps given this context, it’s not surprising that it took until 2018 for net-http to introduce a write timeout. But overall, there’s a tendency for ruby HTTP clients to provide timeouts to monitor read/write IO readiness, i.e. “tcp read syscall should not take more than 3 seconds”, instead of a more “cancellation-oriented” approach, “i.e. should receive HTTP response in 3 seconds”. This is a leaky default, as it still exposes clients to slowloris type of situations: if you set 15 seconds read_timeout using net-http, it can still take you minutes to receive a response, if the server sends one byte every 15 seconds. That’s why httpx supports cancellation-type timeouts: write_timeout, read_timeout, and request_timeout options all cover the total time taken to write an HTTP request, receive an HTTP response, or the combination of both, respectively.

Some of the clients will also provide extra timeout options to add similar semantics, but they’re usually incompatible with the defaults, or broken when used alongside other unrelated features.

# please download hackernews main page
uri = "https://news.ycombinator.com/news"

# httpx
# 10 seconds to write the request, 30 seconds to receive the response
# raise `HTTPX::WriteTimeoutError` or `HTTPX::ReadTimeoutError` (both `HTTPX::TimeoutError`)
response = HTTPX.get(uri, timeout: { write_timeout: 10, read_timeout: 30 })
# 3 seconds to fully establish the TLS connection, 40 seconds to send request AND get the response
# raise `HTTPX::ConnectionTimeoutError` or `HTTPX::RequestTimeoutError` (both `HTTPX::TimeoutError`)
response = HTTPX.get(uri, timeout: { connect_timeout: 3, request_timeout: 40 })

# excon
# monitors IO "read" readiness and connection establishment, via `IO.select`
# raises `Excon::Error::Timeout`
response = Excon.get(uri, connect_timeout: 2, read_timeout: 2, write_timeout: 2)

# faraday
# timeout mechanism implemented by adapters
# raises `Faraday::TimeoutError` on error
# requires construction of a connection object
# supports a general timeout for the whole request
conn = Faraday.new("https://news.ycombinator.com", request: { timeout: 5 })
# support granular timeout options
conn = Faraday.new("https://news.ycombinator.com", request: { open_timeout: 5, read_timeout: 2, write_timeout: 2})
response = conn.get("/news")

# but what happens if:
# :timeout is mixed with granular timeouts
conn = Faraday.new("https://news.ycombinator.com", request: { timeout: 2, open_timeout: 5, read_timeout: 2, write_timeout: 2})
# answer: :timeout is ignored.

# timeouts are also set in the adapter
conn = Faraday.new("https://news.ycombinator.com", request: { read_timeout: 2}) do |conn|
  conn.adapter :httpx, timeout: { read_timeout: 0.1 }
end
# `HTTPX::ReadTimeoutError` is raised, i.e. you can set timeouts both for faraday and adapter if the adapter allows it!!

# HTTPrb
# monitors IO "read" readiness, via `IO.wait_readable` and `IO.wait_writable` for operation timeouts
# uses Timeout.timeout for TCP/SSL Socket connect timeout
response = HTTP.timeout(connect: 5, write: 2, read: 10).get(uri)
# single timeout for the whole request/response operation
response = HTTP.timeout(10).get(uri)

# meaning a bit unclear in the block form: it is in fact a timeout for the whole block, which goes a bit
# against its "upper bound of how long a request can take" documentation
HTTP.timeout(5).persistent("https://news.ycombinator.com") do |http|
  r1 = http.get("/news").to_s
  r2 = http.get("/news?page=2").to_s
end

# httparty
# supports the same timeouts as the underlying net-http "engine"
response = HTTParty.get(uri, { open_timeout: 5, read_timeout: 2, write_timeout: 2})
# has a default_timeout, which will be used everywhere in replacement of
# open_timeout, read_timeout and write_timeout, which is a bit confusing.
response = HTTParty.get(uri, { default_timeout: 5 })

# curb
# just uses curl request/response cancellation-based timeout under the hood
# setting a default timeout
Curl::Multi.default_timeout = 5

res = Curl.get(uri) do |http|
  # raises exception if request/response not handled within 5 seconds
  http.timeout = 5
end

# net-http
# monitors IO "read" readiness, via `IO.wait_readable` and `IO.wait_writable`
# uses Timeout.timeout for TCP/SSL Socket connect timeout
uri = URI(uri)
Net::HTTP.start(uri.host, uri.port, open_timeout: 5, read_timeout: 5, write_timeout: 5) do
  # ...
end

To sum up, when in comes to timeouts, there are two libraries, httpx and (in a less granular way) curb, which use a cancellation-oriented mechanism towards a more resilient experience, whereas everything else defaults to readiness-based IO APIs which do not completely protected against slow peers overtaking operations beyond what’s acceptable (which means, you still have to build your own mechanism on top of it). Some of the alternatives try to build a more encompassing timeout on top, but, as in the case of httprb, it results in an inconsistent experience when combined with other features (such as the “persistent” block).

Error handling

In ruby operations, errors can be represented in two ways: a value representing an error, or an exception being raised. HTTP clients may choose one of the two to signal errors in its method calls. For instance, we just talked about timeouts; when a request times out, an HTTP client may raise a “timeout exception” (typhoeus, for example, may use response.code == 0 to signal errors, which is just confusing). Of course, in HTTP requests, not all errors are alike. For instance, 4xx and 5xx response status codes are considered “error responses”, and its up to the client whether to model these as exceptions to be raised, or plain response objects.

Given these options, it’s no wonder that there will be no consensus in how HTTP client handle errors.

uri_ok = "https://httpbin.org/status/200"
uri_404 = "https://httpbin.org/status/404"
uri_timeout = "https://httpbin.org/delay/10"

# httpx
# does not automatically raise exception
http = HTTPX.with(timeout: { request_timeout: 5 })
ok_response, error_response, timeout_response = http.get(uri_ok, uri_404, uri_timeout)
# ok_response is a HTTPX::Response object, with status 200
# error_response is a HTTPX::Response object, with status 404
# timeout_response is a HTTPX::ErrorResponse, wrapping the HTTPX::RequestTimeoutError exception
# .raise_for_status allows for explicit raise

ok_response.raise_for_status #=> 200 response, so does nothing
error_response.raise_for_status #=> raises an HTTPX::HTTPError, which wraps the 404 error response
timeout_response.raise_for_status #=> raises the wrapped exception

# httpx also allows using pattern matching
[ok_response, error_response, timeout_response].each do |response|
  case response
  in { error: error }
    # timeout_response will be here
  in { status: 400... }
    # error_response will be here
  else
    # ok_response will be here
  end
end

# excon
# returns a plain response for HTTP errors
error_response = Excon.get(uri_404)
error_response.status #=> 404
# raises an exception on timeout
Excon.get(uri_timeout, read_timeout: 5) #=> raises Excon::Error::Timeout

# faraday
# same as excon
error_response = Faraday.get(uri_404)
error_response.status #=> 404
conn = Faraday.new(uri_timeout, request: { read_timeout: 5 })
conn.get #=> raises Faraday::TimeoutError

# HTTPrb
# same as excon
http = HTTP.timeout(read: 5)
error_response = http.get(uri_404)
error_response.status #=> 404
http.get(uri_timeout) #=> raises HTTP::TimeoutError

# httparty
# same as excon, with a twist
error_response = HTTParty.get(uri_404, timeout: 5)
error_response.code #=> 404
# does not wrap errors coming from net-http engine
HTTParty.get(uri_timeout, read_timeout: 5) #=> raises Net::ReadTimeout

# curb
Curl::Multi.default_timeout = 5
error_response = Curl.get(uri_404)
error_response.status #=> "404"
Curl.get(uri_timeout) do |http|
  http.timeout = 5
end #=> raises Curl::Err::TimeoutError

# net-http
uri_404 = URI(uri_404)
uri_timeout = URI(uri_timeout)
Net::HTTP.start(uri_404.host, uri_404.port, use_ssl: true) do |http|
  request = Net::HTTP::Get.new(uri_404.request_uri)
  error_response = http.request(request)
  error_response.code #=> "404"
end
Net::HTTP.start(uri_timeout.host, uri_timeout.port, read_timeout: 5, use_ssl: true) do |http|
  request = Net::HTTP::Get.new(uri_timeout.request_uri)
  http.request(request)
end #=> raises Net::ReadTimeout

From the examples above, one can see that the approach of most HTTP clients is remarkably consistent: HTTP errors result in plain responses, whereas networking errors result in errors under the HTTP client namespace. The outlier is httpx, which returns a (different) response object in both cases, that can be “raised on demand”, and where HTTP and networking errors will result in (different) exceptions. This results in (arguably) better semantics and more options for the end user (at the cost of perhaps breaking rubyists expectations, and at least 1 more instruction in order to get the behaviour of other clients).

Extensibility

This is ruby: even if a library was not designed for extensibility, extending it is still possible; monkey-patching is the last resort.

That being said, it’s still good to rely on libraries with extension capabilities. This usually favours composability and code reuse over controlled contracts, and makes it more difficult to have separate patches stepping on each other, when customizing its usage for one’s needs.

Some of our HTTP clients have supported extensions from the “get go”, and even “dogfood” it by implementing some of its internals using the same contracts. Others supported them only much later, and mostly as an “external” interface. And some of them (like net-http…) just don’t.

httpx comes with a plugin system, which was directly inspired by similar systems found in gems from Jeremy Evans, like roda or sequel; and just like the mentioned examples, most features it provides ship as plugins (which means users don’t pay the cost for features they don’t use). For instance, this is how one enables retries:

http = HTTPX.plugin(:retries)
http.get("https://news.ycombinator.com") # will retry up to 3 times by default

Plugins are essentially modules acting as namespaces for other modules which add functionality to core structures of the library:

module MyPlugin
  module ResponseMethods
    # adds the method to the response object
    def get_server_metric
      @headers["x-server-response-time"]
    end
  end

  module ConnectionMethods
    def send(request)
      start_time = Time.now
      request.on(:response) do
        puts "this is how much it took: #{Time.now - start_time}"
      end
    end
  end
end

http = HTTPX.plugin(MyPlugin)
resp = http.get("http://internal-domain-with-metrics/this")
puts resp.get_server_metric

httpx plugins are also composable, and a topic in itself.

Alternatively, httpx also provides event-based hooks one can register on the session object:

started = {}
http = HTTPX.on_request_started do |request|
  started[request] = Time.now
end.on_response_completed do |request, response|
  puts "this is how much it took: #{Time.now - started[request]}"
end.get("http://internal-domain-with-metrics/this")

The difference between both being, event-based hooks are a “high-level” way of intercepting the request/response lifecycle which is easy to learn and use, whereas plugins are more powerful and low-level, but also more involved, and requiring knowledge about httpx internals, to some extent.

excon supports middlewares as extension points, essentially modules defining 2/3 callbacks. It’s relatively simple, and used internally to build features such as following redirects, response decompression, among others. You can define and call it like this:

class MyMiddleware < Excon::Middleware::Base
  # can override request_call, response_call and error_call

  def response_call(data)
    puts data[:headers]["x-server-response-time"]
    @stack.response_call(data)
  end
end

Excon.get("http://internal-domain-with-metrics/this",
  # don't forget to add defaults...
  middlewares: Excon.defaults[:middlewares] + [MyMiddleware]
)

Middlewares are called in order. And that has some drawbacks. For instance, a data structure may be changed by one middleware, that will interfere with the execution of the next one. For instance, there’s a middleware to capture cookies, and another to follow redirect responses; If the second is set before the first, it means that cookies won’t be applied to the redirected request. This type of design is more prone to errors.

As mentioned earlier in the article, faraday uses a similar design, inspired from the rack middleware stack:

class Middleware < Faraday::Middleware
  def on_request(env)
    # do smth with request env
  end

  def on_complete(env)
    puts env[:response_headers]["x-server-response-time"]
  end

  ### or alternatively, you could instead do:

  def call(request_env)
    @app.call(request_env).on_complete do |response_env|
      puts response_env[:response_headers]["x-server-response-time"]
    end
  end
end

conn = Faraday.new do |conn|
  conn.request Middleware # registers #on_request
  conn.response Middleware # registers #on_complete
  # registers #call
  conn.use Middleware
end

Compared to the previous approach, it’s a bit confusing having two ways to accomplish something. And the same drawback applies: order matters. And with that, the inevitable questions follow.

httprb provides a feature called features, which is quite undocumented, albeit used internally to implement de/compression or debug logs. Looking at a few internal examples, the approach is relatively similar to excon’s:

class MyFeature < HTTP::Feature
  def wrap_request(request)
    # do smth
    request # must return
  end

  def wrap_response(response)
    puts response.headers["x-server-response-time"]
    response # must return
  end
end

# optional: register here
HTTP::Options.register_feature(:my_feature, MyFeature)

http = HTTP.use(MyFeature)
http.get(...)

Being so similar to the examples above, the same drawbacks apply here. And you’ll also have to take into account that, because httprb responses are “lazy”, the wrap_response hook can be called before the response is fully on the client side.

httparty does not provide extension mechanisms like the previous ones. Instead, it promotes its class injection API as a way for users to decorate behaviour around API calls (which is the most popular way of using it):

class Google
  include HTTParty
  format :html
  base_uri 'https://www.google.com'

  def q(options = {})
    q_query = URI.www_encode_form(options)
    self.class.get("/search?#{q_query}")
  end

  # intercepting all requests, invoke the monkeypatch:
  class << self
    def perform_request_with_log(*args)
      puts "this: #{args}"
      perform_request_without_log(*args)
    end
    alias_method :perform_request_without_log, :perform_request
    alias_method :perform_request, :perform_request_with_log
  end
end

As the example shows, there are limits to the extensions this approach enables: decorating behaviour is easy, but introspecting the client isn’t a first-class abstraction, and you’ll soon be adding a potentially unhealthy dose of monkey-patching to fill in the gaps.

curb does not support anything of the kind. Either your needs are fulfilled by the wide array of curl features it integrates with, or you’ll have a harder time beating it into shape.

And as for net-http… let’s just say that there are several net-http-$feature gems around, which, at their best, inject APIs into core classes which work in isolation but rarely build well on top of each other, and at their worst, monkey-patch their way in (several tracing / logging / mock libraries do this).

To sum up, and discarding the ones which are not built for extension, most libraries allow extension based on a standard around chained hooks for “sending the request” and “getting a response” (the interpretation of which is library-dependent), and support a more or less friendlier (depending of which example, and personal opinion) API for registering extensions. In most cases, features are provided via these APIs. These extensions cover most of high-level use-cases, but start getting rather limiting for more advanced cases (such as getting information about DNS / socket-handshake / byte-level progress). And that’s where httpx flexible approach to extensions works best, by providing a higher- and low-level way of doing it, and on the latter, by building on a standard which has proven itself with some of the most respected gems within the ruby community.

Performance

The first thing one can say about performance benchmarks, is that you cannot fully trust them. Some of the numbers you’ll see will always be context- or environment-specific: does the gem use a C extension optimized for x86, but that’s not the CPU arch from the machine the benchmark runs on? Is the network IPv4 optimized, thereby penalizing traffic going via IPv6? Are payloads exactly the same?

There are ways to ensure some level of confidence though. First, you must have access to the benchmark code, in order to gain context; you should also have access to the run logs and history; also, benchmarks must run regularly.

Because I didn’t find an acceptable public benchmark which fits these requirements, I went ahead and rolled my own in order to measure the performance difference between some of ruby HTTP clients. While you’re free to inspect it, the gist of it is essentially a pair of containers running in a Gitlab CI pipeline, one with a test HTTP server, and another running the benchmark against it. It runs monthly, so it’s very up-to-date. Local area network ensures negligible network interference in the measurements. There’s a warmup phase, and garbage collection is turned off, ensuring no potential “stop-the-world” interference as well. The benchmark uses the stdlib benchmark gem to measure “real time”, and composes of a series of use-cases (alternatives may not support all of them, hence why not all of them are displayed in all graphs).

While there could be more use-cases in the benchmarks (feel free to suggest by creating A Merge Request), this shows us that the performance gap between alternatives is not huge, which makes sense: even for such contained scenarios, most time is spent waiting on the network. As httpx maintainer, it’s definitely reassuring seeing it keeping up with the “top of the pack”, particularly when you consider that it is pure ruby (both the HTTP/1 and HTTP/2 parsers are written in ruby), and some of the alternatives claim much better performance due to using C-optimized code, ultimately not delivering (httprb uses the nodeJS HTTP parser via FFI, and used to do it via a C extension; curb and typhoeus use libcurl under the hood as well).

Honorable mention to net-http, which actually shows quite good numbers, which may mitigate a bit some of its UX shortcomings (caveat though: the “pipelined” and “persistent” benchmarks were performed using net-http_pipeline and net-http_persistent gems respectively).

Packaging

With the advent of containers as the ultimate deployment target, the art of setting up VMs has slowly been lost, and shifted into writing recipes, of which dockerfiles are the most popular today. That’s not to say everyone deploys to containers though: there’s also serverless platforms. And “on-premise” never went anywhere either (it’s just under-practised). And what about ruby-based scripting tools (like Homebrew) for your laptop? Don’t forget Windows either: that <2% of the community will chase you in your dreams if they are faced with difficulties. Last resort, you can “write it in JRuby once and run it everywhere”. Bottom line, ruby is everywhere, and when building gems, you best take all this diversity into account, lest you’ll be reminded periodically by someone having troubles with the things you build.

System

So, first thing, how hard it is to install any of our candidates? The options range from “relatively hard”, to “easy”, to “zilch”. Let’s start by the end. net-http is already there. Done. Now that we got that out of the way, we can go to the easy part of the equation: pure ruby gems. Which ones are they? As already mentioned, httpx is pure ruby; the only thing you need to do is use the gem command, or bundler, like you do with any of the other alternatives. excon and httparty are no different: they’re also pure ruby. On the moderate side, you’ll find httprb; it requires the compilation of the llhttp C extension or FFI binding (for the aforementioned nodeJS parser). This means that, in order to install it, you’ll require the whole “C compilation toolchain” including CMake, gcc, and the like. And that includes the deployment environment, as all of them compile-on-install (take that into account in your slim/alpine images). And last of this bunch, you have curb, which not only carries the same requirement of compiling a C extension on install, it also requires a (compatible) installation of libcurl (and bear in mind what was discussed about libcurl-based libs when you need something specific). While not nokogiri-bad in terms of compilation times, its still setup overhead (credit to nokogiri though for adopting pre-compiled binaries, something which none of the extension-dependent libraries researched here does). I’ll omit faraday from the conversation here, as the bulk of the cost lies in the chosen adapter.

Rubygems

# dependency list
httpx
  http-2-next
excon
faraday
  faraday-net_http
  ruby2_keywords
http
  addressable
    public_suffix
  http-cookie
    domain_name
      unf # C Extensions
  http-form_data
  llhttp-ffi # C Extensions of FFI
    ffi-compiler
httparty
  mini_mime
  multi_xml
curb # C extensions
net-http

Dependency-wise, the mileage also varies. As mentioned, net-http is all standard library built. excon also ships with no direct dependencies, which is impressive all things considered. httpx ships with one (the http-2-next parser, which is at least maintained by the same person). httparty ships with 2 (why is multi_xml even required? Not sure). faraday has at least 2 (that is, if you do not switch from the default adapter for net-http); httprb has 4 direct dependencies, 8 total. curb has no direct dependencies either (ruby dependencies that is; it does require libcurl).

Is that all necessary? Perhaps, it depends. But I don’t see the point in httprb carrying so much baggage by default: besides the aforementioned parser complication, it also declares http-form_data (same-team maintained, for multipart support), http-cookie, and addressable, aka things that could be optional (ruby already ships with a URI parser), or not loaded by default (I doubt that the majority of its users have used the cookies feature, although everyone seems to be paying the cost). The same could be said of httparty requiring multi_xml (who’s still using XML?). For instance, consider httpx and excon’s approach, where certain features do require the installation of a separate gem, but you only pay the cost if you enable the feature (excon supports addressable as an alternative URI parser, and just to name an example for httpx, the grpc plugin requires the protobuf gem).

Nevertheless, if packaging is the most important variable to consider, you can’t really beat “shipped with ruby”, i.e. net-http.

Features

The feature set that can be built on top of HTTP client is so immense, that it’s impossible to cover in a single blog post (I’d need a book for that, or several). Fortunately, nahi, the former maintainer of httpclient, made my job easier by having built a “common feature matrix” for a presentation he did many years ago in a ruby conference, that I’ll partially use here to highlight the intersection of features across the alternatives covered:

One important thing to take into account is, just because the ✅ is there, it does not necessarily mean that all alternatives implement a feature the same way. For instance, curb support of GSSAPI requires a curl build compiled with gssapi; httprb proxy support does not cover the http_proxy/https_proxy/no_proxy environment variables (which will always come out as surprising if you’re a sysadmin); all of the alternatives, except httpx and curb (via libcurl), implement poor, or simply do not implement, mime type detection of file parts (as already mentioned in the multipart-related section); and as I’ve exposed earlier, the question about streaming response support is not “if”, but “how”.

Still it does show that, when it comes to having the obvious features expected from an HTTP client, the set of alternatives do cover a sufficient chunk not to be considered useless. The only option ticking all the boxes here is httpx, but then again I selected the boxes, so I’d be interested to know whether you think this feature matrix is fair.

Extensions

An HTTP client is not an island. In most cases, it’s a really small from a large program. This program will have certain expectations from some of its dependencies. In the context of an http client, it’ll probably not want to send real requests in test mode. Some metrics / tracing support is usually a must. Can it easily log request information? The answer to these question may make or break the chance of a library being adopted in a given project.

While there’s plenty of tooling available, the ruby community has been settling on a group of dependencies which provide these type of extensions on top of well-known libraries. For instance, there’s webmock or vcr for mocking HTTP requests. Tracing is usually vendor-specific (the datadog SDK, for instance, ships API and shims for well-known libraries in its SDKs), although things are slowly getting a bit more standardized thanks to the Open Telemetry toolchain. And there are several tools for logging HTTP information (of which httplog is one of).

How these libraries choose which HTTP clients to support is up to how standard, or how popular they are, how many users rely on them (and for how long), or how much community “weight” these alternatives command. It’s expected, for instance, that net-http is supported by all of the above (no matter how anti “built-for-extension” it is).

This list is not exhaustive, but it does show where more recent alternatives like httpx struggle: joining the group of “well-known” libraries is hard work. Specially when the library was created post-2014, and missed the heyday of when every exciting application in the internet was being built in ruby, and every option was getting a slice of the pie.

What sets httpx apart

So far, the focus of this analysis was to provide perspective, and a wider overview of how well the current well-maintained ruby HTTP clients cover a reasonable set of MUST HAVE and NICE TO HAVE features, enough at least to make this reading enjoyable.

Still, there are things that only httpx does, which you’ll never think about until things don’t work and you need them.

For instance, did you know that httpx is the only pure ruby (excluding curl-based tools here) HTTP client (the only networking library, I think?) that does connection establishment using Happy Eyeballs 2? It will hardly be noticeable to you if production is about “always on IPv4” server-side deployments, or perhaps you don’t care as long as the tool “just works”, no matter whether the tool you’re using gives preference to IPv4 (this is what excon does by the way), until it doesn’t, and then you blame the server. It is certainly a SHOULD HAVE when doing client-side programs on multi-homed networks where connectivity may not be properly set. Such as games, or running bundle install as well (in fact, it’s so important that bundler has its own monkey-patch around TCP connection establishment which half-implements Happy Eyeballs).

It also supports DNS resolution via DoH, a feature so hard to backport to existing networking tools in general, that there are products (such as Cloudflare Zero Trust) which will intercept local UDP/TCP-based DNS traffic through a program installed in your machine, and “translate” them to DoH-based DNS traffic. (curl supports DoH, but curb does not seem to interface with it).

The ability to perform concurrent requests, very useful for scraping scripts for example, is also not to be found often (typhoeus provides something similar, and via a less user-friendly API, as well as curb via Curl::Multi).

It ships a plugin to perform GRPC requests, in case you want to forego the heavy dependency that is the grpc gem (over 100Mb pre-compiled, it can take you Gbs of space if you have to compile it) or are on JRuby. And another supporting WebDAV.

Even something as simple as passing the IP address to use for a given request and which hostname to set in the SNI extension, or in the “Host” header, is practically impossible with any other library, and dead easy with httpx.

Bottom line, while most HTTP clients cover the 70% just fine, and 85% with a few adjustments, httpx works really hard in making the 99% of use-cases accessible.

(Speaking about coverage, httpx publishes how much of the code is covered in CI. Good luck finding numbers for any of the others.)

Conclusion

The main takeaway from this “state of the ruby HTTP clients” is that, no matter whether the “HTTP fringe features” aren’t of your interest, and you’re just interested in covering the 80%, choose a library which is still maintained. If you have a favourite library that wasn’t taken into account in this article, that’s probably why it isn’t here.

Beyond that, the choice will probably be based on prior experience and risk apettite for “trying new toys”, and the requirements you favour the most, which I (hopefully) have outlined and made a good analyis about. Whether it’s API UX, adoption rate, performance or anything else, any of these options will give you some level of acceptable quality.

And when in doubt, use httpx. As it was shown, it stacks well against the competition in any available metric, and is working hard to curb the adoption gap. So help me change that :)

In order to talk about its value, and defend some of choices made, some background is required.

Context

For the problem of offloading processing resulting from a given business transaction, the ruby community defaults to using background jobs. Most of us have used sidekiq at one point or another in the last 10 years, while the elders among us may also be familiar with resque or delayed_job, and here’s an honourable mention to shoryuken, as integration with SQS is something that every other framework lacks.

These frameworks have mostly commoditized the “how do I defer this business flow after another one completes, while not making the client wait for it to finish” problem for us all. They achieve this by providing some sort of simple DSL to delegate the execution of a routine, by serializing and writing the required state into some broker, only to have another process read and and execute it:

class Foo
  def activate
    # heavy duty
  end
end

# then

foo.async.activate

# service object style, most common nowadays
class ActivateJob < SpecialFrameworkSubclass
  def perform(user)
    user.activate
  end
end

ActivateJob.perform_async(user)

The solution is fairly similar for all of them (they mostly “stole” features from each other), so they differentiate themselves on other aspects, such as performance of the execution model (process/thread based), choice of broker (database, redis, SQS, rabbitMQ…), or advanced features (plugins, retry configuration, on complete callbacks, etc…).

One problem that is common to all of them, is how one needs to be aware of the storage and execution characteristics of the deferred routines, in order not to be surprised by some unexpected behavior. Argument serialization is one: while rails provides a solution for serializing model instances for activejob, most complex objects can’t be serialized, so documentation and FAQ sections will contain caveat warnings and recommendations about which types of objects can be used. Primitive types tend to be supported, however simple objects such as symbols aren’t supported everywhere (as an example, sidekiq only accepts primitives which can be serialized into json).

But the main problem that gets everyone at some point in their careers, is when the state being stored in the database before deferring a function, is not available once it gets executed. In fact, one of sidekiq wiki FAQ oldest entries contains the following:

Why am I seeing a lot of “Can’t find ModelName with ID=12345” errors with Sidekiq?

Your client is creating the Model instance within a transaction and pushing a job to Sidekiq. Sidekiq is trying to execute your job before the transaction has actually committed. Use Rails’s after_commit :on => :create hook or move the job creation outside of the transaction block.

Database transactions

Most rubyists building web services are using an ACID-compliant database, usually over their favorite ORM; mine is sequel, but the majority probably knows activerecord the most. For the context of this post, the most important property of the ACID family is Atomicity, which ensures that all operations from a group all completes, or not at all. This includes errors in the operations themselves, but also “out of our control” events such as power outages or computer crashes. This is achieved by wrapping this group of operations (or SQL statements) in a database transaction:

BEGIN; -- transaction starts here
-- UPDATE / INSERT / DELETE statements here
COMMIT; -- or ROLLBACK; transaction ends here

A transaction is a first-class citizen of your business logic, as it has to be explicitly started and finished. Ruby ORMs usually expose block-based DSLs to manage transactions:

# using sequel
DB.transaction do # BEGIN
  DB[:foo].insert(bar: 1) # INSERT
end # COMMIT; ROLLBACK if an error is raise inside the block

# using activerecord
ActiveRecord::Base.transaction do
  Foo.create(bar: 1)
end

Transactions are also managed via other features, such as model callbacks, and one has to be aware of it when using deferred routines:

class User
  after_save :activate

  def activate
    ActivateJob.perform_async(self)
    # TRANSACTION DID NOT COMMIT YET HERE!
  end
end

(The above is fine if you’re using delayed_job, as the broker is your database; not as fine if you’re using sidekiq or shoryuken though.)

And then there are some other 3rd party gems which hide these calls under layers of DSL (looking at you, state machine gems). Given all the options available, and how convenient these deferred DSLs seem, it’s no wonder that, when using them, one is either oblivious, or lost, on whether a transaction is open. Specially if this feature needs to be shipped by next Friday.

And if you deferred a function before a transaction is committed, and you need the state you’re writing into the database, and that transaction either fails, or takes too long to commit, you’ll see yourselves staring at some similar FAQ like the one I shared above.

# service object style, most common nowadays
class ActivateJob < SpecialFrameworkSubclass
  def perform(user)
    user.activate #=> Exception raised, RecordNotFound
  end
end

But let’s say you lived to fight another day, you learned your lesson, untangled that 3rd party code you don’t own, and now you’re sure that the deferred function call happens after the transaction successfully commits. Problem solved, right?

Storage/Broker consistency

So you’re committing a database transaction to fully store the state of your business transaction, and then you’re invoking the “defer function” routine, which will push the serialized state into your broker:

foo = ActiveRecord::Base.transaction do
  Foo.create(bar: 1)
end
ActivateJob.perform_async(foo)

What if there’s an outage between the transaction committing and the job being enqueued? It’s terrible, given that your “jobs to be done” will probably be silently lost.

Such a conundrum is only possible to avoid if the database and the broker are protected by the same transaction guarantees, i.e. if the broker is the same database where your business resources are stored. From the background job alternatives mentioned above, only delayed_job fits the bill, given that the queue is a database table. Everything else (yes, including sidekiq) is vulnerable to this problem.

This has been discussed at length in this 2017 blog post.

Transactional outbox pattern

While the description of the problem above mostly focuses on the background jobs ruby frameworks use-case, the same type of problem happens if your business transaction requires to perform some rpc call (HTTP, GRPC) to a separate system, which happens a lot if you’re using microservices.

A solution for this general problem was formalized in the transaction outbox pattern. The gist of it is, business transactions store their “events to be emitted” in a separate database table (typically called “outbox”) within the same database transaction. This in itself ensures that the events associated with the business resources will always be stored if the resources are stored successfully. Then there is a separate worker (thread in same process, separate process…) reading entries from the “outbox” table, and doing the actual publishing of the event (or enqueuing of the job) before deleting the entry.

tobox

So what is tobox again? In a nutshell, it’s a “transactional outbox” framework.

I built it because I needed its properties, and I couldn’t find a transactional outbox implementation for any programming language, just blog posts on how to hypothetically do your own.

The DSL is declarative and “event-based”, which means that one can register handlers bound to specific events:

# this is the config DSL
# tobox.rb
on("order_processed") do |event|
  Payment::Start.call(event)
end
on("order_cancelled") do |event|
  CustomerSupport::Notify.call(event)
end

# if handling multiple events
on("order_processed", "order_cancelled") do |event|
  Logs::Order.call(event)
end

### app/handlers/payment_start.rb
module Payment::Start
  module_function

  def call(event)
    data = event[:after]
    # do something with the event data hash, perhaps enqueue it as a background job?
  end
end

An entry point script is also provided, to start a separate process to consume events from the outbox table:

> tobox -r ./app.rb —config tobox.rb

# if you’re using rails
> tobox -r ./config/environment.rb —config tobox-dsl.rb

In the process, it handles the complexity of the “plumbing” involved in building a transactional outbox consumer, using a set of conventions and tricks:

Thread and Fiber-based worker pools

tobox, by default, uses threads to handle many events at the same time in the same process, just like sidekiq’s. You can tweak the number of threads in the config:

# tobox.rb
concurrency 25

You can, however, switch to using fibers instead of threads, if your event handling is very IO-bound (if you’re just relaying the events to SNS, it is):

# tobox.rb
worker :fiber # :thread by default
concurrency 100 # max 100 fibers running at the same time

(This requires using the fiber_scheduler gem).

SKIP LOCKED

When enabling multiple consumers for a given queue, one has to have the guarantee that a given event won’t be processed more than once by separate workers at the same time. One way to achieve that using the database is by locking the row where the event is stored, and delete it after it has been handled. However, if two workers try to lock the same row, one of them will remain idle, instead of picking up the next available event.

While the database row-level locking model wasn’t built to support the queue use-case, some recent features were added to some of the most popular database engines to accommodate it. One of these features is the SKIP LOCKED clause, a non-standard SQL clauuse which can be used with SELECT …. FOR UPDATE, and will result in already locked rows being ignored (“skipped”) by the SELECT statement.

This feature is core to how tobox works, which is why it only supports databases including the SKIP LOCKED feature.

(Supporting this many databases is only possible thanks to the sequel gem, by the way).

Plugin DSL

tobox ships with a simple plugin system which supports intercepting handlers before and after they’re handled (or error out). It’s the foundation of a few plugins which already ship with the gem:

# tobox.rb
plugin(:zeitwerk)
plugin(:datadog)
plugin(:sentry)

multilang support

Until now, I didn’t show how to insert events into the queue. That’s because, for now, tobox does not provide any DSL for it. The reason is, working with database objects is probably already such a big part of your day-to-day work, that moving that concern into a 3rd party gem may end up having more drawbacks than benefits. Moreover, perhaps this way it’s clear that you can use a transactional outbox even if your application is not made in ruby.

For instance, here are several examples of how to write an event into the outbox:

ruby

# using sequel dataset API
DB[:outbox].insert(type: "order_created", data_after: to_json(order))
# or an ActiveRecord model
OutboxEvent.create(type: "order_created", data_after: to_json(order))

python

# SQLAlchemy
event = OutboxEvent(type="order_created", data_after=to_json(order))
db.session.add(event)

elixir

OutboxRepo.insert %OutboxEvent{
  type: "order_created",
  data_after=to_json(order)
} do …

database triggers

There are also other ways to “go implicit”, if that fits your use-case. One way you can do it is by using database triggers, such as this postgresql example:

CREATE OR REPLACE FUNCTION order_created_outbox_event()
  RETURNS TRIGGER
  LANGUAGE PLPGSQL
  AS
$$
BEGIN
	INSERT INTO outbox(event_type, data_after)
		 VALUES('order_created', row_to_json(NEW.*));
	RETURN NEW;
END;
$$

CREATE TRIGGER order_created_outbox_event
  AFTER INSERT
  ON orders
  FOR EACH ROW
  EXECUTE PROCEDURE order_created_outbox_event();

Conclusion

tobox is a lightweight tool that you can use to ensure robustness and guarantee at-least-once semantics in your business workflows with little to no performance impact. It’s therefore not a silver bullet: it trades off some E2E latency (the extra step of putting and taking the event from the database) to achieve that robustness.

While it may “quack” like a background job framework, it is not designed to be one. Its features (do check the README) are more focused on the transactional outbox use-case, so if you require background job features, you should use tobox alongside such a framework.

The declarative DSL is a departure from the current “standard” for background jobs, IMO leaner, and eliminates the antipattern of creating a job class, only to call some other service object in the #perform method.

Some edges are still rough, and some features are still missing (no web dashboard yet, for example). But it already does “one thing well”, so that’s the 80% right there.

How you get to “quickly” is usually a combination of a few factors: team skill, project scope, and a healthy dose of pragmatism. Shipping quickly means refusing perfect solutions; it’s focusing about “0 to 1” before you consider “1 to 100”; it means “good enough” first. And you have to be comfortable absorbing the right amount of “tech debt”. Why the right amount? Because there’s a very big chance that the thing you’re building quickly now is going to be the thing you’ll be maintaining in 1 year (no matter how much product calls it “throw-away POC” or your engineering manager tells you will be able to rewrite it in kubernetes when it “wins”). Given enough experience, you learn how to compartimentalize debt in a way that it doesn’t “leak” too much into other sections of the codebase. You learn how to leverage these limitations, and reuse it to solve other problems. You learn how to do some forecasting, and ask yourself some questions such as “can the way this feature was architected survive the next 3 months, or the next 3 years? How long can this reasonably hold in production until something entirely different needs to be considered?”.

After the initial core features, the next thing we knew was going to be very valuable for our customer base, was providing customer-facing analytics dashboards. The product team wasn’t sure exactly how that would look like, and constantly debated how “can we learn from our users quickly”. A team of engineers was assigned with the task of scoping the technical aspects of the project, and they went about designing a full-fledged “analytics pipeline”, with some of the most “state-of-the-art” technologies, such as Spark, flink or openwhisk, to solve not only the immediate product’s analytics needs, but also aggregating analytics data for all the other products and services from the company. With that, scope only grew, and needless to say, none of this was going to be ready in 2 weeks. Or 2 months. No “quickly”.

The plan of building an analytics pipeline could have other theoretical compound benefits for the company, but it’d take more than 6 months to ship. That’s quite risky, considering company priorities change all the time, and in times of economic uncertainty (hello 2023!), long-running costly projects are quickly thrown into the doghouse when they do not generate immediate revenue, and kept there until the good times roll again. So it could take years, in reality.

I proposed another approach for a shortcut: we could provide a couple of API endpoints for querying data aggregations, around which some dashboards could be built; the analytics team could then focus on building those dashboards immediately, while designing the long term analytics pipeline for when this proposed solution would not scale anymore. When I mentioned this could be shipped in about 2 weeks, everyone was sold on the idea.

So the question was, how could we deliver something in 2 weeks, that would not simply fall off the rails in 2 months, and could potentially still be operational 2 years from now, at a reasonable scale, if need be?

Data

Without going too much into detail, the product revolves around allowing customers defining user journeys, and costumer users running them. This is what an over-simplification of the database would look like:

CREATE TABLE definitions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    client_id UUID,
    # ....
);

CREATE TABLE journeys (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    client_id UUID,
    client_user_id UUID,
    definition_id UUID,
    status varchar(255), -- "pass", "fail", "review"
    error_code varchar(255), -- "network_error", "file_error" ...
    created_at TIMESTAMP WITHOUT TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITHOUT TIME ZONE NOT NULL DEFAULT NOW(),
    CONSTRAINT fk_definition FOREIGN KEY(definition_id) REFERENCES definitions(id)
);

The initial requirements revolved around fetching, for example, how many journeys were in-progress, percentage of completed/cancelled, average, or max or median duration, all this for time intervals. The data could be fully aggregated or time-series split (initially, by day).

Planning

Querying product data tables directly was not an option; the required queries would be very complex, and require all kinds of indexes, which would affect write performance; and even if that would be done, certain client accounts volume could render all those optimizations uselss, and even moreso if the requested time intervals stretched more into the past.

We decided it’d be better to pre-aggregate data in a separate table. It’d be aggregated by day, as this would be the requested time range minimal unit (“today”, “last 15 days”…). And we are using Postgresql, so using something like table partitioning as time passes and data grows, gave us enough confidence that this solution could scale well, in case something better never came along.

So there were two things to be done: aggregate data, and serve it via the API.

Aggregating

Aggregation was to be done on the fly. While bulk-aggregating it in cron jobs was certainly possible, we would like to serve data as fresh as possible, as the default time interval would be “current day”. UTC was to be used everywhere.

This was to be done using two pieces: one of them would be tobox, a transactional outbox framework I was developing at the time, and which I was already considering integrating to solve other issues in our architecture (which will deserve its own post), and sequel, the best ORM/database toolkit you can find in any stack. Period.

The analytics table was created as per the following sequel migration:

Sequel.migration do
  up do
    create_table?(:journeys_analytics_daily) do
      column :client_id, :uuid, nullable: false
      column :journey_id, :uuid, nullable: false
      column :definition_id, :uuid, nullable: false
      column :day, :date, nullable: false
      column :started_count, :integer, nullable: false, default: 0
      column :completed_count, :integer, nullable: false, default: 0
      column :cancelled_count, :integer, nullable: false, default: 0
      column :status_count, :jsonb, nullable: false, default: Sequel.pg_jsonb_wrap({})
      column :error_code_count, :jsonb, nullable: false, default: Sequel.pg_jsonb_wrap({})
      column :min_duration, :integer, nullable: true, default: 0
      column :max_duration, :integer, nullable: true, default: 0
      column :avg_duration, :integer, nullable: true, default: 0
      column :median_duration, :integer, nullable: true, default: 0
      primary_key %i[client_id journey_id definition_id day]
    end
    # ...

Why were status and error_code data aggregated into JSONB columns, and not have each possible value be mapped into its own column? The reason was that new statuses and error codes would eventually be defined, which would then cause new columns to be added/removed, therefore requiring schema changes. JSONB could satisfy the same requirements without the need for it, with a bit of postgres “JSON function-fu”.

In tobox, two handlers were subscribed to the “journey started” and “journey finished” outbox events:

# tobox config fifle
on("journey_started") do |event|
  Aggregation::JourneyStartedEvent.call(event)
end
on("journey_finished") do |event|
  Aggregation::JourneyFinishedEvent.call(event)
end

# Event structure:
# {
#   "event" => "journey_finished" # or "journey_started"
#   "event_id" => "e30aedaa-8eba-462c-b2c8-086b5c6ee824",
#   "emitted_at" => "2022-12-24T00:00:00Z",
#   "client_id" => "4cffca63-7f1f-48b6-a8bc-5b39b515d854",
#   "journey_id" =>"87536aa8-de39-4428-9567-5824287111ff",
#   "definition_id" => "9ee3d3b9-2930-4212-bbdb-ef4e5852bde4",
#   "status" => "pass" # or "fail", "review", "drop", "delay"...
#   "error_code" => nil # or "network_error", "file_error", etc...
#   "created_at" => "2022-12-23T00:00:00Z",
#   "updated_at" => "2022-12-24T00:00:00Z",
# }

The handlers would then use the data sent in the event to craft an SQL query which would atomically increment counter columns and duration calculations.

Events could be processed “out of order”, and a “started” event could receive the respective “finished” event the following day. UPSERTs could help manage that.

Atomic counter increments for integer columns were a no brainer. But how could this work for JSONB columns? The solution is a combination of Postgresql JSONB functions. Let’s look at the sequel code first:

TABLE_NAME = :journeys_analytics_daily
# Aggregation::JourneyStartedEvent
def call(event)
  event_time = Time.parse(event["emitted_at"])

  DB[TABLE_NAME].insert_conflict(
    constraint: :journeys_analytics_daily_pkey,
    update: { started_count: Sequel[TABLE_NAME][:started_count] + 1}
  ).insert(
    client_id: data["client_id"],
    journey_id: data["journey_id"],
    definition_id: data["definition_id"],
    day: event_time.strftime("%Y-%m-%d"),
    started_count: 1,
  )
end


# Aggregation::JourneyCompletedEvent
def call(event)
  event_time = Time.parse(event["emitted_at"])

  insert_args = {
    client_id: data["client_id"],
    journey_id: data["journey_id"],
    definition_id: data["definition_id"],
    day: event_time.strftime("%Y-%m-%d"),
  }

  update_args = {}

  if error_code = event["error_code"]
    # journeys with errors aren't accounted for in duration metrics
    insert_args[:cancelled_count] = 1
    update_args[:cancelled_count] = Sequel[TABLE_NAME][:cancelled_count] + 1

    error_code_column = Sequel[:error_code_count].pg_jsonb
    insert_args[:error_code_count] = Sequel.pg_json_wrap({error_code => 1})
    update_args[:error_code_count] = error_code_column.set(
      "{#{error_code}})",
      (Sequel.function(:coalesce, error_code_column[error_code], "0").cast_numeric + 1).cast(:text).cast(:jsonb),
      true
    )
  else
    insert_args[:completed_count] = 1
    update_args[:completed_count] = Sequel[TABLE_NAME][:completed_count] + 1

    status = event["status"]
    status_column = Sequel[:status_count].pg_jsonb
    insert_args[:status_count] = Sequel.pg_json_wrap({status => 1})
    update_args[:status_count] = status_column.set(
      "{#{status}})",
      (Sequel.function(:coalesce, status_column[status], "0").cast_numeric + 1).cast(:text).cast(:jsonb),
      true
    )

    # duration
    duration = (Time.parse(event["updated_at"]) - Time.parse(event["created_at"])).to_i
    insert_args[:min_duration] = insert_args[:max_duration] = insert_args[:avg_duration] = insert_args[:median_duration] = duration

    update_args[:min_duration] = Sequel.function(:least, Sequel[TABLE_NAME][:min_duration],  Sequel[:excluded][:min_duration])
    update_args[:max_duration] = Sequel.function(:greatest, Sequel[TABLE_NAME][:max_duration],  Sequel[:excluded][:max_duration])
    update_args[:avg_duration] = (
      ( (Sequel[TABLE_NAME][:avg_duration] * Sequel[TABLE_NAME][:completed_count]) + Sequel[:excluded][:avg_duration])
      / (Sequel[TABLE_NAME][:completed_count] + 1)
    )

    # median calculation is a bit more involved and requires a query to product data
    update_args[:median_duration] = DB[:journeys].where(
      :client_id => data["client_id"],
      :journey_id => data["journey_id"],
      :definition_id => data["definition_id"],
      :error_code => nil
    ).where(Sequel.cast(Sequel[:journeys][:updated_at], :date) => event_time.strftime("%Y-%m-%d"))
      .select(
        Sequel.function(:coalesce,
          Sequel.function(:percentile_cont, 0.5)
            .within_group(Sequel.extract(:epoch, Sequel[:journeys][:updated_at] - Sequel[:journeys][:created_at])),
          0)
      )
  end

  DB[TABLE_NAME].insert_conflict(
    constraint: :journeys_analytics_daily_pkey,
    update: update_args
  ).insert(insert_args)
end