Blog

HTTPX multipart plugin, lessons in the wild

Some say that open source maintainers should “dogfood” their own software, in order to “feel the pain” of its users, and apply their lessons learned “from the trenches” in order to improve it. Given that no one else is better positioned to make improvements big and small, it’s hard to dispute against this.

But in the real world, OS maintainers work in companies where they not always get to decide about software tools, processes or coding style. Sometimes it’s hard to justify using your “part-time” project when there’s already something else in-place which “does the job”.

Also, the feature set might just be too wide for one maintainers to exercise it all. To name an example, cURL has over 9000 command-line options and supports 42 protocols (numbers completely made up, but probably not too far off from the real ones), I honestly do not believe bagder has had a chance to use them all in production.

httpx is an HTTP client. In ruby, there are over 9000 HTTP client libraries (numbers completely made up, but probably not too far off from the real ones), and 10 more will be released by the time you finish reading this post. An HTTP client is a well-known commodity, and a pretty large subset of HTTP is implemented widely. So it’s pretty hard, however good and useful you client is, to convince your co-workers to ditch their tool of choice (even something as bad as httparty), because “it works”, “I’ve used it for years”, “it has over 9000 stars on Github” (numbers completely made up, but probably not too far off from the real ones), are good arguments when you’re just downloading some JSON from a server.

(The popularity argument is pretty prevalent in the ruby community, which probably means that no one bothers to read the source, and I’m just not that popular. Gotta probably work on that.)

So you shrug, you move the wheel forward and patiently wait for your turn, and roll your eyes while you write some code using rest-client, and life goes on. Sorry httpx users, not there yet!

And then, an opportunity hits.

Media files

Where I work, we handle a lot of media files, such as photos and videos. Users upload photos and videos from their devices to our servers, where we pass them around across a number of services in order to verify the user’s identity. Pretty cool stuff. Downloading, uploading, transferring, happens a lot.

Some tasks often involve uploading data in bulk. And although uploading photos is not a big deal, uploading videos is, as some information about what happens in the video must also be transferred, as extra metadata, in a widely known encoding format.

And for that, the multipart/form-data media type is used.

multipart/form-data

RFC7578 describes the multipart/form-data media type. In layman’s terms, this media type is what powers file upload buttons from HTML forms. For instance, in your typical login form with an email and a password:

<form method="post" action="/login">
  <input type="text" name="email" placeholder="Your email..." />
  <input type="password" name="password" placeholder="Your password..." />
  <input type="submit" value="Login" />
</form>

The browser will submit a POST request to the server, where the form data is sent to the server in an encoding called application/x-www-form-urlencoded. This encoding is one giant string, where (huge simplication disclaimer) key/value pairs are encoded with an "=" between them, and all pair are then concatenated with the "&" as the character separator:

POST /login HTTP/1.1
....
Content-Type: application/x-www-form-urlencoded
....

email=foo@bar.com&password=password

This works great for text input, but what if you want to support binary data, such as in the scenario of, uh, uploading a profile picture? That’s where multipart/form-data comes to the rescue:

<form method="post" enctype=multipart/form-data action="/upload-picture">
  <input type="file" name="file" />
  <input type=text" name="description" placeholder="Describe your picture..." />
  <input type="submit" name="submit" value="Upload" />
</form>

The browser will send the request while encoding the data a bit differently:

POST /upload-picture HTTP/1.1
....
Content-Type: multipart/form-data; boundary=--abc123--
....

--abc123-- 
Content-Disposition: form-data; name="file"; filename="in-the-shower.jpg"
Content-Type: image/jpeg

{RANDOM-BINARY-GIBBERISH}
--abc123--
Content-Disposition: form-data; name="description"

Me coming out of the shower

This is a simplified explanation of what kind of “magic” happens when you submit forms in the web. In fact, these two types of form submissions are the only ones which user agents are required to implement.

So where does httpx fit?

Multiple content types

As I explained above, in my company’s product, a user uploads videos to our servers from their devices, along with the metadata, and that all is done via a single multipart request. Our “videos” services know how to process these requests.

Recently, during an integration of video support in a product my team owns, it was necessary to test a certain scalability scenario, and we needed some “dummy” videos on our test servers. Which meant, uploading them using multipart requests.

Although there are certain restrictions regarding adding new dependencies to some projects, this is a general-purpose script, so you can do what you want. Still, my first approach was adapting the same script we use for uploading photos using rest-client, widely used within the company. But I couldn’t make it work, as these multipart requests are too elaborate for rest-client: it involves 1 text value, 2 json-encoded values, and the video file itself. (if you know how to do this using rest-client, do let me know).

My next attempt was trying to use cURL. But even cURL didn’t make such a scenario obvious (I’ve since learned about the ";type=magic/string" thing, don’t know if it’s applicable to my use-case yet though).

And then I thought, “why don’t I just use httpx”?

The httpx solution: the good

Building this using httpx and its multipart plugin was more straightforward than I thought:

require "httpx"

session = HTTPX.plugin(:multipart)
response = session.post("https://mycompanytestsite.com/videos", form: {
  metadata1: HTTP::Part.new(JSON.encode(foo: "bar"), content_type: "application/json"),
  metadata2: HTTP::Part.new(JSON.encode(ping: "pong"), content_type: "application/json"),
  video: HTTP::File.new("path/to/file", content_type: "video/mp4")
})
# 200 OK ....

It just works!

http-form_data: the secret sauce

httpx doesn’t actually encode the multipart body request. Credit for that part of the “magic” above goes to the http/form_data gem, maintained by the httprb org.

(It’s also used by httprb, so the example above could also have been written with it.)

Great! So now I can apply the example script above to handle multiple videos, multiplex them using HTTP/2 (the “killer feature” of httpx), and I’m done, right?

Not so fast.

The httpx solution: the bad (but not that bad, really)

My first attempt at uploding multiple videos ended up being something like this:

require "httpx"

session = HTTPX.plugin(:multipart)
               .plugin(:persistent) # gonna upload to the same server, might as well

opts = {
  metadata1: HTTP::Part.new(
    JSON.encode(foo: "bar"), 
    content_type: "application/json"
  ),
  metadata2: HTTP::Part.new(
    JSON.encode(ping: "pong"),
    content_type: "application/json"
  ),
}

Dir["path/to/videos"].each do |video_path|
  response = session.post(
    "https://mycompanytestsite.com/videos",
    form: opts.merge(
      video: HTTP::File.new(
        video_path,
        content_type: "video/mp4"
      )
    )
  )
  response.raise_for_status
end
# first request: 200 OK ....
# second: 400 Bad Request ... smth smth encoding

400… Bad Request. Well, that was unexpected!

I must admit I flipped seeing that error. So I started troubleshooting. Uploading the second file standalone worked as expected. Then I thought the persistent plugin was the issue, and I removed it as well. Still didn’t worked. And then I thought that something about the HTTP/2 layer was wrong, and as the best positioned person to take a look at the issue, I started agressively putsing the HTTP/2 parser, getting some PTSD about the worst bug reproductions I’d had since starting the project. “This is going to be a long week”, I thought. And then my co-worker reminded me that we didn’t have a week.

So I called it a day, and started thinking about going back to square 1 and doing some spaghetti with rest-client.

I went to sleep, I woke up, and then it suddenly came to me. http-form_data was the reason.

The httpx solution: the ugly (or just, not so pretty)

The issue here was sharing the parameters, and how http-form_data dealt with them. Recently, it added streaming support, by implementing the Readable API in the parts; this enables partial reads on large files, so that programs don’t need to fully load a file in memory and potentially run out of memory. And this “Readable” API was implemented for all parts, including non-file parts.

The Readable API requires an implementation of def read(num_bytes, buffer = nil), and by definition, once you’ve read it all, you don’t read it from the beginning again.

This means that parts could only be read once, before returning nil. So in the example above, metadata1 and metadata2 payloads for the second request were empty. And that caused the encoding issues leading to the “400 Bad Request”.

This was rather unfortunate. However, the fix was simple: share nothing!

require "http"

session = HTTPX.plugin(:multipart)
               .plugin(:persistent) # gonna upload to the same server, might as well

Dir["path/to/videos"].each do |video_path|
  response = session.post("https://mycompanytestsite.com/videos", form: {
    metadata1: HTTP::Part.new(JSON.encode(foo: "bar"), content_type: "application/json"),
    metadata2: HTTP::Part.new(JSON.encode(ping: "pong"), content_type: "application/json"),
    video: HTTP::File.new(video_path, content_type: "video/mp4")
  })
  response.raise_for_status
end
# 200 OK ....
# 200 OK Again!
# Another 200 OK!
# ...

And just like that, I could move on to the next thing!

Conclusion

This was a humbling learning experience. Writing complex HTTP requests using httpx did prove to be as simple and straightforward as I’d hope it would be. However, the “dogfooding” session proved that when it doesn’t work, it’s not very obvious about the why. I spent an entire afternoon plotting on why the hell that second request was failing, and I’m the maintainer. How would another user react? File an issue? Write a blog post about the “sad state of httpx multipart requests”? Or just abandon it and write it in net/http instead?

So I decided to do something about it. Firstly, I updated the wiki from the multipart plugin, warning users not to share request state across requests.

Second, I’ll probably be replacing http-form_data at some point in the future. As a dependency should, it served me quite well, and allowed me to iterate quickly and concentrate on other parts of the project. I’ve even contributed to it. But at this point, owning the multipart encoding is something I’d like to keep closer to the project, and I think that I know enough about multipart requests by now, to warrant “writing my own” and kill yet another dependency. It’ll not happen immediately, though.

Lastly, I’ll try to make a better job at talking about my work, so that hopefully my co-workers one day ask me if I know httpx. And it starts with this post.

Gitlab CI suite optimizations, and how I decreased the carbon footprint of my CI suite

Eco

One of the accomplishments I’m most proud of in the work I have done in httpx is the test suite. Testing an http library functionally ain’t easy. But leveraging gitlab CI, docker-compose and different docker images from known http servers and proxies provides me with an elaborate, yet-easy-to-use setup, which I use fo CI, development, and overall troubleshooting. In fact, running the exact same thing that runs in CI is as easy as running this command (from the project root):

# running the test suite for ruby 2.7
> docker-compose -f docker-compose.yml -f docker-compose-ruby-2.7.yml run httpx

This has been the state-of-the-art for +2 years, and I’ve been learning the twists and turns of CI setups along with it. All that to say, there were a few not-so-obvious issues with my initials setups. The fact that Gitlab CI documentation for ruby has been mostly directioned towards the common how to test rails applications with a database scenario certainly didn’t help, as the gitlab CI docker-in-docker setup, felt like a second class citizen, back in the day. I also ran coverage in the test suite for every ruby version, while never bothering to merge the results, culminating in incorrect coverage badges and inability to fully identify unreachable code across ruby versions.

But most of all, it all took too damn long to run. Over 12 minutes, yikes!

While certainly not ideal, it still worked, and for a long time, I didn’t bother to touch that running system. It did annoy me sporadically, such as when having to wait for CI to finish in order to publish a new version, but all in all, from the project maintenance perspective, it was a net positive.

Still, things could be improved. And experience at my day job with other CI setups and services (sending some love to Buildkite!) gave me more of an idea of what type of expectations should I have from a ruby library test suite. Also, a desire to lower the carbon footprint of such systems started creeping in, and faster test suites use less CPU.

So, after months of failed experiments and some procrastination, I’ve finally deployed some optimizations in the CI suite of httpx, not only with the intent of providing more accurate coverage information, but also to make the CI suite run significantly faster. And faster it runs, now in around 6 minutes!

What follows are a set of very obvious, and some not so obvious, recommendations about CI in general, CI in gitlab, CI for ruby (and multiple ruby versions), quirks, rules and regulations, and I hope by the time all is said and done, you’ve learned a valuable lesson about life and software.

Run only the necessary tasks

By default, the CI suite runs the minitest tests and rubocop, for all ruby versions supported.

A case could be made that linting should only be run for the latest version, and “just work” for the rest. However, rubocop runs so fast that this optimization is negligible.

Except for JRuby. JRuby has many characteristics that make it a desirable platform to run ruby code, but start up time and short running scripts aren’t among its strenghts. Also, rubocop cops are way slower under JRuby.

Also, the supported JRuby version (latest from the 9000 series) complies with a ruby version (2.5), which is already tested by CRuby. Therefore, there is no tangible gain from running it under JRuby.

And that’s why, in gitlab CI, rubocop isn’t run for JRuby anymore.

Cache your dependencies

So, let’s go straight for the meat: the 20% work that gave 80% of the benefit was caching the dependencies used in the CI.

Well, thank you, captain obvious. It was not that simple, unfortunately.

Gitlab CI documents how to cache dependencies, and the advertised setup is, at the moment of this article, this one here:

#
# https://gitlab.com/gitlab-org/gitlab/tree/master/lib/gitlab/ci/templates/Ruby.gitlab-ci.yml
#
image: ruby:2.6

# Cache gems in between builds
cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - vendor/ruby

before_script:
  - ruby -v                                        # Print out ruby version for debugging
  - bundle install -j $(nproc) --path vendor/ruby  # Install dependencies into ./vendor/ruby

This setup assumes that you’re using the “docker executor” mode, you’re running your suite using a docker ruby image, and you’re passing options to bundler using CLI arguments (the -j and --path). The script will run, gems will be installed to vendor/ruby. As the project directory is a volume mounted in the docker container, vendor/ruby will be available in the host machine, and the directory will be zipped and stored somewhere, until the next run.

This couldn’t be applied to httpx though, as the container runs a docker-compose image, and tests run in a separate service (the docker-in-docker service), so this mount isn’t available out of the box. This meant that, for a long time, all dependencies were being installed for every test job in every ruby version all the time!.

Think about that for a second. How many times the same gem was downloaded in the same machine for the same project. How many times C extensions were compiled. Not only was this incredibly expensive, it was unsafe as well, as the CI suite was vulnerable to the “leftpad curse”, i.e. one package could be yanked from rubygems and break all the builds.

The first thing I considered doing was translating my docker-compose setup to .gitlab-ci.yml, similar to the example from the documentation. This approach was tried, but quickly abandoned, as docker-compose provides a wider array of options, some of them which couldn’t be translated to gitlab CI options; moreover, by keeping the docker-compose setup, so I could still be able to run the test suite locally, now I had to maintain two systems, thereby increasing the maintenance overhead.

Recently I found the answer here: docker-in-docker with a mount-in-mount:

Since the docker:19.03.12-dind container and the runner container don’t share their root file system, the job’s working directory can be used as a mount point for child containers. For example, if you have files you want to share with a child container, you may create a subdirectory under /builds/$CI_PROJECT_PATH and use it as your mount point…

Since /builds/$CI_PROJECT_PATH is the mount-point, all you have to do is make sure that the directory exists and is mounted in the child container which runs the test scripts:

# in gitlab.ci

variables:
  MOUNT_POINT: /builds/$CI_PROJECT_PATH/vendor
  # relative path from project root where gems are to be installed
  BUNDLE_PATH: vendor

script:
  - mkdir -p "$MOUNT_POINT"

# in docker-compose.gitlab.yml, which is only loaded in CI mode, not locally when you develop:
...
services:
  httpx:
    environment:
      - "BUNDLE_PATH=${BUNDLE_PATH}"
      - "BUNDLE_JOBS=${BUNDLE_JOBS}"
    volumes:
      # tests run in /home, so making sure that gems will ben installed in /home/vendor
      - "${MOUNT_POINT}:/home/vendor"
      ...

And all of a sudden, cache was enabled:

CI Cache view

… only for the ruby 2.7 build 🤦.

Which takes me to my next recommendation.

Make sure you cache the right directories

As per the example from the gitlab CI documentation, you should be caching vendor/ruby. And if you pay closer attention, you’ll notice that the same directory is passed as an argument to bundle install, via the --path option.

However, if you do this, more recent versions of bundler will trigger the following warning:

[DEPRECATED] The `--path` flag is deprecated because it relies on being remembered across bundler invocations, which bundler will no longer do in future versions. Instead please use `bundle config set path '.bundle'`, and stop using this flag.

If you also look at the docker-compose.gitlab.yml example a bit above, you’ll also notice that I’m setting the BUNDLE_PATH environment variable. This variable is documented by bundler and is supposed to do the same as that bundle config set path '.bundle', which is, declare the directory where the gems are to be installed.

I thought that this was going to be compliant with the example from the gitlab CI, and the ruby 2.7 build gave me that assurance. But I thought wrong.

By now, you must have heard about bundler 2 already: a lot of people have been complaining, and I’m no exception. I particularly get confused with bundler 2 handling of lockfiles bundled by older versions, and bundler -v showing different versions depending because of that. And due to the described issue, I found out that it also interprets BUNDLE_PATH differently than older versions.

To be fair with bundler, major versions are supposed to break stuff if the change makes sense, and the new way makes sense to me. But I can’t use bundler 2 under ruby 2.1, so I have to find a way to support both.

That was easy: since all of them dump everything under BUNDLE_PATH, I just have to cache it.

# in .gitlab-ci.yml
# Cache gems in between builds
cache:
  ...
  paths:
    - vendor
...
variables:
  BUNDLE_PATH: vendor

And just like that, dependencies were cached for all test builds.

Minimize dependency usage

Although you probably need all dependencies for running your tests, a CI suite is more than that: it’s preparing the coverage reports, build documentation, build wikis, deploy project pages to servers, and all that.

You don’t need all dependencies for all that.

There different strategies for that, and although I’m still not done exploring this part, I can certainly make a few recommendations.

gem install

If you only need a single gem for the job, you can forego bundler and use gem instead. For instance, building coverage reports only requires simplecov. So, at the time of this writing, I’m using this approach when doing it:

scripts:
  - gem install --no-cov simplecov
  - rake coverage:report

The main drawback from it is that the gem won’t be cached, and will be installed every single time, while keeping it all contained in a single file, which is very handy for development.

Different Gemfiles

You can also opt for having dependencies for each task segregated into different Gemfiles. For building this website, there’s a particular job which sets a different Gemfile as the default to install jekyll. This is particularly easy when leveraging bundler supported environment variables:

# in .gitlab-ci.yml
jekyll:
  stage: test
  image: "ruby:2.7-alpine"
  variables:
    JEKYLL_ENV: production
    BUNDLE_GEMFILE: www/Gemfile-jekyll
  script:
    - bundle install
    - bundle exec jekyll build -s www -d www/public

Different Gemfile groups

This one is still in the making, as I’m thinking of segregating the dependencies in the same Gemfile, and then using the --with flag from bundler to point which groups to use.

Then I can have the “test”, “coverage”, and “website” groups, and use them accordingly.

This’ll diminish cache sizes and the time to install when cache is busted.

Work around simplecov quirks

simplecov is the de-facto code coverage tool for ruby. And works pretty much out-of-the-box when running all your tests in a single run.

It also supports “distributed mode”, which is, when execution of your code is exercised across different tools (i.e. rspec and cucumber) or across many machines (the knapsack example). In such a case, you can gather all the generated result files, and merge them, using SimpleCov.collate.

This also fits the case for running multiple ruby versions.

So what I do is: first, set up the directory where simplecov results are to be stored as having something unique to that run:

# in test helper
require "simplecov"
SimpleCov.command_name "#{RUBY_ENGINE}-#{RUBY_VERSION}"
SimpleCov.coverage_dir "coverage/#{RUBY_ENGINE}-#{RUBY_VERSION}"

And set it as artifact:

# in .gitlab-ci.yml
artifacts:
  paths:
    - coverage/

Then, gather all the results in a separate job, and collate them:

coverage:
  stage: prepare
  dependencies:
    - test_jruby
    - test_ruby21
    - test_ruby27
  image: "ruby:2.7-alpine"
  script:
    - gem install --no-doc simplecov
    - rake coverage:report

In and of itself, there is nothing here that you wouldn’t have found in the documentation of the projects. However, there’s a caveat for our case: simplecov uses absolute paths when storing the result files, and requires the paths to still be valid when collating.

This breaks for httpx: tests run in a container set up in its own docker-compose.yml, mounted under /home, whereas coveragejob runs inside a “docker executor”, where the mounted directory is builds/$PROJECT_CI_PATH.

Don’t wait too long for simplecov to start supporting relative paths though, as many issues have been reported and closed as early as 2013.

So one has to manually update the absolute paths, before the result files are merged. This can be achieved using our friends bash and sed:

script:
  # bash kung fu fighting
  - find coverage -name "*resultset.json" -exec sed -i 's?/home?'`pwd`'?' {} \;
  - gem install --no-doc simplecov
  - rake coverage:report

And voilá, we have a compounded coverage report.

Set job in badges

Coverage badges are cool, and they are informative.

coverage badge

They can also be wrong, if you’re not specific about where the badge percentage is to be taken from. Since all of the test jobs emit (partial) coverage information, it’s important that we say we want the value for the “compounded” coverage.

You should then use the badge and append the job name via query param, i.e. https://gitlab.com/honeyryderchuck/httpx/badges/master/coverage.svg?job=coverage.

TODOs

A significant chunk of improvements has been achieved. Still there are a few things to do to lower the carbon footprint. Here’s what I would like to accomplish before I call it a day:

Single Gemfile

I’ve touched on this previously, so you should already be familiar with the topic.

Eliminate job logs

Job logs are pretty verbose. I think that can be improved. By eliminating bundler logs (via --quiet or environment variable) and docker services logs (via run instead of up), much can be achieved.

I’d like to be able to easily turn it on though, for debugging purposes.

Dynamic Dockerfile

I’d like to be able to cache the set of commands in the build.sh script. That would be possible if the build process were leveraged by a Dockerfile, but I’d like not to maintain a Dockerfile per ruby version. So ideally, I’d be able to do this dynamically, using a single Dockerfile and some parameter. Let the research begin.

Improve job dependencies

Part of the reason why the overall pipeline takes so long is because of “bottleneck” jobs, i.e. jobs waiting on other jobs to finish.

Also there’s no gain in running the tests if I only updated the website, or docs. One of the latest releases of Gitlab hinted at a feature in this direction for CI.

I’d like to see if there’s something in that department one could do, however.

Conclusion

Technical debt also exists outside of commercial software. And although open source projects aren’t constrained by the needs of “delivering customer value”, there is sometimes self-imposed pushback in improving less-than sexy parts of projects.

Investing some time in improving the state of httpx CI doesn’t make it a better HTTP client, indeed; but a significant part of the “cost” in maintaining it is in the machines constantly verifying that it works as intended. This cost is not always tangible for open source maintainers, as usually machines are at our disposal for free (even if not available all the time), so one tends to chalk it up to “economies of scale” and other bulls*** we tell ourselves.

But the environment feels the cost. The energy to run that EC2 instance running linux running docker running docker-in-docker running ruby running your CI suite comes from somewhere. So the less you make use of it, the more the planet and future generations will thank you for it.

And that concludes my “jesus complex” post of the month.

Ruby 2 features, and why I really like refinements

This is my second entry in a series of thought-pieces around features and enhancements in the ruby 2 series, initiated in the Christmas of 2013, and now scheduled for termination in the Christmas of 2020, when ruby 3 is expected to be released. It’s supposed to be about the good, the bad and the ugly. It’s not to be read as “preachy”, although you’re definitely entitled not to like what I write, to what I just say “that’s just my opinion”, and “I totally respect that you have a different one”.

But after “beating my stick” on keyword arguments, now it’s time to talk about one of the things I’ve grown to like and find very useful in ruby.

Today I’m going to talk about refinements.

Origin story

So, in its teenage years, ruby was bitten by a spider… superpowers yada-yada… and then Active Support.

The origins of refinements can be traced to Active Support, one of the components of Ruby on Rails. And Active Support is the origin story of our super-villain of today: the monkey-patch.

Monkey Patch

Ruby gives you many superpowers. The power to add methods to existing core classes, such as arrays. The power to change core class methods, such as Integer#+, if you so desire. It’s part of what is known as meta-programming. It’s that flexible. But that flexibility has a price. For instance, if you use an external dependency, code you don’t own, that might rely on the original behaviour of that Integer#+, its usage might be compromised with your monkey-patch.

Of course, ruby being ruby, you can always “monkey-patch” that external dependency code you don’t own. But this can happen more than once, and once the monkey-patch ball grows, you realized you were better off before you went down that monkey-patching road. With great power comes great (single-)responsibility(-principle).

Active Support is a big ball of monkey patching in ruby core classes, and then some more. All of these “core extensions” have legimitate usage within Rails code, so they weren’t born out of nothing; they’re an early example of a philosophical approach, of individuals taking over and extending a language and its concepts. In Rails-speak, this is also called the “freedom patch”. It’s a catchy name.

The gospel of Active Support grew along with ruby. The community adopted it and made its word its own. Gems included Active Support as a dependency, where they used a very small subset (sometimes only one function) of these patches. Some maintainers understood the trade-off of carrying this huge dependency forward. Some of them copied the relevant code snipped to its codebase, and conditionally loaded it when Active Support wasn’t around (a long time ago, Sidekiq used this approach). Others suggested porting these patches to ruby itself. One notable example is Symbol#to_proc (once upon a time, writing array.map(&:to_s) was only possible with Active Support).

Monkey-patched core classes became bloated; it caused many subtle errors, and made projects hard to maintain. Gems monkey-patching the same method, when loaded together, completely broke.

The core team took notice, and aimed at proposing a way to enhance core classes in a safe manner. Refinements were released in ruby 2.0.0 .

How

Refinements are a way to scope patches. Want to change Integer#+? Then do it without changing Integer#+ everywhere else:

module Plus
  refine Integer do
    def +(b)
      "#{self}+#{b}"
    end
  end
end

module Refined
  using Plus
  1 + 2 #=> "1+2"
end
1 + 2 #=> 3 

Release

When refinements were annnounced, they weren’t a flagship feature. In fact, the core team wasn’t even sure whether it would be well received, and marked the feature as “experimental”. And reception was certainly cold: the rails core team refused to rewrite Active Support using refinements; the JRuby core team strongly disapproved them, and considered not implementing refinements at all (JRuby has since then implemented refinements, and although Active Support is still refinement-free, usage of refinements can already be found in Rails).

There were real limitations in refinements since the beginnning, and although some have been addressed, some are still there (can’t refine a class with class methods, only instance methods), but the negative backlash from ruby thought-leaders at its inception seems to have slowed experimenting with it, to the point that, around 2015/2016, a lot of “Nobody is using refinements” blog posts started popping up. In fact, it’s still perceived as some sot of “black sheep” feature, only there to be scorned at.

I was once in that bandwagon. And then, slowly, the sowing gave way to reaping.

Signs

In time, refinement-based gems started popping up. The first example coming into my attention was xorcist, which refined the String class with the #xor byte stream operation. Then I noticed that sequel also began adopting refinements.

At some point, I started considering using refinements for a specific need I had: forwards-compatibility.

Loving the alien

Something started happening around the release of ruby 2.2. I don’t remember how it started, but suddenly, prominent ruby gems started dropping support for EOL rubies. The idea seemed to be, following the same maintenance policy as the ruby core team would drive adoption of more modern rubies.

Upgrades don’t happen like that, for several reasons. Stability is still the most appreciated property of running software, and upgrades introduce risk. They also introduce benefits and hence should obviously happen, but when they do, they happen gradually, in order to reduce that risk.

Deprecating “old rubies” interfered with this strategy; suddenly, you’re stuck between keeping your legacy code forever and ignore the CVE reports, or do a mass-dependency upgrade and spend months in QA. You know what businesses will do when they are presented with the options: if it ain’t broke, don’t fix it. Just ask Mislav, or think why you’re still maintaining a ruby 2.2 monolith at work. It’s 2020, and the most used ruby version in production is version 2.3 .

Me, I prefer “planned obsolence”. It works for Apple. Is your code littered with string.freeze calls, and you want the ruby 2.3 frozen string literal feature? Do it. Older rubies will still run the code, just their memory usage will not be optimal. Users will eventually notice and upgrade. Want to compact memory before forking? Go ahead. Code still runs if there’s no GC.compact. I know, it’ll go slower. Just don’t break user code.

Want to use new APIs? Well, there used to be backports. But now you have something better: Refinements.

Backwards compatibility is forwards compatibility

In life, you must pick your battles.

In all the gems I maintain, I start with these two goals: set a minimum ruby version and alwayw support it (aspirational goal); also, I want to use new APIs whenever it makes sense.

How do I choose the minimum version? It depends. ruby-netsnmp supports 2.1 and higher because I didn’t manage to make it work with 2.0 at the time using “celluloid”; httpx supports 2.1 and higher because the HTTP/2 parser lib sets 2.1 as its baseline, and although I maintain my fork of it, there hasn’t been a strong reason to change this; rodauth-oauth supports 2.3 and higher mostly because it started in 2020, so I don’t need to go way back; also, 2.3 is still the most used ruby version, so I want to encourage adoption.

In all of them, I use refinements. I use the xorcist refinement in ruby-netsmp; in rodauth-oauth, I refine core classes in order to use methods widely used in more modern ruby versions.

But the refinement usage I like the most, is the one from httpx. There, not only I implement slighty-less performant versions of modern methods to keep seamless support for older rubies, I also enhance the concept of URI, and add the concept of origin of authority. The implementation is very simple, so here it is:

refine URI::Generic do
  def authority
    port_string = port == default_port ? nil : ":#{port}"
    "#{host}#{port_string}"
  end

  def origin
    "#{scheme}://#{authority}"
  end

  # for the URI("https://www.google.com/search")
  # "https://www.google.com" is the origin
  # "www.google.com" is the authority
end

Why do I like it? Because the concept of what a URI “origin” or “authority” are, is well defined and written about in RFCs. The HTTP specs are filled with references to a server “authority” or “origin”. Everybody heard and used the HTTP “:authority” header (formerly “Host” header), or CORS (the O stands for “Origin”).

And yet, the ruby URI library doesn’t implement them. Yet. Should it? Maybe. I could definitely contribute this to ruby. Maybe I will. But I still have to support older rubies. So my refinements ain’t going nowhere. This refinement is the present’s forwards compatibility and the future’s backwards compatibility. It keeps my code consistent. And by making it a refinement, I don’t risk exposing it to and breaking user code.

This is refinements at its finest.

Conclusion

Refinements are a great way to express your individuality and perception of the world, while not shoveling that perception of the world onto your users; a safe way for you to experiment; and a great way to keep backwards compatibility, and by extension, your end users happy.

Unfortunately they will never accomplish its main goal, which was to “fix” Active Support. But maybe Active Support was never meant to be fixed, and that’s all right. Refinements have to keep moving forward, and so do we. Hopefully away from Active Support.

Ruby 2 features, and why I avoid keyword arguments

Some politician one day said “may you live in interesting times”. Recently, it was announced that the next ruby release will be the long-awaited next major ruby version, ruby 3. That’s pretty interesting, if you ask me.

As the day approaches, it’s a good time to take a look at the last years, and reminisce on how ruby evolved. There have been good times, and there have been bad times. I can single-out the 1.8-1.9 transition as the worst time; the several buggy releases until it stabilized with 1.9.3, a part of the community pressuring for removal of the Global Interpreter Lock and getting frustrated when it was not, the API breaking changes… yes, it was faster, but people forget that it actually took years (2 to 3) until the community started ditching ruby 1.8.7 . True, not a Python 2-to-3 schism, but it fractured the community in a way never seen since.

Ruby 2 was when this all changed: upgrades broke less, performance steadily increased, several garbage collector improvements… ruby became adult. Matz stressed several times how important was not to break existing ruby code. Companies out of the start-up spectrum adopted ruby more, as they perceived it as a reliable platform. Releases generated less controversy.

With the exception of new language additions.

I can’t remember the last addition to the language that didn’t generate any controversy. From the lonely operator (.&), to refinements, all the way up to the recent pattern-matching syntax, every new language feature stirred up debate, opinions and sometimes resentment. Some features have been received so negatively, that they’ve been removed altogether (such as the “pipeline” operator).

The ruby community and the ruby core team still don’t have a way to propose and refine language changes, and the latter prefers to release them with the “experimental” tag. Interest hasn’t developed in adopting standards from other programming languages (such as python PEPs, or rust RFCs), as the core team, or Matz, are still wary of the consequences of “designed by committee” hell, but the current approach of “let me propose new feature and wait for Matz’s approval, and then let’s release and see what people think” hasn’t worked in ruby’s favour lately. Here’s hoping to a solution coming up for this conundrum someday.

Of course, “controversial” means that some people will love them, some people will hate them. Although it kinda feels that way, I’m not against all changes, I did like some of them. This is my blog after all, so expect opinions of my own, and if you don’t like it, feel free to comment about it (perhaps constructively), somewhere in the internet. I’ll also write in the future about features I love that came from the ruby 2 series. But that won’t be today. Today is about a feature I’ve grown to abandon.

Today is about keyword arguments.

Where they came from

Keyword arguments were introduced in Ruby 2.0.0.p0, so they’re ruby 2 OG. They’re a result of an old usage pattern in ruby applications, which was to pass a hash of options as the last argument of a method (shout out extract_options!). So, pre-2.0.0, you’d have something like this:

def foo(foo, opts={})
  if opts[:bar]
  # ...
  # ...
end

foo(1, bar: "sometimes you eat the bar")

Usage of this method signature pattern was widespread, and things generally worked.

There were a few drawbacks of such an approach though, such as proliferation of short-lived empty hashes (GC pressure), lack of strictness (for example, what happens if I pass foo(1, bar: false)? is foo(1, bar: nil) the same as foo(1)? How should that be handled?), and primitives abuse (let’s face it, Hashes are to ruby what the Swiss Army Knife is to MacGyver; Sharp tools).

The core team decided to address this usage pattern in the language. They took a look at existing approaches, most notably python keyword arguments, and proposed a more ruby-ish syntax.

And on Christmas day 2013, keyword arguments were born.

So, where did it go wrong?

Lack of support for hash-rocket

One of the additions in ruby 1.9 was JSON-like hash declarations. This was, and still is, a very controversial addition to the language. Not because it does not look good or because it’s fundamentally flawed, but mostly because its limitations do not allow the deprecation of the hash-rocket syntax. You see, JSON-like hash declarations only allow having symbols as keys:

{ a: 1 } #=> key is :a
{ "a" => 1 } #=> key is "a", and you can't write it any other way
{ a => 1 } #=> key is whatever the variable holds. must also use hash rocket

So you get many ways to slice the pie. How can you convince your teammates? How can you spend more time producing, and less time debating what is the correct way to write hashes in ruby?

For keyword arguments, the core team went up a notch and decided to only support the JSON-like hash.

kwargs = { bar: "bar" }
foo(1, **kwargs) # works
kwargs = { :bar => "bar" }
foo(1, **kwargs) # still works, but hard to reconcile with signature
kwargs = { :bar => "bar", :n => 2 }
foo(1, **kwargs) #=> ArgumentError (unknown keyword: :n)
kwargs = { :bar => "bar", "n" => 2 }
foo(1, **kwargs) # ArgumentError (wrong number of arguments (given 2, expected 1; required keyword: bar))

This is confusing: you can create hashes in two ways, for certain type of keys there’s only one way, but then keyword arguments only support the other way, but they’re not hashes, so they’re you go… I can’t even…

The httprb team has refused pull requests in the past changing all internal hashes from rocket to JSON-like and introducing keyword arguments, because they don’t see the benefits in maintaining this inconsistency. They avoided keyword arguments.

It’s a hash, it’s not a hash

The 3rd example above shows what one loses when adopting keyword arguments: suddenly, an options “bucket” can’t be passed down without filtering it, as all keys have to be declared in the keyword arguments signature.

There is some syntactic sugar to enable this, though:

def foo(foo, bar: , **remaining)
  # ...

However this approach creates an extra hash; although bar is a local variable, remaining is an hash. A new hash. And more hashes, more objects, more GC pressure.

The pre-2.0.0 hash-options syntax has the advantage of passing the same reference downwards, and you can always optimize default arguments. You can improve resource utilization by avoiding keyword arguments.

Performance

Due to some of the things talked about earlier, particularly around keyword splats (the **remaining above), performance of keyword arguments can be much slower when compared to the alternative in some cases.

Take the following example:

# with keyword arguments
def foo(foo, bar: nil)
  puts bar
end
# without keyword arguments
def foo(foo, opts = {})
  puts opts[:bar]
end

Using keyword arguments here is faster than the alternative: arguments aren’t contained in an internal structure (at least in more recent versions), and there is no hash lookup cost to incur.

However, in production, you’re probably doing more of this:

# with keyword arguments
def foo(foo, bar: nil, **args) # or **
  puts bar
end

And here, a new hash has to be allocated for every method call.

So if you think you’re incurring the maintenance cost for some performance benefit, you might be looking at the wrong feature. You can solve it by avoiding keyword arguments.

Don’t think I came up with this by myself, Jeremy Evans, of sequel and roda, mentioned this in a very enlightening Rubyconf Keynote. He also avoids keyword arguments, and came up with a strategy to keep allocations down while using old-school options hash.

It’s not quite like python keyword arguments

Although they’re not everyone’s cup of tea for pythonistas, keyword arguments make more sense there. Let me give you an example:

# given this function
def foo(foo, bar):
    pass
#  you can call it like this
foo(1, 2)
foo(bar=2, foo=1)

This is a contrived example, but you get the idea: you can refer to the arguments positionally, or doing it out-of-order if you so prefer.

In ruby-land, keyword arguments were a solution for the specific usage pattern already described above, and they will probably never become what we see in python. But the flexibility and consistency of the keyword arguments in python is one of the 2 features I’d welcome in ruby (the other being import. Still not a fan of global namespace).

That ruby 2.7 upgrade

If you upgraded your gems or applications to ruby 2.7, you already know what I’m talking about.

If you don’t, the TL;DR is: Ruby 2.7 deprecates converting hashes to keyword arguments on method calls, and all the historical inconsistencies that came along (official TD;DR here), and will discontinue it in ruby 3.

The reasoning behind it is sound, of course. But 7 years have passed. A lot of code has been written and deployed to production since then, significant chunks of it using keyword arguments. Some of this code is silenty failing, some of it worked around the issues. And now all of that code is emitting those pesky warnings, telling you about the deprecation. All of that open source code you don’t control, some of it abandoned by its original author.

Github recently upgraded to ruby 2.7, and singled out the keyword arguments deprecation as the main challenge/pain. “We had to fix over 11k warnings”, they say. A lot of it in external libraries, some of which github provided patches for, some of which were already abandoned and github had to find a replacement for.

I do understand the benefit of making your code ready for ruby 3, or the advertised performance benefits, but I can’t help to think how business looks at the amount of resource spent yak-shaving and not delivering customer value.

This could all have been avoid, had github and its external dependencies avoided using keyword arguments.

YAGNI

Keyword arguments are a niche feature: they’re a language-level construct to solve the problem of “how can I pass an optional bucket of options” to a certain workflow. They were never meant to fully replace positional arguments. They were meant to enhance them (although I’ve been making the point that they don’t fully accomplish the goal).

However, what you got was a sub-culture that only uses keyword arguments. Why? Because they can. Did you ever see a one-argument method, where that one argument is a keyword argument? Congratulations, one of your co-workers belongs to that sub-culture.

I’ve seen this happen for Active Job declarations, which support keyword argument signatures. Sidekiq Workers don’t, but Active Job takes care of the conversion: it converts hashes of options, symbolizes all keys, and passes them at keyword arguments. More hashes, more objects, more GC pressure.

Do keyword argument-only method signatures get more maintainable? I guess you’ll get different answers from different people. But the benefit of it isn’t clear.

Conclusion

Keep calm

Reality check time: keyword arguments aren’t going anywhere any time soon. They’re a language feature now. Some people like it, inconsistencies notwithstanding. And let’s face it, where’s the value in removing keyword arguments from your projects? Code is running. Don’t change it! I know I didn’t. ruby-netsnmp, a gem I maintain, still uses keyword arguments, and that ain’t gonna change, not by my hand at least.

But if you’re authoring a new gem in 2020, or writing new code in the application you work at daily, do consider the advice: avoid keyword arguments. Your future upgrading self will thank you.

Ramblings about initial design decisions, internals, and devise

Yesterday I was reading this twitter thread, where Janko Marohnić, the maintainer of Shrine, who has recently integrated rodauth in Rails and is preparing a series of articles about it, describes the internals of devise, the most popular authentication gem for Ruby on Rails, as “making every mistake in the book”, claiming that rodauth, the most advanced authentication framework for ruby, is much better because its internals are “easier to understand”, thereby sparking some controversy and replies, with some people taking issue with these claims, and also with his approach of criticizing another gem because of “look how awful its internals look like”.

Although Janko does “mea culpa” on his tone, the claim and subsequent comments made me think about it. Is the state of the internals a reliable factor when picking a gem? Does it hamper its future development? Is it actually a goal, to further develop it? Is it possible to extend it, or support newest protocols and standards? Is it feature-complete, according to its initial goals? And what if a project isn’t maintained anymore by its original author, can the community decide it’s not feature-complete, and easily fork it away?

Let me just start by saying that, although I think that “internals” don’t matter much when evaluating a robust and community-approved solution such as devise, recommending it does sound like “no one ever got fired for buying IBM”. And while, as a user of a library, public API, documentation and ease of integration is way more important, as a contributor, quality of internals directly impacts my ability to quickly fix bugs and add features.

In retrospect, the state of the internals of the http gem was the reason that led me to develop httpx.

Taking that into consideration, I’ll just reinterpret Janko’s claim as “all you guys there struggling to maintain legacy devise, keep calm and join the rodauth community”.

Does he have a point though?

Early design decisions

Most libraries start being developed with simple goals, and then evolve from it. Sometimes you want to “scratch a hitch”. Sometimes you want to prove a point. Sometimes you want to play around with a new programming language, and reimplement something in it. Sometimes you start building something for yourself, and then you extract the “plumbing” and share it with the mob.

And sometimes, you’re not happy with the existing tools, and you think that you can (and want to) do better.

Many popular libraries started that way. sidekiq, for example, positioned itself early-on as a “better Resque”, long before it started charging for extra features.

And so did devise.

In the “BD” (Before Devise) era, there were other gems solving authentication in Rails: there was acts_as_authenticatable, there was authlogic, and others I can’t remember well enough (2009’s been a long time ago). They had a lot of things in common: a lot of intrusive configuration in Active Record (it’s 2020 and it’s still happening), required significant boilerplate, and lacked a lot of important features, defaults and extension points. Building authentication in 2009 was certainly not easy nor fun.

devise was developed inside Plataformatec, the brazilian company which gave the Rails core team 3 (or 4?) ex- or current core team members. Its author is José Valim (a huge ruby contributor at the time, before Elixir), and maintenance of the project has been mostly taken over by company employees (although it receives contributions from a large community). It was initially developed for Rails 3. In fact, I’d go as far as saying that devise was built to showcase what could be achieved with Rails 3, as it was the first popular demonstration of a rails engine.

The first time I tried it, it was a breath of fresh air: no handmade migration/model DSL setup (there was a generator for that); “pluggable” modules; default routes/controller/views to quickly test-drive authentication and signup; everything “just worked (tm)”. It was so much better than the alternatives!

I always felt that devise was made to better integrate the then-standard form-based email/password authentication and signup in Rails 3. The main goals, it seemed to me, were:

  • Quick integration in a rails 3 application (increase adoption);
  • Provide better email/password account authentication defaults (commoditization);
  • Become a rails engine success story (community);

All of the goals were successfully reached. It’s one of those gems that always drops in the conversation when “how we’ll do authentication” comes in the conversation for a new project. There is a big community using and fixing outstanding issues. It was so ubiquitous at one point, that there was a subset of the community who thought it should be added to rails, which fortunately never happened, as not all projects need authentication (not all projects need file uploads and a WYSIWYG text editor as well, by the way).

So why are people making a case against it? Why go with rodauth instead?

Welcome to the desert of the real

Desert of the real

The vision for devise was fully accomplished by 2010: a no-friction email/password authentication add-on for Ruby on Rails.

In hindsight, I don’t think that anyone in 2009 could anticipate today’s practices: microservices, SPAs, Mobile applications, cross-device platforms… and authentication also evolved: phone numbers instead of email accounts, multi-factor authentication, SMS tokens, JWTs, OTPs, OpenID, SAML, Yubikeys, Webauthn… and stakes are higher, especially since Edward Snowden and PRISM proved that theoretically breaking into accounts isn’t so theoretical after all.

Probably everyone was anticipating an ecosystem of “extensions” to flourish around the core library. And eventually, the “extensions” came to be, although the quality, stability and inter-operability of the bunch left a lot to be desired. And some of it had to do with the foundations devise built on top of.

A Rails engine is, in a nutshell, a way to add a “sub-app” to a rails app. It was a feature introduced in Rails 3 (a “patch” to circumvent a limitation of rails apps being singletons). You can add controllers, views, models, helpers, initializers, etc… to it, while not “polluting” your main app.

devise does all that and more, which works great for vanilla devise when extending yout application. But extending an engine is different. There isn’t an agreed-upon way on how to extend another engine, and devise suffers from this by proxy. Go ahead and take a look at the existing extensions. Here are some highlights:

  • there are two oauth2 integrations (none of them has been updated in the last 8 years), one does it through more controllers/models, the other just adds a new `devise module;
  • there is an openid authenticatable extension, and then there are a lot of provider-specific (twitter, google, facebook) sign-in extensions, which implement OpenID or OAuth internally;
  • there is a devise-jwt integration, surprisingly still being maintained (most of the extensions I click on this list haven’t gotten an update in 5 years or more!), which lists a lot of caveats around session storage, mostly because devise defaults to using the session and doesn’t support tranporting authentication information in an HTTP header without a few workarounds;

So, although it’s easy to customize and extend devise from within your application, extending it through another library is a non-trivial exercise of rails engine-hackery, as it’s not clear where your extension should go, which will make it end up all over the place.

(The state of devise extensions maintenance, at least judging by the ones advertised in the Wiki, doesn’t look solid either.)

And then there’s rails itself.

Looking at the CHANGELOG, rails integration and upgrades have also been the main story since 2016. See the strong_parameters integration in the README, or how devise major version bumps are usually associated with a new rails version support.

It does seem that the main concern has been on stability rather than new features. Which I can relate, breaking other people’s integration does suck. But is this by design? Is devise feature-complete? Did it achieve all its intended initial goals, that nothing is left beyond maintaining it for the community? Is the refactoring of its internals necessary to build new features? Would less logic in models and less AR callbacks help develop new features? I guess only the core maintenanceship can answer that.

But it does feel that devise is legacy software.

To infinity… and beyond!

Buzz Lightyear

OK, so all our tools are irreparably broken, it’s a sad state of affairs, and the end is nigh. Should we all just migrate to rodauth?

The answer is a resounding “it depends…”.

Boring, right?

devise is probably a legacy project, but guess what, so are a lot of rails apps out there. There’s a sunk cost there, after one adopts, integrates and patches all of these tools together, to the point that, when it works, it might be just good enough, and although it sounds like “the grass is greener on the other side”, the unknowns might be too many, making you stick with “the devil you know”. So, until someone devises (pun intended) a tool to auto-migrates an application from devise to rodauth, thereby reducing the migration cost (there’s an OSS idea there), I don’t think that’ll happen, regardless of internals.

However, if you’re starting a project in 2020, you should definitely give rodauth a try. It states “security”, “simplicity” and “flexibility” as design goals right there at the beginning of the README. It still sees active development beyond plain maintenance, and supports all of those mentioned modern authentication features that should be a must in 2020. Its internals aren’t perfect, but Janko is right, they are easier to understand and work with, so much so that I made a library to build OAuth/OpenID providers with it.

It lacked in documentation and guides, so I’m definitely looking forward to those upcoming Rodauth articles!

What about you, how do you value a library’s internals?