Introducing Janus a read/write proxy for ActiveRecord 7.1+

TL;DR

In ancient Roman religion and myth, Janus (/ˈdʒeɪnəs/ JAY-nəs; Latin: Ianvs [ˈi̯aːnʊs]) is the god of beginnings, gates, transitions, time, duality, doorways,[2] passages, frames, and endings. (wikipedia)

Janus is a read/write split proxy for MySQL/ActiveRecord 7.1+. It works by sending any read queries to a replica, and any write queries to the primary database server. If there’s been a write statement within the same thread we stick any future queries to the primary server unless the developer specifically chooses not to.

Other than that Janus should be pretty hands off. We deployed on our infrastructure (migrating from Makara) on the 2024-04-09 and saw no effect on our metrics.

Why did you create it?

We introduced Makara into our setup back in September 2016 when we were approaching the stage of either needing a more powerful database server or moving to the more robust setup of having a primary/replica database setup. Makara was the perfect project for us as it transparently handled directing SQL queries to the appropriate server without any thought from the developer (aside from considering some replica lag for Sidekiq. If we ever needed to specifically read from a primary server we could also tell it using the Makara::Context helper.

After serving us well for many years, we upgraded our main API codebase to Rails 7.1 and after plenty of testing we deployed to our production environment. Immediately we saw the load on our primary database server jump and traffic on our three read replicas drop to almost zero. After investigation we determined that Makara was no longer working with Rails 7.1.

Makara was first pushed to Github back in 2012 initially by TaskRabbit, but eventually taken over by Instacart (please let me know if I’ve got that wrong!). The project was well supported with features and bugfixes coming fairly regularly. Unfortunately, active development of this project ended some time ago, I assume this is because Instacart have moved over to DynamoDB. Just over a year ago I did contact one of the developers via X who confirmed this but were happy to accept pull requests. The last commit on the project was June 2023 however.

Tweet about status of Makara project

This left us in the position of either fixing Makara, finding an alternative solution, or creating something of our own…

But why didn’t you use…

Rails build in read/write split?

With Rails 6 came the ability to use multiple databases both for read/write split and for sharding. We investigated and attempted to proof of concept this within our codebase. Unfortunately we were hit with two issues:

The setup assumed a CRUD style application rather than a rich API. It’s hooked into routes rather than looking at the actual query content itself (we tried creating our own database selectors)
We’d need to implement code through our codebase to manually specify which database to read from. Not only would this be messy but it would take a long time to achieve

Maybe this will come with a future version of Rails!

Makara

The Makara codebase is quite extensive and supports various versions of ActiveRecord as well as different database engines. We started investigating the code, but any contribution we’d make would likely only be for the MySQL adapter (since that’s what we use) and not tested elsewhere. Additionally whether we’d get the contributions merged and released given the project’s state was another concern - we didn’t want “yet another fork” out there in the community.

When we investigated the codebase it soon became clear that a new stripped back library might be the best solution for us.

ProxySQL

ProxySQL seemed like it might be the route we’d go for a period. Rather than run 1-2 ProxySQL instances we’d discussed placing a proxy on each on of our servers and having our Rails app talk directly to that (since we were only interested in the read/write split rather than any connection pooling functionality). Upon further investigation it seemed that ProxySQL would require a lot more work than simply deploy and add simple rules. It would also not directly support reading from primary after a write, releasing that context, and we’d need to setup rules to achieve what we’re after. Additionally there was little other evidence around the internet of people using Rails and ProxySQL together, so we’d be somewhat going into the unknown (Shopify being the exception) .

https://labs.clio.com/rails-on-proxysql-80a18236b093
https://www.cloudthat.com/resources/blog/the-power-of-proxysql-to-split-read-and-write-queries-in-amazon-rds

What’s our setup?

Just so you can have an understanding of what the library needed to achieve for us we’ll share a little about our infrastructure/setup.

We make use of AWS Aurora MySQL. We currently have a single primary server with three read replica servers. Two of our replicas are dedicated to serving data to our Sidekiq job queues with the other replica serving our web traffic (we make heavy use of job queues at Olio).

With Makara you can specify multiple replica servers and it handles servers not being available (temporarily blocklisting them). We don’t need to support this since failovers and availability to handled by AWS via read/write endpoints. In our case if a failover happened we’d just want an exception to be raised and a database reconnection attempted.

We deployed Janus to our production envioronment on 2024-04-09 and saw no effect on our metrics at all. We’ve since fixed a few small bugs with regards to routing and will correct anything that appears whilst running the library at Olio.

Getting started

Installation

Add the current version of the GEM from rubygems in your Gemfile.

gem 'janus-ar'

This project assumes that your read/write endpoints are handled by a separate system (e.g. DNS).

Configuration

Update your database.yml as follows:

development:
  adapter: janus_mysql2
  database: database_name
  janus:
    primary:
      <<: *default
      host: primary-host.local
    replica:
      <<: *default
      password: ithappenstobedifferent
      host: replica-host.local

Run

That’s it. If you look at your server logs we prefix any ActiveRecord queries with [primary] or [replica] so you can see which server is being used.

In development/test environment

If you want to create a read only user for development work you can do so by using:

CREATE USER IF NOT EXISTS 'readonly'@'%' IDENTIFIED BY 'password';
GRANT SELECT ON `#{database}`.* TO 'readonly'@'%';
FLUSH PRIVILEGES;

Future plans

Once we’ve been running Janus for a period and are happy with its stability the next step will be to add a Trilogy adapter in rather than using the MySQL2 adapter. At this stage this is mostly to improve the internal developer experience where installing the mysql2 GEM is often the cause of frustration on MacOS. FYI our current guidance internally is to run the following before bundle install:

bundle config build.mysql2 --with-ldflags=-L$(brew --prefix zstd)/lib

Please do get in touch if you are using Janus, we’d love to have more input and feedback!