This was Gusto's system graph. Each of the black rectangles you see here is a subsystem within Gusto's biggest Rails monolith, and the red arrows are where one subsystem talks to another.
As our business grew, folks started having a hard time making large changes in our codebase. Initially, we tried solving some of these problems with gems, Rails engines, and microservices – with mixed results.
If this story is familiar to you, you may be working in a big Rails application. Gusto has attempted to solve some of these problems through Big Rails.
Big Rails is a system of sociotechnical tools, practices, and conventions that scale Rails development in terms of lifespan, number of contributors, and complexity.
In this post, I'll talk a little bit about how we tried to solve these problems and share some tools we've open-sourced.
To solve the problems of a Big Rails application, there are 5 key principles that drive our approach.
- Accountability and Ownership
- Clear Boundaries
- Thoughtful Dependency Management
- Gradual Adoption
- Intentionally Curated and Sustainable Feedback Loops
Accountability and Ownership
module PayrollSyncer def self.sync_deductions! raise "I am an error" end end
Imagine this code exists in your codebase. When PayrollSyncer.sync_deductions! is called, who receives the error? How is it routed to the right team? Who do you ask about this code when git blame shows dozens of folks who have changed teams or left the company?
Accountability and ownership is all about striving to clearly define which teams own which domains. It should be dead simple for both automated tooling and engineers working within the codebase to identify areas of ownership.
To accomplish this within our own Big Rails application, we created and open-sourced two gems: one to manage code ownership and the other to manage teams.
- github.com/rubyatscale/code_ownership (and github.com/rubyatscale/code-ownership-vscode)
Here is a simple team declaration…
# config/teams/payroll.yml name: Payroll github: team: '@Gusto/payroll' owned_globs: - app/services/payroll/**/** - app/services/payroll_syncer.rb
... and how we tie that team to that code in the snippet above.
team = CodeTeams.find('Payroll') CodeOwnership.for_file('app/services/payroll_syncer.rb') == team CodeOwnership.for_class(PayrollSyncer) == team begin PayrollSyncer.sync_deductions! rescue => ex CodeOwnership.for_backtrace(ex.backtrace) == team end
With this, we could suddenly tie that original piece of code back to the team. As an added bonus, this functionality started conversations about code ownership (or lack thereof) throughout our codebase.
Clear Boundaries and Dependency Management
Having clear boundaries is about working towards easily understandable conceptual and mechanical separation between domains. Each system should only talk to other systems via intentionally maintained public APIs. Thoughtful dependency management means minimizing dependencies. This helps engineers reduce cognitive load and understand how systems work together. When we must take on a dependency, we should do so explicitly. We should avoid creating cycles in our dependency graph, as they reduce our ability to understand a subsystem in isolation.
To move towards this goal, we started with one simple change to a standard Rails convention.
app/ models/ benefits/ payroll/ hr/ views/ benefits/ payroll/ hr/ controllers/ benefits/ payroll/ hr/ services/ benefits/ payroll/ hr/
Here is a contrived example of a standard Rails app. It has an app directory containing secondary directories for "architectural concerns" which point to the various domains.
Imagine you are an engineer on the benefits team at Gusto. You have to jump around the codebase to make a change within one domain. This violates an important principle of coupling and cohesion – things that change together should live together.
So we made that change:
packs/ benefits/ models/ views/ controllers/ services/ payroll/ models/ views/ controllers/ services/ hr/ models/ views/ controllers/ services/
We decided to organize things on the basis of their business domain first, followed by the basis of architectural concern. This hid the technologies underlying a domain while co-locating test files. Although this is possible with vanilla Rails, we open-sourced a tool called stimpack to make this sort of configuration easy.
Now that our app was organized by domain, we leveraged packwerk to systematically manage the relationships between these domains. Kudos to Shopify for this great tool! With packwerk we kept the same structure as above, but added a package.yml and a public folder (where the public API lives) in each domain folder. Each pack is owned by one team, which is implemented by the code_ownership gem.
Packwerk creates a directed graph of the statically analyzable references from one package to another. If one pack calls code defined in another pack it forms a reference, or an edge, in that graph. Packwerk then analyzes these references and declares a reference as a "privacy violation" if one pack is referencing something in the private API of the other pack and a "dependency violation" if it's referencing another pack without a declared dependency in the package.yml file. Packwerk outputs these violations in a YML list which is indispensable to improving system boundaries gradually over time. This is only a brief explanation of how packwerk works. Checkout the Packwerk docs for more details.Packwerk does all of this without being loaded into the runtime. This means packwerk is not a production dependency. Packwerk runs as a separate process on CI or locally. Note that packwerk only parses explicit references to classes, constants, and modules. We find that pairing packwerk with sorbet and typing our methods helps packwerk provide even more meaningful output.
But what about gems and engines?
I mentioned earlier that we tried using inline gems and engines to modularize our application. Packwerk has some key differences from gems and engines:
Here’s a bit more explanation:
- Packwerk, unlike gems and engines, supports gradual modularity. Gems and engines sometimes force you to solve modularization problems in areas of potentially low business value, where packwerk allows you to state the idealized system diagram (by creating packages, moving files into them, creating public API, and stating relationships between packages), and then it gives you the TODO list to get you there. Packwerk decouples statements about system structure and boundaries and the implementation of those boundaries.
- Packwerk packages are inexpensive to create and change. There is very little boilerplate, and changing the relationships between packages is as simple as changing YML files.
- Packwerk does not support distribution or versioning. If that's what you need, you'll definitely need to leverage gems for that.
- Packwerk supports fast tests like gems if you’re using spring and bootsnap.
- Gems support strict boundaries. Since a package with no violations can easily be a gem, we released a tool called package_protections to ensure that a package can remain as clean as a gem or engine.
- Packwerk packages can have engine features, such as being able to have an isolated routes.rb file, when paired with stimpack.
At Gusto we strongly believe gems and engines are and will continue to be a critical component of the modularization toolchain. We have also found domains with gem potential that we are perfectly happy to maintain as a package.
A system never starts off as a Big Rails application – it grows into one organically. This means that the Big Rails tools must be able to be adopted gradually, since a small app might not need them.
As we begin to adopt these tools, we have to remember that the technology transformation must be accompanied by a corresponding cultural change in the way teams use these tools day to day.
We cannot simply drop in these new technologies and expect a transformational change to occur.
This became obvious when we first introduced packwerk. I noticed folks consistently updating the "TODO lists" packwerk provides and maintains, rather than fixing the underlying system design issues. To understand more about why this was happening, I needed to dig in.
To do this, I set up a slack notification to alert me every time a packwerk TODO list was changed. I took each alert as an opportunity to ask folks more about how they were interacting with this tool. After looking at about a thousand pull requests, we were able to identify a short list of all of the valid and not-so-valid reasons for updating the TODO lists.
More importantly, I frequently met with engineers over zoom to chat more about what we were trying to do and how we were trying to scale our system. Over time, a cultural shift took place. Engineers understood what we were doing and why. They began to add proactive context about their interactions with the tool or fix the system design issues in response to its output.
Over time, based onrf user feedback, we built up some tooling. We open-sourced a VSCode Extension to interact with packwerk. We released a tool called danger-packwerk to leave inline, automated feedback on pull requests related to packwerk errors. Lastly, we released a tool called modularization_statistics to be able to track our progress over time.
We're making progress. As for that dependency graph?
Well... we still have a lot of work to do!
Just as Rails is the product of an engaged and passionate community, I hope we can take the same approach with Big Rails applications. At Gusto, we're so grateful for the contributions of the individuals and companies who have helped be part of the solution. I'd love to engage more with the community on questions like:
- In what ways can Ruby and Rails continue to offer great tools and cultural norms that help users create well-modularized systems?
- What can the different conventions of packwerk packages, gemspecs, and other packaging systems learn from each other?
All of these open source gems we've released, along with the tools that the broader community have released, are, like us, imperfect. If you're interested in this problem space, you can:
- Create issues, start discussions, or open PRs in any repository within https://github.com/rubyatscale
- Join us in the Ruby/Rails Modularity Slack Server: https://rubymod.slack.com
- Come work at Gusto – we're hiring for passionate folks to work on problems just like this and more.
- “Gradual Modularization for Ruby and Rails” by Stephan Hagemann
- “Boundaries” by Gary Bernhardt
- “Thinking in Systems” by Donella Meadows
- Gusto Modularization Toolchain (PDF)