Defining an outcome-driven experiment
At Gusto, Engineering has an ongoing discussion about how to quickly ship meaningful features to our customers. I am part of the Partners Engineering team here, building our product for accountants and bookkeepers. As a team, we want to work on the right things for our customers. We want to feel confident that features will be impactful when building something new.
My team decided to take a look at our process. We broke down a typical feature planning timeline and found that we typically spent three to five months from start to finish. This is a significant amount of time to spend on a feature, especially when we are uncertain how it will be received by our users.
In an ideal world, my team would ship smaller features to users more quickly. Doing so would allow us to gather immediate real world user feedback and take smaller steps to correct our course of action. To move toward this ideal world, we wanted to experiment - particularly with delivering features that were built around outcome-based goals.
With this mentality, I embarked on a mission with fellow Gusto engineer Alyssa Hester to find and complete an experiment solely focused on customer outcomes. In this first post of this series, we’ll talk about how Alyssa and I found an outcome-based experiment and defined a realistic minimum lovable product (MLP). In the second post, we’ll take a look at how we brought the experiment to life, building from a hardcoded feature to a fully scalable product.
Focus on outcomes over output
What do I mean when I say outcome-based experiment?
Output is the stuff you build. For example, if a team creates a new reporting feature to generate bulk reports, then your team’s output is the reporting tool itself. This could include the designs, the code, and anything else the team produced as part of this new feature.
Outcomes sometimes grow from output, but they are an entirely separate concept. Outcomes are changes in metrics and customer behavior - basically how customers respond to what we build - these are your team’s true goals. We focus on this because it tells us if customers truly find value in what we build. For example, outcomes for a new reporting tool could include increased retention, an improvement in customer satisfaction score, or an influx of customer referrals. Is your team’s actual objective to create a reporting tool or to bring more customers to your platform? The true focus of our work should be outcomes.
The first step for our project was defining the desired outcomes. With our team’s mission in mind, to help accountants and bookkeepers who I will refer to as Gusto Partners, Alyssa and I decided our goal was to create a feature that would increase the number of clients our accountants add to Gusto. We wanted to find a project that would help both Gusto and our Partners grow. We started searching for a project based on this outcome, with the intention of defining more specific metrics once a project was selected.
Learn about the customer
The engineers on our team, myself included, wanted to have a more detailed understanding of our customer. To identify a project that would increase accountant client adds, we needed to understand what motivates accountants to add clients.
In order to gain a better understanding of our customer, we relied on our customer-facing teams. I shadowed both our customer support and sales teams that specialize in working with our accountants. I was blown away by the level of knowledge they retain about each of their clients. Each sales team member could recite, from memory, a wish list of potential features for each of their clients. This experience provided me with some amazing insight into our accountants’ interactions with Gusto.
Pick an outcome-based project
After learning more about the customer, we sat down to discuss potential projects. Several ideas were turned away for being too focused on output, instead of outcomes. Some ideas were based around outcomes, but not accountant-based outcomes.
Our Head of Accountants and API Engineering, Jeremy Thomas, came up with the final project idea - building a way to connect small businesses on Gusto that don’t currently work with an accountant or bookkeeper with our Gusto Partners.
We have a large pool of trusted accounting firms who are always looking for new clients, and thankfully, Gusto is in a position to help them with that. At Gusto, we serve more than 100,000 business owners. From our research, we know that these businesses are 80% more likely to be in business in five years if they are working directly with an accountant.
By capitalizing on our ability to recommend these trusted accounting firms to small businesses who would benefit from their services, we can help both our small business customers and partners achieve their desired business outcomes — a win win! We decided to bridge the gap between the two parties with a recommendation engine that would generate custom accounting firm recommendations for our small business customers.
In terms of outcomes, our hypothesis was that:
- Bringing more clients to our Gusto Partners would benefit our Gusto small businesses.
- Bringing more clients to our Gusto Partners would encourage them to bring more small business clients to Gusto.
With a project directive in hand, it was time to get to work. As we started to dig in, it became clear that the scope of this project could grow much larger than the four weeks we had left. Design alone could take several weeks. Meanwhile, the list of potential features was growing larger every day. When evaluating new features, we believed many would improve the success of our project, but we almost never had the time to build them.
As a team, we needed to shift our mindset in order to launch an experiment within a reasonable timeframe. We needed to move away from thinking this experiment needed to be perfect before launch. We needed to throw away our perfection mentality.
Escaping our perfection mentality
As Gusto has grown, so has the impact of every new feature we ship. New features are instantly available to users. Every Gusto engineer has immediate access to the new code merged into our mainline branch. With so many eyes on every new change, it is easy to believe that every update or code release needs to be perfect.
Having a perfection mentality means that a feature is not good enough until it is absolutely flawless. Here are a few things “perfection” could mean in software:
- Design that has been fully user-tested
- Code that solves for every potential edge case
- Software that is fully scalable
- Buy-in for this new feature from every department
At face value, these all seem like very valuable qualities to have in a project about to launch. Of course, designs that have been user tested are more likely to be successful. As software engineers, we want to solve for the edge cases. We want our code to be scalable. We want to ensure the work we are doing will not have any negative impact on other projects at the company.
However, these benefits come at a cost - an actual dollar value cost of time spent by our team and a lost benefit cost to our customers. Product managers, engineers, and designers are spending time perfecting these features when we could be launching a more simplified project and collecting quick feedback from real users. In other words, we could be extreme in shortening our iteration cycles. Users could be benefiting from a simplified version of this feature sooner, instead of waiting for something “perfect”.
With four weeks left to ship, we did not have time to fall into this perfectionist mindset. We needed to define a project that would benefit our customers and that we could realistically ship within the given timeframe.
Defining the most basic MLP
We brought on a designer, data scientist Becca Carter, and product manager Ankita Verma to help us draw up the simplest minimum lovable product (MLP) possible.
This is what we abandoned along with our perfection mentality:
- Design that was tested by real users
- Dynamically creating recommendations
- A user-tested recommendation algorithm that would provide the “perfect” recommended accounting firm to each company
- Any additional features
- Buy-in from marketing, sales, and business departments before starting work
With our focus on getting this feature out within a reasonable time frame, this is what defined our MLP:
- A design we sketched out on a whiteboard in one hour
- A plan for the data science team to select several thousand small business owners for whom they could “manually” generate accounting firm recommendations
- A recommendation algorithm based on our best-guess of what companies care about when looking for an accounting firm
- NO additional features, just make the recommendation
- A weekly project update via email to keep any interested parties up to date on our progress
The point of this project was to determine if a recommendation engine could encourage Gusto Partners to bring more clients to our platform while also helping our Gusto businesses find accountants. We wanted to answer the question - should we continue to invest time in this feature? We decided to launch something heavily manual that would provide us with enough data to determine whether we should build a fully scalable recommendation engine.
Get Started
We finally had our project, and it was time to get to work! Alyssa and I were given six weeks to complete this initial experiment and we had already spent three of those weeks working to identify and define the project. Now that we had an MLP, we needed to start building. In the final part of this series, we will walk through the process of taking an experimental feature from MLP to a full-fledged product.
