A step by step guide for how to successfully run a test fest
The work is done, your team has spent the days, weeks, months, building a new feature and it is ready to deploy! But wait - will this new feature break everything?
Feature and unit tests can only go so far. The most certain way to find weak points in a new feature is through real users putting it to the test. It is amazing what real humans (especially non-engineers), who will sit down and click all the buttons, can find. This is where test fests can help.
What is a test fest?
A test fest happens whenever a group of people come together to test a new feature. The purpose of a test fest is to find bugs, unexplored edge cases, even incorrect copy or typos, before deploying to actual users. During a test fest, your feature and code are put through the ringer. Volunteer testers will attempt to find weak spots and edge cases that may have been missed during development.
The invite list
The invite list is determined by the scope and risk of a project. Ideally every test fest would include the engineers, designer, and product manager who worked on the feature. As a feature grows more complex, the test fest invite list should also grow.
Invite folks from customer-facing roles. This could include coworkers from sales, customer care, or user research roles. These groups are closest to the customer and have unique insight into how this feature will be utilized by actual users. Consider inviting people from teams who may not interact directly with the customer but who regularly dive into customer research, such as marketing.
Including engineers, designers, and product managers from other teams who work in tangential areas will provide a more well-rounded perspective to how the feature could be used in conjunction with the rest of the application. The test fest could be an excuse to introduce these other teams to the product change.
While having a cross functional team of testers can provide a more well-rounded perspective when testing, ultimately you are more likely to find unexpected bugs with more testers. When it comes to test fests, the more the merrier!
Test fest doc
Test fests are inherently messy. The testing team may not regularly work together. Testers may have varying levels of experience operating outside of production environments or thinking in terms of edge cases. A wide range of experience can make it difficult to provide instructions that work for all test fest attendees.
A test fest doc will help keep reign in the chaos, providing clear written instructions to the testing team.
Your test fest doc has three goals:
- Introduce the new feature
- Instruct your testers on how to proceed
- Document bugs
Testers should be able to read a test fest doc and understand what the new feature does, how to start testing, and how to document any bugs they find.
Introduce the feature
Start with an overview of the feature to be released. Include any planning documents created by the team in preparation for this release, especially designs. The team’s original planning documentation can provide context to more curious testers, and spark ideas of additional areas to stress test.
It can be tempting for the over-prepared to write down a detailed list of all the ways a tester could interact with the new feature. Do not do this - less is more here. A test fest is meant to bring a wide variety of users to the feature, especially people who do not understand how the product should work. Provide too much instruction and testers may follow the directions exactly. We want testers who will stray from the happy path, using the feature as they would naturally, leading to undiscovered edge cases.
The goal here is to provide just enough instruction to get testers started while remaining vague enough to allow people to explore the feature on their own. One method for providing helpful, but vague, guidance is to replace strict directions with open ended questions. Here are some examples of transitioning from direct instructions to helpful guidance.
- Log in as an accountant ➡ What if you tried using this feature as a different type of user?
- Go to the dashboard then route to the billing page from the left-hand navigation bar ➡ Try approaching this feature from a different page.
- Login as an administrator and access this company from their entry link ➡ What if you wanted to use this feature as a Gusto admin?
Instructions
Start with the basics. Remember that not everyone has attended a test fest before. Include a short description of test fests at the top of the test fest doc for the new testers. Even for the experienced tester, it is helpful to remind the team that the goal of a test fest is to break things. Break things now so they will not break later for real life users. Set this precedent at the top of the test fest doc. Remind testers to click all the buttons, think outside the box, try to find ways to stress test this feature the team may have missed.
More than one test fest has been derailed because the organizer did not provide clear instructions on how to access a staging environment and navigate to the feature in question. This seems like a simple step, but can be surprisingly complex. Some questions to ask when setting up instructions are do my testers:
- Have access to our staging environment?
- Need to create an account? Do I have clear documentation on how to do this?
- Know how to reach my new feature?
Bug Tracking
The final, and most important, section of a test fest doc provides a way to document bugs found during testing. The goal of bug reporting is to make it easy for engineers to duplicate and fix any discovered problems after the test fest is complete.
Testers should report bugs they find directly in the test fest doc. Encourage details, screenshots, even screen recordings. At the start of a test fest this section should be incredibly short. Include an example bug to help testers get started.
Example
- [Name] Summary of what went wrong
- Steps to reproduce
- I went to this page
- Clicked this button
- Saw this thing
- What was expected vs. what happened
- Be detailed - could you repeat the bug? Does it happen for similar components?
- Include photos or video
Prep
Once the test fest doc is prepared, ask a coworker to try it out. Have this volunteer follow the document instructions, as a tester would, and ensure all critical steps are covered. If the testing team will include non-engineers, ensure this volunteer is, likewise, not an engineer. Engineers have expanded access to test environments and administrative permissions. Asking a non-engineer to try out your instructions will ensure that all testers can reach the proper staging environment and interact with the feature.
Timing and scheduling
When it comes to planning a test fest, the earlier the better. Stating the obvious, larger releases are more challenging to test than smaller releases. The more things there are to test, the less likely it is that your testers will be able to cover all edge cases. Testing in smaller batches means more focused testing, and therefore increased bug discovery.
Whenever you get the test fest on the calendar, do not forget to leave time to fix bugs discovered during the test fest. If your product needs to be deployed on Monday, do not schedule the test fest for a Friday. If this is the first time the product is in front of users, other than the engineering team, bugs will be found. Do yourself and your team a favor and plan for plenty of time to fix bugs during the work week.
Party time
Running a test fest can be stressful. I have planned several test fests where I spent an intense hour answering questions, digging into newly discovered bugs, and both worrying that testers would miss bugs and that they would find too many bugs. As the organizer, prepare for an intense hour of questions!
Remember the goal of the test fest is to find bugs, not to fix them during the meeting. Refrain from diving too far into code during a test fest and spend this designated time working with the team of testers. If a bug is found, document it and move on.
Releasing a new feature can be downright terrifying. Test fests allow teams a way to stress test their code with actual humans before it is deployed to users. Do you have any test fest tips or stories? Share them in the comments below!
