Back to all case studies
The situation: With a small QA team and a rapidly-evolving, multi-platform product, YNAB struggled to keep up with end-to-end testing — especially with the burden of test maintenance.
The result: YNAB enlisted a Rainforest test manager to handle all test creation and maintenance, allowing their internal team to focus on higher-value QA tasks, ensuring efficient test coverage, and avoiding the costs of hiring additional staff.
For anyone trying to move extremely quickly and has a small team and is trying to do really good QA with as few resources as possible, having a test manager is extremely valuable.
YNAB, initially released in 2004, is a personal finance app that guides spending decisions through a simple set of habits. Since the company deals in personal finance, it's more sensitive than average to bugs getting into the wild.
As Robert Guillaume, QA Manager at YNAB, puts it:
“People are making real-world decisions based on the data that they're seeing in their budget. If you have a bug and they think they have more money than they do, it has real-life consequences.”
Today, YNAB has seven QA headcount and around 40 engineers working on its web, tablet, and mobile apps. But back in 2016, there was a single, part-time QA person supporting a team of 20 engineers consistently releasing new features and updates.
That one person couldn’t perform the extensive manual testing required to support a personal finance application, especially not at the speed the company wanted.
To help close the gap in QA coverage without having to spend big on hiring, YNAB contracted with Rainforest QA to increase manual testing capacity. With Rainforest, YNAB gained access to a worldwide community of expert QA testers would could execute manual test suites, on-demand, 24x7. Because groups of testers would run the different tests from a suite at the same time in parallel, YNAB could get results back in just 20 minutes.
Rainforest’s tester community solved the testing gap, but a rapidly-evolving product that supported several different platforms required much more test creation and maintenance than just one part-time QA person could handle.
In 2019, YNAB brought on a Rainforest QA Concierge (or “test manager,” as YNAB calls it) to help create and update scripts for manual tests.
“One QA person and our test manager were able to maintain test coverage, which was an extremely difficult task for an app across iPhone, iPad, Android, and web with extensive test cases. That would not have been possible without the test manager.”
More recently, the test manager handled all of the conversion of YNAB’s manual test suite into automated tests on the Rainforest no-code platform.
Today, the Rainforest test manager maintains the entire YNAB test suite herself, requiring zero work by the full-time members of the YNAB team to create or maintain their tests.
For new test coverage, the team provides (1) a video recording of the intended functionalities of new features and (2) a list of prioritized testing scenarios. Using these inputs and her historical context on the app, the test manager creates the necessary tests in the Rainforest platform.
“We basically communicate what's important to us and the details are taken care of by the test manager. It allows us to avoid having to create any tests ourselves.”
When there are test failures, the test manager is solely responsible for triaging, identifying potential bugs, and updating any tests that need to reflect intended changes to the app. She summarizes her findings to the team via Asana, where the team manages all its work.
“We know exactly what to dig into. It's a huge time saver. It saves us from going through every single failure every time.”
The same Rainforest test manager has been embedded with the YNAB team since 2019, which has given her time to prove her reliability and to develop a deep understanding of YNAB’s apps.
“The fact that she's very specialized, very efficient, knows Rainforest incredibly well, and knows our product incredibly well means she can just focus on this one area and do an incredibly good job of it.”
The test manager being reliable and effective means Robert has the peace of mind that the time-consuming-but-critical maintenance of their test suite is in good hands.
“It seems like she's always available, always doing a really good job. We've learned to really rely on her without having to worry about it all the time.”
Since they don’t have to worry about test creation and maintenance, the internal team at YNAB can focus on higher-value QA work like testing strategy and exploratory testing.
“There's less value from a QA analyst writing the test than there is actually thinking through what we test. And [the test manager] gives the QA analysts more time to do exploratory testing. We rely on that a lot and believe it’s one of the things where the analysts offer the most value within the product team.”
Even considering the test manager’s value to the team – including an efficiently- and reliably-maintained test suite, peace of mind for Robert, and time for the rest of the QA team to spend on higher-value tasks – using a Rainforest test manager has still been dramatically less expensive for YNAB than hiring:
“We'd probably have to hire two or three more people to manage all the tests ourselves.”
