“Our engineers can work on developing new features and not worry about regressions on the old ones, because our Rainforest tests will catch them.”
- 25:1 QA ratio: Bluecore’s 25-person UI team is supported by 1 full-time QA manager
- 1x daily deployments: Bluecore’s UI team ships new code on a daily basis
- 1032 hours saved/mo: Bluecore saves approximately 1032 hours each month on manual testing with Rainforest
- Uses Rainforest to bridge the gap between manually testing new features and automating older ones.
Bluecore’s real-time interaction management platform for commerce is designed to simplify the process of synthesizing behavioral data into actionable insights. It’s no surprise that delivering an excellent customer experience is critical to their success. Bluecore turned to Rainforest earlier this year for crowdsourced testing to achieve their goal of balancing engineering hours with adequate test coverage that fits with their deployment strategy.
Bluecore now runs their Rainforest regression tests before each daily deployment. With Rainforest, Bluecore’s UI team is leveraging crowdsourced testing to keep QA coverage high and customers happy as they roll out new features. “Our engineers can work on developing new features and not worry about regressions on the old ones, because our Rainforest tests will catch them,” says Mahmoud Arram, cofounder and CTO of Bluecore.
Adapting to Changing UI Testing Needs
When Bluecore accelerated building a user interface earlier this year, they realized that the move would require significant changes in their testing strategy. “We had a tradition of highly automated unit tests, and we had thousands of them,” says Mahmoud. “But once we reached product-market fit we turned our focus toward building a world-class user interface to support the underlying technology. The iteration happens a little faster. UI interactions happen asynchronously. A lot of our traditional infrastructure for automated unit tests and code testing did not really apply to UI testing.”
With a small UI team to support the product and limited in-house QA resources, Mahmoud wanted to make their testing processes as efficient as possible. “We wanted to give the UI engineers as much bandwidth as possible,” says Mahmoud. Bluecore’s goal was keep the team focused on developing a great user interface, without sacrificing testing processes that were important to keeping quality high.
Adopting a Progressive Crowdsourced QA Strategy in Development
To maximize the benefits of crowdsourced testing, Bluecore has integrated Rainforest into their processes early in development. Rainforest tests are written alongside simple automated tests so that coverage scales as the feature takes shape. “When the team starts building their features, they’re writing very high-level tests. On the server side, they’re writing automated tests. On the UI side, they’re building a simple Rainforest test that exercises the happy path of the feature,” says Mahmoud. “This way changing the code and then changing the test isn’t so arduous.”
By the time the feature is introduced to a limited number of customers, testing is expanded to better reflect how customers use the feature. “This is where we start writing deeper tests, looking at the error paths, and exploring the different nuances of the features. “The idea is the that every team makes the determination as to how much testing they need based on the criticality and adoption rates of a feature” says Mahmoud.
Crowdsourcing manual tests before a feature is ready for fully-implemented automation allows Bluecore’s engineering team to spend less time writing, maintaining and running QA tests and more time working on their product. Bluecore has now brought on their first QA engineer to facilitate moving tests through their evolution.
Leveraging Rainforest to Bridge the Gap to Testing Automation
An additional benefit of Rainforest tests is the ease with which they can be transitioned into automated QA tests. Bluecore still relies heavily on automation to streamline testing for less dynamic features in production. “Once a feature gets into general availability, our rule is that 80% of its tests must be fully automated,” says Mahmoud.
Because Rainforest tests provide all the context that crowdsourced testers need to test without previous knowledge of the product, they provide a solid foundation for automation scripting. As a result, Bluecore is able to outsource automated test scripting easily and convert Rainforest tests to Selenium tests.
Building a Better Customer Experience with Rainforest
Since implementing Rainforest QA, Bluecore has seen marked improvements in quality metrics: the number of bugs filed internally and in the field, and the amount of test coverage per feature. Mahmoud explained, “Rainforest has shortened our iteration cycle, as fewer bugs made it to the support and client success teams and eventually to our customers.”
Using Rainforest has allowed Bluecore’s team to improve test coverage and identify regressions before they ever reach their customers. “We went in and implemented the Rainforest tests and eventually stopped hearing from our Customer Success team. That’s a measure of success right here. We make sure that the whole suite of tests runs before we deploy, to ensure that there are no regressions, says Mahmoud.”