Rainforest QA vs. outsourcing QA engineering

Costs

Budget-friendly

Outsourcing: ✅ Yes

Outsourcing is designed to be a more affordable alternative to hiring.

We surveyed software engineers at 19 startups that have outsourced QA, and they characterized outsourcing as a flexible way to get more testing done at a relatively low cost.

Rainforest: ✅ Yes

Rainforest aims to give you many of the benefits of a full-time hire, at a much lower cost.

Rainforest's annual plans start at a of the cost of hiring an experienced QA engineer in the U.S. — even when your plan includes a dedicated Test Manager who handles 100 percent of test writing and maintenance for your team.

Low financial risk

Outsourcing: ✅＊ Yes, with caveats

Relative to hiring a full-time QA engineer, QA outsourcing presents a lower financial risk in that you’re risking less money to try them out.

But, to our knowledge, very few outsourcing companies (if any) offer any sort of money-back guarantee. If you’re not happy with the services, there’s no certainty you can recoup the money you’ve spent.

Rainforest: ✅ Yes

Rainforest offers a 60-day money-back guarantee any time we add a Test Manager to your team.

Before you sign up for a plan, we’ll agree on your measurable success criteria for your first 60 days with us. If we don’t meet these goals within 60 days, you can ask for a refund. We’ll give you your money back, no questions asked.

Includes tooling & infrastructure

Outsourcing: ? It depends

When you outsource a QA engineer, you’ll often need to provision and manage the tools and infrastructure they’ll use. It depends on which outsourcing firm you hire — only some of them provide tooling in addition to services.

For example, you might need:

• Plugins to extend the functionality of the open-source testing framework (e.g., for visual regression testing, detailed test results, or test recording).
• Real or virtual machines on which to run your tests — whether you use your own machines or a third-party “test grid” like BrowserStack.
• Software to manage your test suite.

Rainforest: ✅ Yes

Rainforest includes all the tooling you need for the entire testing workflow.

Within the platform, you can:

• Create, edit, and organize your automated tests.
• Run hundreds of tests in parallel on our cloud of Windows and macOS virtual machines — no configuration required.
• Get detailed test results, including debugging steps, video recordings, and http and browser logs.

Velocity

Fast time to initial results

Outsourcing: ✅＊ Yes, with caveats

If you can quickly find the right outsourcing partner, you might get contributors added to your account who can provide value relatively promptly. But every outsourcer has a different onboarding process — some are more efficient than others.

Rainforest: ✅ Yes

Rainforest is designed to help you move fast. We can have your team in place within days of signing a deal. Usually, your Rainforest Test Manager can start creating automated test coverage on their first day, getting your smoke suite up and running within a week.

Fast test creation & maintenance

Outsourcing: ❌ No

Creating and maintaining tests in open source automation frameworks — as outsourced QA engineers do — is notoriously painful and slow, especially for a software team trying to move fast. A person with the right technical skills has to dig around in and manipulate code, often bottlenecking the release pipeline.

And, at least so far, AI doesn't appear to help speed things up.

The more complex your app becomes, the more someone needs to investigate and then update your failing tests.

Rainforest: ✅ Yes

The Rainforest platform is both intuitive and no-code, so your Test Manager (or anyone else on your team) can quickly create and maintain tests up to 3x faster than a QA engineer using open source. So you can keep the release pipeline moving.

When test steps are in plain English, it's easy to quickly interpret and edit them.

Plus, Rainforest’s AI can automatically update — or “heal” — test steps when you make intended changes to your app, which makes the maintenance burden even lighter.

Fast test execution

Outsourcing: ✅＊ Yes, with caveats

Automated tests are inherently faster than manual tests. But to unlock the true speed potential of automated testing, you’ll want to run your tests in parallel (i.e., simultaneously).

If you want to execute tests in parallel with an open source framework (like the ones outsourced QA engineers use), you usually either need to provision and configure your own machines, or pay for and configure a 3rd-party test grid like BrowserStack. (QA Wolf is an exception — when you use their QA engineering services, you can run unlimited parallel tests on their infrastructure.)

The more tests you want to run in parallel, the more complex — or expensive — it gets. As of May, 2024, BrowserStack will charge you $649 per month (paid annually) if you want to run just five desktop browser tests at a time, compared to just $129 for running a single test at a time.

Rainforest: ✅ Yes

With Rainforest, you can run hundreds of tests in parallel on our cloud-based infrastructure without having to do any configuration. And we don’t charge extra to run tests massively in parallel — it’s included in every plan.

Our customers get automated test suite results in about four minutes, on average.

Efficient communication

Outsourcing: ❌ No

In our survey of 19 startups that have outsourced QA, software engineers revealed their biggest challenge with QA outsourcing: communication issues.

The survey respondents cited two specific issues around communication: working across distant time zones and language barriers.

Some software teams value around-the-clock testing coverage that comes with having testers spread across different time zones. But most startups don’t like the negative impacts to velocity that come with overseas QA partners. When you have to wait overnight for a response to a request or question, it puts a serious bottleneck in the development process.

And language barriers can make the problem even worse. What if the response you get from your outsourced QA contributors isn’t clear, or maybe they didn’t understand your request or question in the first place? (A survey respondent described it as “a breakdown in understanding.”) Then you have to wait yet another day ‌for a round of back-and-forth to complete.

Rainforest: ✅ Yes

We’ve set up our solution to emulate the benefits of a full-time hire, making it fast and convenient for your team to work with our contributors.

The Rainforest Test Manager assigned to your account speaks fluent English, works in or near your time zone, embeds in your preferred project and communications tools (e.g., Slack, Teams, Linear, Jira), and can join your planning meetings. They have the language skills, context, and availability to make communication smooth and efficient. You just tell them what you want tested, and they take care of the rest.

“The flow has been really great. [The Test Managers] are really proactive, they triage all the errors and things that happen in tests, they fix up the tests, they write new tests, and they ask crazy-insightful questions on the product. They’re really polite, cool people. It’s a very similar style to people internally.”

Tyrone Erasmus, Co-founder / CTO at Push Security

Confidence in testing

Reliable service quality

Outsourcing: ❌ No

According to our survey, other than communication issues, the other big challenge with QA outsourcing is the poor quality of services.

Since one of the main benefits of outsourcing is the relatively low price, outsourcing firms don’t necessarily hire the most skilled or experienced people, but instead the most cost-efficient.

This might explain why our survey respondents described service quality issues including “poor work quality,” “a problem meeting deadlines,” and “reporting things that weren’t actually issues.”

Rainforest: ✅ Yes

All Rainforest Test Managers have been with us since at least 2017 and undergo regular training and evaluations. And they’re all consistently highly rated by customers — otherwise they wouldn’t still be with us.

Software teams working with Rainforest can have the confidence that they’re working with a team with the skills and experience needed to meet high expectations.

“We really didn’t have to put all that much effort into onboarding them. I want to give kudos to our Test Managers for jumping in and more or less figuring the app out on their own. We gave them a long list of things to test. They dove right in and figured it out based on our four-year-old video tutorials.”

Dale Smith, CTO at PayHOA

Understands your product & priorities

Outsourcing: ❌ No

Another issue reported in our survey: these distant QA contributors don’t tend to join the software team’s meetings or embed in the team’s workflows, so they don’t have the context necessary to meaningfully understand how the team’s product works, let alone the team’s priorities.

That means outsourced contributors require more onboarding, training, oversight, and corrections. And the work itself isn’t as useful.

You see this critique reflected when you ask surveyed engineers to describe the benefits of hiring vs. outsourcing QA: they wish outsourced QA personnel had more context.

Rainforest: ✅ Yes

When you opt to include test management services in your Rainforest plan, we’ll dedicate one or more Test Managers to your account. They’ll embed in your workflows and join your project and comms tools (e.g., Slack, Teams, Linear, Jira) and planning meetings, deeply learning your product and priorities.

Just like full-time hires, as they learn, your Rainforest team members will contribute even more effectively and efficiently.

“The fact that our Test Manager is very specialized, very efficient, knows Rainforest incredibly well, and knows our product incredibly well means she can just focus on this one area and do an incredibly good job of it.”

Robert Guillaume, QA Manager at YNAB

Tests the actual user experience

Outsourcing: ❌ No

Outsourced QA engineers use open-source frameworks to automate end-to-end tests. These frameworks interact with and evaluate the DOM, or the code behind the visual layer of a web app. That is, they test a proxy for the user experience, but not the user experience itself.

Rainforest: ✅ Yes

Rainforest takes a visual-first approach to interacting with and evaluating your app, so it tests exactly what your users will experience.

Plus, our approach means you can test anything that appears on a Windows or Mac virtual machine, not just what appears in the browser. For our customers, common use cases outside the browser window include testing browser extensions and file uploads and downloads.

No black boxes

Outsourcing: ❌ No

An outsourced QA engineer has specific technical expertise to create and maintain automated tests with code. If you don’t have the right skills, you can’t evaluate the quality of their work.

Rainforest: ✅ Yes

Because all Rainforest tests are no-code and readable in plain English, anyone on the team can quickly validate test functionality and coverage.

Test automation for native mobile apps

Outsourcing: ✅ Yes

An outsourced QA engineer can use an open-source framework like Appium to automate tests for native mobile apps on iOS and Android.

Rainforest: ❌ No

Rainforest has been optimized to test web applications — it doesn’t automate tests for native mobile apps. (But we do offer on-demand crowd testing services for native mobile apps that return scripted test results in under 30 minutes, on average.)  

Strategic benefits

Establishes QA processes & best practices

Outsourcing: ❌ No

Typically, outsourced QA contributors will help you execute your testing plan, but won’t help you design a QA strategy. Unless you also hire a QA consultant, you’re left to figure out processes and best practices on your own.

Rainforest: ✅ Yes

Every Rainforest customer gets a Customer Service Manager (CSM) dedicated to their account. These CSMs aren’t just experts in our services and platform; they’re also QA experts who have helped hundreds of startups develop their software QA strategies.

As a consultative partner, your CSM will work with you to design processes and practices for your team that cultivate a culture of quality.

Scalable

Outsourcing: ✅ Yes

QA outsourcing is designed to allow you to add more resources as you need them.

Rainforest: ✅ Yes

Similarly, Rainforest has the resources to give you all the functional QA testing and services you need, so we can scale up with you.

Secure

Outsourcing: ? It depends

When we surveyed 42 startup software teams about their top concerns or hesitations about outsourcing QA, “security” was the most frequent response after “communication,” “quality of work,” and “understanding our product.”

Specifically, these teams worried about the security of their data and intellectual property in the hands of distant, overseas contributors who have no particular loyalty to them, the customer.

Rainforest: ✅ Yes

Rainforest is SOC 2 Type II certified, HIPAA compliant, and ISO 27002 compliant. We’ve implemented a number of measures and systems to make sure your data is safe both on our platform and with our personnel.

This includes having each of our Test Managers sign a non-disclosure agreement (NDA) to protect any information they learn about your products or company.

You can learn more on our Security page.

No single point of failure risk

Outsourcing: ? It depends

If you’ve outsourced a single QA engineer (e.g., from a site like Upwork), they can represent a single point of failure, particularly if they’re your only QA contributor. If they leave, don’t work out, or are otherwise unavailable, it can bottleneck the entire QA and testing process until you replace them.

On the other hand, if you go with an outsourcing firm with many resources available to you, it largely removes this type of risk.

Rainforest: ✅ Yes

Rainforest has the personnel and systems in place to guarantee you’ll never be left in a lurch.

Other considerations

• Browser support: The Rainforest platform supports multiple versions of all major browsers.
• Integrations: In addition to integrations with JIRA, Slack, and Microsoft teams, Rainforest has an API and a CLI so you can integrate your automated tests into any CI/CD pipeline.
• Exploratory testing: Like outsourcers who offer manual testing, Rainforest offers exploratory testing services to find the bugs and issues off the “happy paths” covered by ‌your automated tests, giving you even more confidence in the quality of your app.