No QA team? 5 ways to automate your tests and what to consider

Is your startup operating with no QA team and trying to figure out your options for improving software quality?

Perhaps you’re proactively thinking about your QA process before quality becomes a problem. Or maybe quality issues have forced you to finally address your QA process (or lack thereof).

Either way, end-to-end (e2e) automated software testing is always part of the answer. Compared to manual testing, it’s cheaper and faster to execute, more reliable (humans get distracted and bored!), and is a better fit with modern devops pipelines featuring CI/CD.

But when you don’t have a dedicated QA team, how do you make the transition to automation?

You have five different options, each with its own tradeoffs:

• Have your developers adopt open-source automation
• Hire a QA engineer
• Outsource QA
• Use low-code/no-code tools
• Use AI-powered tools

In this piece, we’re going to cover the pros and cons of each of these options so you can make informed decisions for your startup and get to a point where you’re releasing fast, with confidence.

The information we’ll share is based on a decade of our team’s experience in this market, plus surveys we’ve run and hundreds of conversations we’ve had with startup engineering leaders.

Full disclosure: we sell a QA solution. But we’re going to present it (and the other options) with as much objectivity as we can so you can make a smart decision involving all the tradeoffs that matter most to you.

Have your developers adopt open source automation

One way to start automating tests is to have your devs do all QA. In addition to unit testing, they write and maintain their own e2e tests in an open source framework like Selenium, Cypress, or Playwright.

This approach is easier on your budget than hiring a full-time specialist to manage these tests, but the costs show up in other ways. Specifically, you’ll pay for this approach with less productivity from your expensive software developers and lower velocity in your release pipeline.

It’s painful and time-consuming for developers who have to dig around in automation code to maintain e2e tests instead of doing what they’re incentivized to do, which is shipping code and new features.

And they’ll be doing a lot of test maintenance, because tests in these frameworks tend to be quite brittle. That means even small, non-functional changes to your app can break things in your test suite, requiring investigation and maintenance.

Because maintaining these tests is such a monotonous drag, software developers often skimp on keeping their e2e test suites up-to-date in favor of delivering more releases, more quickly. Maintaining tests is a serious time commitment, and startup software teams have to ruthlessly prioritize how they spend their time.

“We have some Selenium tests. I’m sure you guys are very familiar with just how painful and time-consuming it is to build Selenium tests. And so we don’t build nearly enough of them.”

Startup CTO

In fact, based on our survey of almost 100 startup software teams in the U.S. and Canada, a full 36% indicated that they’re failing to keep their automated tests updated.

These teams spend hours every week writing and updating automated tests, but still fail to create something that can give them confidence that what they’re shipping is high-quality.

Of the surveyed teams who do keep their test suites up-to-date, well over half of them spend at least 11-20 hours per week on test maintenance. About a quarter of these teams spend at least 31 hours per week on maintaining tests, and 15 percent spend at least 40 hours per week — the equivalent of a full-time hire.

You’ll also need to consider the financial and productivity costs of the tooling and infrastructure you’ll need to provision and manage.

• 3rd-party plugins to extend functionality (e.g., to get more detailed test results or to add visual validation to your tests)
• Test management software to manage and organize your test suite(s)
• Machines on which to run your tests, whether the machines are real or virtual. Want to run tests in parallel? You’ll need more machines. Manage them yourself or pay for a 3rd-party test grid like BrowserStack or Sauce Labs.

Tl;dr

There are startups who have their developers using open source test automation frameworks and it works for them to one extent or another. The tradeoffs in productivity and velocity (and often, developer satisfaction) are, to them, worth an approach where development team members are product owners who truly own responsibility for the quality outcomes of their work.

But what we’ve found is that this approach tends to be an anti-pattern for startups interested in moving fast and keeping their engineers focused on delivering code.

If continuous delivery is your goal, you should consider one of your other options.

Hire a QA engineer

Hiring a QA engineer means adding a full-time headcount who applies specialized skills to writing tests and maintaining tests in an open-source framework. Hiring this specialized contributor allows your development team to execute faster, without the drag of e2e test management.

Plus, if you can find an experienced QA person as your first hire, they can implement a testing strategy and process, help you define quality metrics, and otherwise help you implement best-practice methodologies.

A good candidate integrates tightly into your dev team’s workflows to keep releases moving as smoothly and quickly as possible. They’ll develop a deep knowledge of your product, which allows them to be more efficient and effective. And, as part of the team, they’ll have a sense of ownership and accountability that’d be difficult to find in outsourced headcount.

In short, finding the right QA engineering hire for your team can be a boon for both product quality and your peace of mind.

But can you afford it?

The main drawback of hiring a QA engineer is the cost of your time, energy, and money.

Finding good people is hard — in QA or otherwise. In a survey of startup software engineers we ran, the number one response to “​​What are the main challenges when it comes to hiring QA?” was, “Finding good candidates.”

Hiring can eat up a lot of time and energy for any hiring manager. ​​A typical process involves writing and posting job descriptions, screening candidates, interviewing candidates, negotiating, onboarding, training, and management. Once you decide to add headcount, it can take months before someone’s adding value in that position.

Plus, managing a QA person creates a new type of labor for engineering leaders. For them, managing contributors from a non-engineering discipline requires a different set of knowledge and a different way of thinking. QA contributors have different goals, viewpoints, and communication styles.

​​If you do manage to find a good hire, you’ll pay for the privilege. Publicly-available salary databases show that hiring an experienced QA engineer in the U.S. tends to cost north of $100K. 

If a QA hire doesn’t work out, that’s an expensive mistake.

And that’s not the only risk when you hire QA. There’s also single point of failure risk, where the success of an entire function — in this case, quality assurance — depends on the continued contributions of a single person. If you lose your QA engineer, your test suite doesn’t get updated and you lose whatever confidence you had in your test coverage.

Finally, you still have to pay the time, money, and velocity costs of using open source automation frameworks:

• The costs of inevitable bottlenecks caused by test maintenance. (Remember how some startups spend 40+ hours per week keeping their automated test suites up-to-date?)
• The costs of provisioning and managing additional tooling and infrastructure.

Tl;dr

If you’ve got the budget and are comfortable with the risks, hiring a QA engineer to automate your tests and lead your QA process can be a winning move.

If you don’t have the budget but still want to enjoy the benefits of a full-time hire, you could consider an offshore hire. You’ll save money, but could introduce productivity challenges stemming from your team working across distant time zones and potential language barriers.

The difference between hiring QA and hiring in some other categories — like say, HR or marketing — is that the market is full of practical alternatives for QA. (Specifically, the ones covered in this post.) There’s less need to hire to solve a problem in QA than there is in other disciplines.

So, if you don’t have the budget, time, and/or the risk appetite for hiring, keep reading to learn about your other options.

Outsource QA

Outsourcing could mean hiring a firm like Applause or QA Wolf, or it could mean hiring individual contractors on Fiverr, Upwork, or similar.

We surveyed software engineers at startups that have outsourced QA, and they characterized outsourcing as a flexible and scalable way to get more testing done at an affordable cost (while freeing up coders to focus on shipping).

While outsourcing is a faster and cheaper way to scale up your QA resources — relative to making a full-time hire — it comes with some notable drawbacks.

The same survey also revealed the biggest challenges with QA outsourcing: communication and the quality of services delivered.

The survey respondents cited two issues around communication: working across distant time zones and language barriers.

Some software teams value around-the-clock testing coverage that comes with having testers spread across different time zones. But most startups don’t like the negative impacts to velocity that come with overseas QA partners. When you have to wait overnight for a response to a request or question, it puts a serious bottleneck in the development process.

And language barriers can make the problem even worse. What if the response you get from your outsourced QA contributors isn’t clear, or maybe they didn’t understand your request or question in the first place? (A survey respondent described it as “a breakdown in understanding.”) Then you have to wait yet another day ‌for a round of back-and-forth to complete.

Meanwhile, since one of the main value propositions of outsourcing is the relatively low price, outsourcing firms don’t necessarily hire the most skilled or experienced people, but instead the most cost-efficient.

This might explain why our survey respondents described service quality issues including, “poor work quality,” “a problem meeting deadlines,” and “reporting things that weren’t actually issues.”

That last critique points to another issue reported in the survey: these distant QA contributors don’t tend to join the software team’s meetings or embed in the team’s workflows, so they don’t have the context necessary to meaningfully understand how the team’s product works, let alone the team’s priorities.

That means training and onboarding take longer, ‌contributors require more oversight and corrections, and the work itself isn’t as useful.

You see these critiques — and others, including security issues — reflected when you ask these surveyed engineers to describe the benefits of hiring QA vs. outsourcing.

Tl;dr

Outsourcing is an affordable and flexible way to ramp up your QA resources, at least relative to hiring full-time contributors. The trick is to find outsourcing partners who won’t undermine your velocity and confidence with communication and service-quality issues.

Also consider that if you outsource your test automation, the outsourced contributors will write and maintain your automated tests using an open-source framework. You’ll still need to deal with maintenance bottlenecks and provision and manage any plugins, test management tools, and the machines on which to run your tests.

Use a no-/low-code test automation tool

We’ve done a whole breakdown of the important things you should know about no-code test automation tools, but here are the highlights:

The promise of no- and low-code tools is that anyone on your team can create and automate tests without any technical expertise, at a low cost. You don’t need to hire an expensive QA test engineer to automate your tests. Some no-/low-code solutions even offer entry-level free plans, and for a few hundred dollars a month, you can run your tests in parallel on cloud machines provided by the tool vendor. Some of the tools offer things beyond functional testing, like API testing.

These tools can seem pretty compelling to engineering leaders who don’t have the budget for hiring or outsourcing, don’t want their software developers to get bogged down in open-source frameworks, and don’t want to deal with managing test infrastructure.

They think, “Our devs can use this and move a lot faster than they can with open-source tools because it’s low-code. Maybe we can even have our non-technical product managers help out.”

We know this because our automation platform started its life as a no-code tool with a free, self-serve plan.

And it’s true: broadly speaking, test writing and maintenance tends to be easier in no-/low-code tools, relative to working in an open-source framework. (Assuming the tool has been designed to be intuitive and easy to use — this isn’t the case for a number of low-code tools.)

The main gotcha! of these tools is that relatively easier test maintenance doesn’t mean maintenance is suddenly easy.

Our no-code tool, Rainforest, is very intuitive and easy to use — just ask our customers. But even with our tool — at least, before we implemented AI and other enhancements (more on that in the last section, below) — we found that for teams whose apps are consistently evolving, test maintenance becomes a time-consuming burden. It’s a particular burden for agile teams practicing continuous delivery.

When teams adopt no-/low-code tools, they tend to underestimate the costs of maintenance. They think they’re saving money by not hiring QA engineers, but they end up having to hire QA headcount to own test automation in these tools and to free up their development teams to focus on delivering code.

There are two other notable tradeoffs you have to make with no-/low-code tools:

First, some tools claim to be “codeless” but still require you to code. Specifically, for any test step requiring even a bit of complexity, you’ll need to use custom JavaScript. This excludes non-technical users from creating and maintaining anything but simple tests.

Two, most no-/low-code tools — just like all open-source frameworks — interact with and evaluate what happens in the DOM, the code behind what happens in the user interface (UI) of your web app.

This approach ignores the visual appearance of what happens in the UI your app. That is, the approach of these tools ignores what your end users will experience in your app. Instead, they test a proxy for the user experience (i.e., the behind-the-scenes code), which often fails to accurately represent what users will see.  ​​

Some tools offer visual validation through a 3rd-party plugin like Applitools, but in other cases you’re stuck with DOM-based software testing.

Tl;dr

If you don’t have the budget for hiring or outsourcing, no-code tooling can seem appealing. But if you’ve never used these tools before, you’ll almost certainly underestimate the amount of time-consuming test maintenance your team will need to do to keep your automated regression suite up-to-date.

So it’s worth exploring in the next section how generative AI is making these tools even more useful.

Use an AI-powered tool

Many QA tool vendors use machine learning in various ways to make their tools incrementally more performant.

But in this case, we’re talking about tools that use generative AI (genAI) to make creating and maintaining tests even easier than it is in other tools and frameworks.

Like no-/low-code tools, they have the benefit of being much more affordable than hiring or outsourcing, but generally present less of a burden on your team to keep your test suite updated.

Some AI-powered tools like Rainforest remove more of the maintenance burden than others due to how deeply and thoughtfully they’ve integrated the benefits of generative AI.

For example, a number of AI-powered QA tools (e.g., Rainforest, Katalon, and Reflect) simplify test creation by automatically transforming simple, plain-English prompts into test scripts.

While some of these tools require you to describe test flows step-by-step using detailed prompts, Rainforest allows you to create a series of steps based on a single prompt.

For instance, to test your app’s sign-up flow, instead of having to create step-by-step instructions with prompts like these:

1. Click on the Sign Up for Free link
2. Fill the email field with test@test.com
3. Fill the password field with 12345
4. Click the Sign Up button

…in Rainforest you can simply provide a prompt of, “Sign up for an account with dummy data,” and our AI will intelligently create all the necessary steps.

Second, using genAI, test steps in Rainforest can automatically update themselves — or “self-heal” — to reflect intended changes to your app. 

In a tool like Mabl, only individual test steps can self-heal. To our knowledge, Rainforest is the only AI test automation platform where a series of test steps created with a genAI prompt can automatically update themselves when your app changes. 

Revisiting the “sign up” example above: if your signup flow ever changes, the AI will update the steps based on your original prompt, adding, removing, or changing steps as necessary.

You can see Rainforest self-healing in action in this video.

When you create tests, both Rainforest and Reflect use genAI to automatically generate descriptions of the app elements your tests interact with, like buttons or form fields.

When your tests run, if the system can’t find an element based on its default identifier (e.g., a screenshot or DOM selector) the system will look for the element based on its auto-generated AI description (e.g., “Pricing” roughly in the top middle of the screen).

This helps your team avoid annoying false-positive test failures and maintenance steps whenever you make minor, intentional changes to your app.

These last two AI-powered features are the game-changers when it comes to avoiding false-positive test failures and test maintenance tasks that you’d otherwise get in other tools and frameworks.

Other ways Rainforest is different

Tests focus on the user experience, not the DOM

Unlike many other tools, Rainforest takes a visual-first (not DOM-first) approach to interacting with and evaluating your app, so it tests exactly what your users will experience. We use machine learning to simulate how a human QA tester would evaluate your app — if a visual change is so small that a human QA tester wouldn’t notice or care, Rainforest will pass it, avoiding annoying false-positive test failures.

Plus, our visual-first approach means you can test anything that appears on the screen of a Windows or Mac virtual machine, not just what appears in the browser.

GenAI optimized for QA and reliability

Most genAI implementations use LLMs from a 3rd-party vendor like OpenAI. Rainforest also uses OpenAI, but we’re different in that we have over a decade’s worth of behavioral human QA testing data to make our AI agent more effective in the QA context. We also use a novel, patent-pending approach to make our AI agents more reliable.

Some test cases are too complex to describe with a simple, plain-English prompt. Sometimes it’s easier to create test steps manually. Before integrating genAI, Rainforest started its life as an intuitive, no-code tool that anyone could use without training, so it’s designed to make test creation and maintenance as easy as possible.

AI + managed services to handle all test maintenance

Our long-term vision is to use genAI and other technology to reduce test maintenance to zero. In the meantime, for scenarios that genAI currently can’t handle, we offer experienced QA test managers to take test creation and maintenance completely off your team’s plate and allow your devs to focus on shipping.

These QA professionals have each worked with us and our platform since at least 2017, work in or near your time zone, speak English, and embed in your workflows and comms tools (like Slack, Microsoft Teams, or Jira) to deeply learn your app and your priorities. You just tell them what you want tested, and they take care of the rest. They also keep an eye on the AI’s work to correct any hallucinations, giving you one less thing to worry about.

“The fact that [our test manager] is very specialized, very efficient, knows Rainforest incredibly well, and knows our product incredibly well means she can just focus on this one area and do an incredibly good job of it.”

Robert Guillaume, QA Manager at YNAB

Final thoughts

Rainforest isn’t going to be the right fit for every team looking to solve QA problems, but when you’ve got no QA team, we’ve specifically designed Rainforest to have the big upsides of your other QA options, without their big downsides.

It’s an intuitive, no-code test automation platform featuring deep genAI integration, so:

• It’s a lot more affordable than hiring
• It saves your team significant time on test creation and maintenance vs. other tools and frameworks, so your devs can focus on shipping
• It includes all the features and zero-management infrastructure you need to manage your test suites, run them massively in parallel, and get detailed test results for easy debugging

Even when you add an experienced, reliable Rainforest test manager to your account to handle all of your test creation and maintenance, it’s still less than half as expensive as hiring a QA engineer in the U.S.

We even offer exploratory testing services to find bugs off the “happy paths” covered by your automated tests.

If your startup is looking to solve its QA issues, we invite you to talk to us.