In this guide to smoke testing, we cover the different uses of the term, and how you can use this strategy to validate your SaaS product before you build it.
New SaaS entrepreneurs are tasked with an important project: make sure people actually want to buy their software before they build it. Building a SaaS MVP can range from $50,000 to $90,000 when developing a true MVP.
Unfortunately, SaaS entrepreneurs often get roped into working with agencies that specialize in building apps for established organizations.
In this case, these new startups end up sending $200,000 to $1,000,000 to build a launchable product, because the agency has convinced them to launch with more features than needed. Or the agency hasn’t talked the entrepreneur off the ledge of feature overwhelm.
In order to save money, SaaS entrepreneurs do best when taking some of the effort into their own hands. They can smoke test their product idea, or work with a company who offers smoke testing as a service before committing to building the entire product.
What is smoke testing?
In software, smoke testing can refer to two different things. For QA, it means an initial test of critical functionality prior to in-depth QA. And in product development, it refers to building a landing page for a product concept to validate it. This is done to test the ease of acquiring waitlist sign ups before actually developing the app.
In either case, smoke testing refers to an initial test before moving onto more time-consuming and costly efforts.
The two main types of smoke testing
Learn about the main types of smoke testing.
Smoke testing for software quality assurance
The term smoke testing is commonly used in the quality assurance world. Smoke testing means QAing the most critical functions of a software build right after it is built in order to determine if the build should move on to other forms of testing, such as functional testing, exploratory testing, regression testing, automated testing, etc.
Think of a smoke test as a smoke signal. Does the app immediately reveal major bugs? Or do the critical features work? If they do, then it’s time to hand the build over to a manual tester so they can dive deep and try to break it.
Smoke testing is never an adequate method of quality assurance all on it’s own. Rather, it’s the first state gate on the path to in-depth testing that companies need to execute on in order to ship quality products and updates to customers.
Smoke testing for product validation
When it comes to product validation, smoke testing is used to check market demand before building a product.
You might be wondering how this differs from a design sprint.
Here’s the main difference between smoke testing and a design sprint:
Smoke testing - Marketing your product concept before it is built to verify market demand. This is done at the end of a design sprint process in order to further validate the concept and ease of marketing it before investing money into the product and its go to market strategy.
Design sprint - One or two week process to come up with a winning product concept to a problem. This process includes building prototypes, conducting user testing, and reiterating on your prototype. At the end, smoke testing may be done to check the market demand and ease of acquiring waitlist sign ups.
As you can see, smoke testing is not the same thing as a design sprint.
While a design sprint validates the product and its target audience, smoke testing accepts that validation and moves on to validate the product’s marketing and messaging strategy.
When to do smoke testing
You should run a smoke testing experiment as early on in the product development process as possible.
It should be the first thing you do.
For iron-clad product validation that leaves nothing to chance, proceed in this order:
Smoke testing
User testing and user feedback
User testing and user feedback
Product development
Smoke testing comes before prototyping, because it’s typically easier to achieve. However, if you want to run sales demos as part of your smoke testing process, you can combine low fidelity prototypes with smoke testing. In this case you’ll build the prototype and share it with potential buyers instead of creating a product landing page and testing conversions.
Smoke testing cycle
Smoke testing follows a simple and clear cycle: Build, Test, Reiterate, Develop.
You should reiterate until you’ve got enough data from your smoke test to validate your product idea.
Follow these four steps:
Build - Create the strategy and materials you’ll need to run your test. This could mean developing a landing page to collect email addresses and creating social media ads. Or it could mean writing your sales call script and coming up with cold emails. (Read the step-by-step guide and best practices sections below for more guidance on how to construct a smoke test.)
Test - The next step is to run your test. For high-volume, self-serve products you should collect data from at least 100 target customers. If you’re selling a low-volume, enterprise product, aim to talk with at least 20 target buyers.
Reiterate - Based on your test results, you might need to reiterate. If not enough people are willing to buy your product, come up with a different solution and test that. Or, choose to solve a different, more pressing problem.
Develop - Tested and reiterate your product concept enough to get a clear and resounding answer to the question “Will people buy this?” Once you’ve got a yes, it’s time to develop your product and bring it to market. Your first customers should come from the target customers you’ve been in contact with during your smoke test.
5 advantages to smoke testing
Smoke testing offers a lot of business-critical advantages, making it a strategy that’s hard to ignore.
Validate your product before you build it - At its core, smoke testing allows you to be sure you’re investing time and money into a product that has a high probability of succeeding (rather than guessing and hoping).
Reduce the risk of losses - You can reduce your risk of losing money. This is particularly for software development, which can cost $100,000 - $300,000 to build an MVP.
Bring the right product to market faster - When you use smoke testing, you can launch a viable product and drive revenue much faster. You’re not wasting time developing something no one wants and continuously having to go back to the drawing board.
Create a short feedback loop with customers - You’ll be building your business on a strong foundation of customer feedback. You’ll already have processes in place for listening to your market (such as survey methods, beta user recruitment, etc.). This gives you a competitive advantage as you move forward with other product ideas or add-ons.
Increase chances of getting investors - If you need investors to be able to develop your product, you’re a lot more likely to raise money if you can present smoke testing data that validates your concept.
How to do smoke testing for SaaS product validation
While smoke testing is very different from a design sprint, you can’t hop straight into smoke testing.
You shouldn’t attempt to validate the marketing of a product before you’ve validated the product itself. That’s why we’re including some elements from a design sprint in this step-by-step process as well.
1. Understand the problem you’re trying to solve
The first step is to deeply understand the problem that you’re trying to solve. This is often easier if you have experienced the problem first hand. If not, you’ll need target users to interview and ideally shadow on the job.
During our design sprints with clients, we like to map things out with sticky notes. We cover the target users, what they’re currently doing, the direct and indirect competitors, and other elements that address the entire landscape of the problem.
2. Come up with a solution
Next, you need to come up with a solution. Usually, this looks like coming up with 2 to 5 different solutions, each with a different (and possibly overlapping) feature set.
As a team, you need to work together to pick the best solution. Which one addresses customer needs in the most lightweight, efficient, and affordable way?
3. Make a low-fidelity prototype
With the right product concept, it’s time to make a low-fidelity prototype. This might be nothing more than a drawing on a piece of paper, a digital mockup, a PDF, or a slideshow.
When we did our first demos with prospective customers to validate the product, it wasn’t even built yet. In fact, our “product” was a PowerPoint deck with a UI and hyperlinks that jumped from one slide to the next. - Garrett Moon, founder of CoSchedule
Your low-fidelity prototype shouldn’t be fancy, but it should clearly communicate what features would be included in the MVP, and what jobs or tasks those features would achieve. You can use the Jobs to Be Done framework to help communicate these.
4. Conduct user testing to get initial feedback
Next up, you’ll want to conduct user testing to get feedback. Show your low-fidelity prototype to real target users and listen to what they have to say.
They might say that your initial concept is missing an important feature, or that what it solves isn’t as important as a much bigger problem they are dealing with.
Come in with an open mind and no assumptions so you can address the most profitable problem possible.
5. Reiterate and create a high-fidelity prototype
With feedback to guide your way, you need to reiterate on your prototype. Make a high-fidelity version that looks and feels like your final product will. It should have the core UX of the product, even if clicking on buttons doesn’t trigger any server calls.
The point is to have something that users can play around with and that you can screenshot for your smoke test.
At this stage, you’ll want to validate this high-fidelity prototype with further user testing.
6. Make a landing page for your SaaS product
Next, it’s time to make a landing page for your product. This is where you’ll send traffic using various marketing strategies.
Here’s an example of a waitlist landing page we built for our new SaaS product RaiseIQ. Make sure to include images of your high-fidelity prototype, and also add quality, engaging copy that hits at the heart of the problem.
7. Get waitlist signups (market before you build)
Here are some ways you can drive traffic to this landing page:
SEO blogging
LinkedIn posts
LinkedIn articles
LinkedIn ads
LinkedIn outreach
Facebook ads
YouTube ads
Cold email
Cold calling
Direct mail
You’ll have to pick the right strategies for your target audience and price point.
We started blogging as a company before we even had a product...as a way of validating early on if the problem we were trying to solve for marketers actually existed - Jennifer Pepper, Marketing Manager at Unbounce.
8. Evaluate your marketing, demand, and commitment level
Make sure to set up Google Analytics tracking before you begin marketing your SaaS so you can see which channels are referring the most traffic to your website. Direct traffic will likely be coming from your cold outreach campaigns.
You should check which marketing campaigns worked the best. Also, your conversion rate should be at least 25% since they are only signing up for your waitlist.
Not only do you need to validate marketing campaigns and demand, but you also need to take an honest look at your own commitment level.
Do you have what it takes to commit to this business? Is the problem you’re solving important enough? Are you passionate about it?
The great thing about smoke testing is that it helps you validate yourself as a founder of this particular business, not just your new product.
6 smoke testing best practices
Getting your smoke test right is essential.
You don’t want inaccurate or insufficient data skewing your results, especially if you have an emotional connection to the business idea (which can set you up to use your smoke test as confirmation bias).
Rely on these best practices to ensure that your smoke test will give you clear yes or no validation results.
1. Create a list of smoke testing questions
For any good experiment, you need good questions. Smoke testing is no different.
By kickstarting your experiment with clear questions, you’ll be able later consolidate your research data into clear answers. This can help you communicate clearly with stakeholders, co-founders, investors, and yourself!
Here are some example questions:
Is our target audience willing to use this product?
Is our target audience willing to switch to this product from what they’re currently using?
Is our target audience willing to pay for this product?
How much is our target audience willing to pay for this product on average?
Does this product solve a pressing problem for our target audience?
Does this product solve the most pressing problem for our target audience?
Remember, you’ll be looking for answers to these questions just for your minimum viable product. So don’t consider what people might be willing to pay for the long-term vision you have, but for the MVP. The long-term vision will likely change, but right now you need to know how people feel about the MVP.
2. Choose the right smoke testing method based on your target market
Above, we’ve listed a step-by-step process for smoke testing with a landing page. This is a really great method for most entrepreneurs, but it’s not the only one.
Some products simply aren’t sold via landing pages. If you’re not building a self-serve SaaS but rather an enterprise product, you might get more accurate results with a sales-based smoke testing process rather than a marketing-based one. You can reach out to your target market with a variety of outbound sales methods and attempt to sell something that doesn’t exist yet.
You’ll then be able to find out if your target customers are willing to pay for your product and whether it falls into the “essential expense” or “nice to have” category. If it’s not a priority expense for enterprises, don’t build it.
3. Drive targeted traffic to your smoke test
The quality of the traffic you’re directing to your smoke test is just as important as the test itself.
The wrong traffic will definitely skew your results. So only use traffic channels that you’re confident you can target the right people. Now’s not the time to try to run Facebook ads for yourself. Hire an expert agency for a short-term contract.
Here are the best channels for smoke testing traffic:
Social media ads - (This method is best reserved for self-serve products.) Platforms like Facebook, Instagram, X, TikTok, and Pinterest are great for driving targeted traffic to your smoke testing landing page.
Search ads - (This method is best reserved for self-serve products.) You can show your landing page to people searching for exactly what you want to offer, or very closely related keyphrases.
Cold email - (This method can be used for self-serve or enterprise products.) Reach out to your target audience using cold email campaigns that send traffic to your landing page or pitch a demo call.
Direct messages - (This method can be used for self-serve or enterprise products.) Strike up a conversation with target buyers, and then direct them to check out your landing page or book a call with you for a demo call.
When booking demo calls with potential buyers, you can walk them through a low-fidelity prototype of your product since you won’t have the actual product built yet.
4. Have people vote with their money
One of the most accurate ways to test a product page is to actually collect customer payments, even before the product has been built.
Some of the best digital product creators will launch a page, drive Facebook and Instagram ad traffic to it, and then refund anyone who purchased and let them know that the product isn’t ready, but they’ll let them know if and when it becomes available. This way, the entrepreneur can review their return on ad spend (ROAS) and make sure that the campaign and product are profitable before deciding to move forward.
For the fast-moving consumer packaged goods industry, Veylinx runs real auctions where consumers have to actually pay for the product tests. This way, brands get real data on what customers would rather buy.
Dependent on your target audience and smoke testing budget, having people vote with their money might not be the right strategy for you, but it’s certainly worth considering.
5. Use a variety of customer research methods to gather qualitative data
When you run a smoke test, you’re going to collect a lot of quantitative data like ad clicks, ad conversion rates, email opens, email click-through rates, etc.
It’s important to actually speak with customers and collect qualitative data so you can really understand what your target market is trying to tell you.
Some of the best qualitative research methods are surveys and customer interviews. Whether your smoke test succeeds or fails, use these methods to understand the result. Why are people interested or uninterested in your product? That’s what you want to find out, because it will make sure that you reiterate or develop your idea correctly.
6. Find out if there’s a more pressing problem to solve
We mentioned this in the list of validation questions above, but it’s worth mentioning here.
Just because someone is willing to pay for your product, doesn’t mean it’s the right concept for you to develop. There might be a more pressing issue to solve…one that, if you solve it, would help you build a business with higher conversion rates, profit margins, and exit potential than your current idea.
Smoke testing examples
Check out how these wise entrepreneurs used smoke testing to their strategic advantage.
Buffer’s smoke testing sign-up page
Joel Gascoigne was an avid tweeter when he got the idea to create an app that would pre-fill a Twitter account in 2010. Before investing in its development, however, he created a landing page to collect interest.
He made the landing page look like a real sign-up page with information about the product and pricing plans. But then when users would click on “plans and pricing,” they’d receive a message saying that the product wasn’t quite ready but that they could enter their email address to be notified of the launch.
This was a smart smoke testing strategy for multiple reasons:
Simply shared the MVP idea
Collected data on how many people clicked on the plans and pricing button
Collected email addresses
In fact, creating a landing page before you build your product is one of the easiest and most affordable product validation methods.
CoSchedule’s faux software platform
CoSchedule is a SaaS platform that helps content marketers publish to all of their channels (blog, social media, email, etc.) from one place.
When creating CoSchedule, the founders built the interface for the product before investing time and money to create the complex backend. They were running their agency and had provided both software development and content marketing services to clients. The founders sold CoSchedule to some of their clients before it was complete to see if people were really willing to pay for it. While users were clicking buttons to pre-schedule their content on the frontend, the founders were manually publishing that content on the backend.
Once they realized that marketing teams were desperate for their solution, they quickly got to work building the real backend of the product that would connect the content to popular channels.
You can find famous SaaS MVP stories here for more inspiration.
Should you work with a smoke testing company?
You can hire DevSquad to partner with you on:
Design sprint - Our 10-day design sprint process is designed to validate a true MVP using real target users.
Smoke testing - We’ll build out your landing page for you featuring screenshots of your validated high-fidelity prototype.
MVP development - We build a real MVP (as small as possible) in 3 to 9 months, using our pay-as-you-go monthly development model. We can also offer UX design alone, if you have backend developers on hand.
Smoke testing carries your product validation a step further. Without great marketing, a product can’t succeed. Just because people say your product would help them with their problem, doesn’t mean they would actually pay for it.
Use smoke testing to get as close to product-market fit as possible before you actually invest money in building a software product.
DevSquad excels at building brand new SaaS products. Learn more about DevSquad.
