What Makes A Good Hypothesis Statement
Update: I’ve since revised this hypothesis format. You can find the most current version in this article:
“My hypothesis is …”
These words are becoming more common everyday. Product teams are starting to talk like scientists. Are you?
The internet industry is going through a mindset shift. Instead of assuming we have all the right answers, we are starting to acknowledge that building products is hard. We are accepting the reality that our ideas are going to fail more often than they are going to succeed.
Rather than waiting to find out which ideas are which after engineers build them, smart product teams are starting to integrate experimentation into their product discovery process. They are asking themselves, how can we test this idea before we invest in it?
This process starts with formulating a good hypothesis.
These Are Not the Hypotheses You Are Looking For
When we are new to hypothesis testing, we tend to start with hypotheses like these:
- Fixing the hard-to-use comment form will increase user engagement.
- A redesign will improve site usability.
- Reducing prices will make customers happy.
There’s only one problem. These aren’t testable hypotheses. They aren’t specific enough.
A good hypothesis can be clearly refuted or supported by an experiment. – Tweet This
The 5 Components of a Good Hypothesis
To make sure that your hypotheses can be supported or refuted by an experiment, you will want to include each of these elements:
- the change that you are testing
- what impact we expect the change to have
- who you expect it to impact
- by how much
- after how long
The Change: This is the change that you are introducing to your product. You are testing a new design, you are adding new copy to a landing page, or you are rolling out a new feature.
Be sure to get specific. Fixing a hard-to-use comment form is not specific enough. How will you fix it? Some solutions might work. Others might not. Each is a hypothesis in its own right.
Design changes can be particularly challenging. Your hypothesis should cover a specific design not the idea of a redesign.
In other words, use this:
- This specific design will increase conversions.
- Redesigning the landing page will increase conversions.
The former can be supported or refuted by an experiment. The latter can encompass dozens of design solutions, where some might work and others might not.
The Expected Impact: The expected impact should clearly define what you expect to see as a result of making the change.
How will you know if your change is successful? Will it reduce response times, increase conversions, or grow your audience?
The expected impact needs to be specific and measurable. – Tweet This
You might hypothesize that your new design will increase usability. This isn’t specific enough.
You need to define how you will measure an increase in usability. Will it reduce the time to complete some action? Will it increase customer satisfaction? Will it reduce bounce rates?
There are dozens of ways that you might measure an increase in usability. In order for this to be a testable hypothesis, you need to define which metric you expect to be affected by this change.
Who Will Be Impacted: The third component of a good hypothesis is who will be impacted by this change. Too often, we assume everyone. But this is rarely the case.
I was recently working with a product manager who was testing a sign up form popup upon exiting a page.
I’m sure you’ve seen these before. You are reading a blog post and just as you are about to navigate away, you get a popup that asks, “Would you like to subscribe to our newsletter?”
She A/B tested this change by showing it to half of her population, leaving the rest as her control group. But there was a problem.
Some of her visitors were already subscribers. They don’t need to subscribe again. For this population, the answer to this popup will always be no.
Rather than testing with her whole population, she should be testing with just the people who are not currently subscribers.
This isn’t easy to do. And it might not sound like it’s worth the effort, but it’s the only way to get good results.
Suppose she has 100 visitors. Fifty see the popup and fifty don’t. If 45 of the people who see the popup are already subscribers and as a result they all say no, and of the five remaining visitors only 1 says yes, it’s going to look like her conversion rate is 1 out of 50, or 2%. However, if she limits her test to just the people who haven’t subscribed, her conversion rate is 1 out of 5, or 20%. This is a huge difference.
Who you test with is often the most important factor for getting clean results. – Tweet This
By how much: The fourth component builds on the expected impact. You need to define how much of an impact you expect your change to have.
For example, if you are hypothesizing that your change will increase conversion rates, then you need to estimate by how much, as in the change will increase conversion rate from x% to y%, where x is your current conversion rate and y is your expected conversion rate after making the change.
This can be hard to do and is often a guess. However, you still want to do it. It serves two purposes.
First, it helps you draw a line in the sand. This number should determine in black and white terms whether or not your hypothesis passes or fails and should dictate how you act on the results.
Suppose you hypothesize that the change will improve conversion rates by 10%, then if your change results in a 9% increase, your hypothesis fails.
This might seem extreme, but it’s a critical step in making sure that you don’t succumb to your own biases down the road.
It’s very easy after the fact to determine that 9% is good enough. Or that 2% is good enough. Or that -2% is okay, because you like the change. Without a line in the sand, you are setting yourself up to ignore your data.
The second reason why you need to define by how much is so that you can calculate for how long to run your test.
After how long: Too many teams run their tests for an arbitrary amount of time or stop the results when one version is winning.
This is a problem. It opens you up to false positives and releasing changes that don’t actually have an impact.
If you hypothesize the expected impact ahead of time than you can use a duration calculator to determine for how long to run the test.
Finally, you want to add the duration of the test to your hypothesis. This will help to ensure that everyone knows that your results aren’t valid until the duration has passed.
If your traffic is sporadic, “how long” doesn’t have to be defined in time. It can also be defined in page views or sign ups or after a specific number of any event.
Putting It All Together
Use the following examples as templates for your own hypotheses:
- Design x [the change] will increase conversions [the impact] for search campaign traffic [the who] by 10% [the how much] after 7 days [the how long].
- Reducing the sign up steps from 3 to 1 will increase signs up by 25% for new visitors after 1,000 visits to the sign up page.
- This subject line will increase open rates for daily digest subscribers by 15% after 3 days.
After you write a hypothesis, break it down into its five components to make sure that you haven’t forgotten anything.
- Change: this subject line
- Impact: will increase open rates
- Who: for daily digest subscribers
- By how much: by 15%
- After how long: After 3 days
And then ask yourself:
- Is your expected impact specific and measurable?
- Can you clearly explain why the change will drive the expected impact?
- Are you testing with the right population?
- Did you estimate your how much based on a baseline and / or comparable changes? (more on this in a future post)
- Did you calculate the duration using a duration calculator?
It’s easy to give lip service to experimentation and hypothesis testing. But if you want to get the most out of your efforts, make sure you are starting with a good hypothesis.
Did you learn something new reading this article? Keep learning. Subscribe to the Product Talk mailing list to get the next article in this series delivered to your inbox.
Filed Under: Experimentation
What is a Hypothesis?
A hypothesis is a tentative, testable answer to a scientific question. Once a scientist has a scientific question she is interested in, the scientist reads up to find out what is already known on the topic. Then she uses that information to form a tentative answer to her scientific question. Sometimes people refer to the tentative answer as "an educated guess." Keep in mind, though, that the hypothesis also has to be testable since the next step is to do an experiment to determine whether or not the hypothesis is right!
A hypothesis leads to one or more predictions that can be tested by experimenting.
Predictions often take the shape of "If ____then ____" statements, but do not have to. Predictions should include both an independent variable (the factor you change in an experiment) and a dependent variable (the factor you observe or measure in an experiment). A single hypothesis can lead to multiple predictions, but generally, one or two predictions is enough to tackle for a science fair project.
Examples of Hypotheses and Predictions
|How does the size of a dog affect how much food it eats?||Larger animals of the same species expend more energy than smaller animals of the same type. To get the energy their bodies need, the larger animals eat more food.||If I let a 70-pound dog and a 30-pound dog eat as much food as they want, then the 70-pound dog will eat more than the 30-pound dog.|
|Does fertilizer make a plant grow bigger?||Plants need many types of nutrients to grow. Fertilizer adds those nutrients to the soil, thus allowing plants to grow more.||If I add fertilizer to the soil of some tomato seedlings, but not others, then the seedlings that got fertilizer will grow taller and have more leaves than the non-fertilized ones.|
|Does an electric motor turn faster if you increase the current?||Electric motors work because they have electromagnets inside them, which push/pull on permanent magnets and make the motor spin. As more current flows through the motor's electromagnet, the strength of the magnetic field increases, thus turning the motor faster.||If I increase the current supplied to an electric motor, then the RPMs (revolutions per minute) of the motor will increase.|
|Is a classroom noisier when the teacher leaves the room?||Teachers have rules about when to talk in the classroom. If they leave the classroom, the students feel free to break the rules and talk more, making the room nosier.||If I measure the noise level in a classroom when a teacher is in it and when she leaves the room, then I will see that the noise level is higher when my teacher is not in my classroom.|
What if My Hypothesis is Wrong?
What happens if, at the end of your science project, you look at the data you have collected and you realize it does not support your hypothesis? First, do not panic! The point of a science project is not to prove your hypothesis right. The point is to understand more about how the natural world works. Or, as it is sometimes put, to find out the scientific truth. When scientists do an experiment, they very often have data that shows their starting hypothesis was wrong. Why? Well, the natural world is complex—it takes a lot of experimenting to figure out how it works—and the more explanations you test, the closer you get to figuring out the truth. For scientists, disproving a hypothesis still means they gained important information, and they can use that information to make their next hypothesis even better. In a science fair setting, judges can be just as impressed by projects that start out with a faulty hypothesis; what matters more is whether you understood your science fair project, had a well-controlled experiment, and have ideas about what you would do next to improve your project if you had more time. You can read more about a science fair judge's view on disproving your hypothesis here.
It is worth noting, scientists never talk about their hypothesis being "right" or "wrong." Instead, they say that their data "supports" or "does not support" their hypothesis. This goes back to the point that nature is complex—so complex that it takes more than a single experiment to figure it all out because a single experiment could give you misleading data. For example, let us say that you hypothesize that earthworms do not exist in places that have very cold winters because it is too cold for them to survive. You then predict that you will find earthworms in the dirt in Florida, which has warm winters, but not Alaska, which has cold winters. When you go and dig a 3-foot by 3-foot-wide and 1-foot-deep hole in the dirt in those two states, you discover Floridian earthworms, but not Alaskan ones. So, was your hypothesis right? Well, your data "supported" your hypothesis, but your experiment did not cover that much ground. Can you really be sure there are no earthworms in Alaska? No. Which is why scientists only support (or not) their hypothesis with data, rather than proving them. And for the curious, yes there are earthworms in Alaska.
|What Makes a Good Hypothesis?||For a Good Hypothesis, You Should Answer "Yes" to Every Question|
|Is the hypothesis based on information from reference materials about the topic?||Yes / No|
|Can at least one clear prediction be made from the hypothesis?||Yes / No|
|Are predictions resulting from the hypothesis testable in an experiment?||Yes / No|
|Does the prediction have both an independent variable (something you change) and a dependent variable (something you observe or measure)?||Yes / No|
Educator Tools for Teaching about Hypotheses
Using our Google Classroom Integration, educators can assign a quiz to test student understanding of hypotheses. Educators can also assign students an online submission form to fill out detailing the hypothesis of their science project.
Support for Science Buddies provided by: