What are p-values?
What are p-values?
A p-value, or probability value, is a number between 0 and 1 which indicates the likelihood that results occurred by chance. Generally speaking, the smaller the p-value, the more solid the result.
A p-value of 0.04 can be read in any of the following ways:
- There’s a 4% chance the result of this analysis occurred randomly and cannot be relied on.
- There’s a 96% chance the result of this analysis is real and can be relied on.
Why do p-values matter?
Noise - or randomness - in data is inevitable.
Even in identical scenarios where both Group A and Group B perform equally, you can still expect small fluctuations in the data.
This presents a challenge.
If you can see a difference (in our example here: ±4%), at what point should you be concerned?
p-values tell you how confident to be
The p-value allows you to determine whether or not to believe the difference you are seeing.
The lower the p-value, the more confident you can be that the results are real. The greater the p-value, the more likely the results are coincidental (the effect of normal random noise), and hence, you should not draw any conclusions.
What influences p-values?
- Sample size: The effect of random error decreases as the size of a sample increases. So the larger your sample, the more confident you can be. This is why it’s hard to make a judgement based on 2 data points, but it’s easy when you have 2,000.
- Distribution of results: The less variance in results, the more confident you can be. The more outliers there are, the larger the p-value will be.
How should you interpret p-values?
In the natural sciences, a p-value of less than 0.05, which is 5%, is generally agreed to be significant. This implies a less than 5% likelihood that results occurred by random chance, so conclusions are therefore deemed to be noteworthy. Similarly, a p-value less than 0.01, or 1%, is considered highly significant.
When considering impact-driven business decision, we need to add risk and reward. For example, a p-value of 0.15 may be acceptable if the cost of an initiative is low and the reward is high.
“You need to ask yourself, for the action I’m considering taking, am I happy to accept there’s an X% chance we made the wrong conclusion from the data?”
As a general approach, Pirical will highlight to clients where p-values are less than 0.10.
Analysing p-values in Pirical On Demand
You will find p-values throughout Pirical On Demand where two populations are being compared.
P-values below 0.1 will be highlighted.
Stay up-to-date with the latest market insights and law firm rankings
Subscribe to email updates
Subscribe to content
