How to Calculate Precision

Reviewed by Grace Imson, MA

Last Updated: May 22, 2023 Fact Checked

Precision means that a measurement using a particular tool or implement produces similar results every single time it is used. For example, if you step on a scale five times in a row, a precise scale would give you the same weight each time. In math and science, calculating precision is essential to determine if your tools and measurements work well enough to get good data. You can report precision of any data set using the range of values, the average deviation, or the standard deviation.

Method 1

Method 1 of 4:

Calculating the Range

Download Article

1
Determine the highest measured value. It helps to begin by sorting your data in numerical order, from lowest to highest. This will ensure that you do not miss any values. Then select the value at the end of the list. ^{[1]
X
Research source}
- For example, suppose you are testing the precision of a scale, and you observe five measurements: 11, 13, 12, 14, 12. After sorting, these values are listed as 11, 12, 12, 13, 14. The highest measurement is 14.
2
Find the lowest measured value. Once your data has been sorted, finding the lowest value is as simple as looking at the beginning of the list. ^{[2]
X
Research source}
- For the scale measurement data, the lowest value is 11.
Advertisement
3
Subtract the lowest value from the highest. The range of a set of data is the difference between the highest and lowest measurements. Just subtract one from the other. Algebraically, the range can be expressed as:
- ${\text{Range}}=x(max)-x(min)$
- For the sample data, the range is:
  - ${\text{Range}}=x(max)-x(min)=14-11=3$
4
Report the range as the precision. When reporting data, it is important to let the readers know what you have measured. Because there are different measures of precision, you should specify what you are reporting. For this data, you would report Mean=12.4, Range=3, or simply that the Mean=12.4±3. ^{[3]
X
Research source}
- The mean is not actually part of calculating the range or precision, but it is generally the primary calculation for reporting the measured value. The mean is found by adding up the sum of the measured values and then dividing by the number of items in the group. For this set of data, the mean is (11+13+12+14+12)/5=12.4.

Method 2

Method 2 of 4:

Calculating the Average Deviation

Download Article

1
Find the mean of the data. The average deviation is a more detailed measure of the precision of a group of measurements or experiment values. The first step in finding the average deviation is to calculate the mean of the measured values. The mean is the sum of the values, divided by the number of measurements taken. ^{[4]
X
Research source}
- For this example, use the same sample data as before. Assume that five measurements have been taken, 11, 13, 12, 14, and 12. The mean of these values is (11+13+12+14+12)/5=12.4.
2
Calculate the absolute deviation of each value from the mean. For this calculation of precision, you need to determine how close each value is to the mean. To do this, subtract the mean from each number. For this measurement, it does not matter whether the value is above or below the mean. Subtract the numbers and just use the positive value of the result. This is also called the absolute value. ^{[5]
X
Research source}
- Algebraically, the absolute value is shown by placing two vertical bars around the calculation, as follows:
  - ${\text{Absolute deviation}}=|x-\mu |$
  - For this calculation, $x$ represents each of the experimental values, and $\mu$ is the calculated mean.
- For the values of this sample data set, the absolute deviations are:
  - $|12-12.4|=0.4$
  - $|11-12.4|=1.4$
  - $|14-12.4|=1.6$
  - $|13-12.4|=0.6$
  - $|12-12.4|=0.4$
3
Find the average deviation. Use the absolute deviations and find their mean. As you did with the original data set, you will add them together and divide by the number of values. This is represented algebraically as: ^{[6]
X
Research source}
- ${\text{Average deviation}}={\frac {\Sigma |x-\mu |}{n}}$
- For this sample data, the calculation is:
  - ${\text{Average deviation}}={\frac {0.4+1.4+1.6+0.6+0.4}{5}}$
  - ${\text{Average deviation}}={\frac {4.4}{5}}$
  - ${\text{Average deviation}}=0.88$
This result may be reported as the mean, plus or minus the average deviation. For this sample data set, this result would look like 12.4±0.88. Note that reporting precision as the average deviation makes the measurement appear much more precise than with the range. ^{[7]
X
Research source}

Method 3

Method 3 of 4:

Calculating Standard Deviation

Download Article

1
Use the correct formula for standard deviation. For any size data set, the standard deviation is a reliable statistic for reporting precision. There are two formulas for calculating standard deviation, with a very slight difference between them. You will use one formula if your measured data represents an entire population. You will use the second formula if your measured data is from only a sample of the population. ^{[8]
X
Research source}
- Your data represents an entire population if you have collected all the measurements possible from all possible subjects. For example, if you are conducting tests on people with some very rare disease, and you believe that you have tested everyone with that disease, then you have the entire population. The standard deviation formula in this case is:
  - $\sigma ={\sqrt {{\frac {\Sigma (x-\mu )^{2}}{n}}}}$
- A sample set is any group of data less than an entire population. This is actually going to be used more often. The standard deviation formula for a sample set is:
  - $\sigma ={\sqrt {{\frac {\Sigma (x-\mu )^{2}}{n-1}}}}$
- Notice that the only difference is in the denominator of the fraction. For an entire population, you will divide by $n$ . For a sample set, you will divide by $n-1$ .
2
Find the mean of the data values. As with calculating the average deviation, you will begin by finding the mean of the data values. ^{[9]
X
Research source}
- Using the same set of measurements as above, the mean is 12.4.
3
Find the square of each variation. For each data point, subtract the data value from the mean, and square that result. Because you are squaring these variations, whether the difference is positive or negative does not matter. The square of the difference will always be positive. ^{[10]
X
Research source}
- For the five data values in this sample, these calculations are as follows:
  - $(12-12.4)^{2}=(-0.4)^{2}=0.16$
  - $(11-12.4)^{2}=(-1.4)^{2}=1.96$
  - $(14-12.4)^{2}=1.6^{2}=2.56$
  - $(13-12.4)^{2}=0.6^{2}=0.36$
  - $(12-12.4)^{2}=(-0.4)^{2}=0.16$
4
Calculate the sum of the squared differences. The numerator of the standard deviation fraction is the sum of the squared differences between each value and the mean. To find this sum, add together the figures from the previous calculation. ^{[11]
X
Research source}
- For the sample data set, these are:
  - $0.16+1.96+2.56+0.36+0.16=5.2$
5
Divide by the data size. This is the one step that will differ for either a population calculation or a sample set calculation. For a full population, you will divide by $n$ , the number of values. For a sample set, you will divide by $n-1$ . ^{[12]
X
Research source}
- This example has only five measurements and is therefore only a sample set. Thus, for the five values being used, divide by (5-1) or 4. The result is $5.2/4=1.3$ .
6
Find the square root of the result. At this point, the calculation represents what is called the variance of the data set. The standard deviation is the square root of the variance. Use a calculator to find the square root, and the result is the standard deviation. ^{[13]
X
Research source}
- $\sigma ={\sqrt {1.3}}=1.14$
7
Report your result. Using this calculation, the precision of the scale can be represented by giving the mean, plus or minus the standard deviation. For this data, this will be 12.4±1.14.
- The standard deviation is perhaps the most common measurement of precision. Nevertheless, for clarity, it is still a good idea to use a footnote or parentheses to note that the precision value represents the standard deviation.

Method 4

Method 4 of 4:

Deciding How to Report Precision

Download Article

1
Use the word precision correctly. Precision is a term that describes the level of repeatability of measurements. When collecting a group of data, either by measurement or through an experiment of some kind, the precision describes how close together the results of each measurement or experiment are going to be. ^{[14]
X
Research source}
- Precision is not the same as accuracy. Accuracy measures how close experimental values come to the true or theoretical value, while precision measures how close the measured values are to each other.
- It is possible for data to be accurate but not precise or to be precise but not accurate. Accurate measurements are close to the target value but may not be close to each other. Precise measurements are close to each other, whether or not they are close to the target.
2
Choose the best measure of precision. The word “precision” does not have a single meaning. You can represent precision using several different measurements. You need to decide the best one.
- Range. For small data sets with about ten or fewer measurements, the range of values is a good measure of precision. This is particularly true if the values appear reasonably closely grouped. If you see one or two values that appear far from the others, you may wish to use a different calculation.
- Average deviation. The average deviation is a more accurate measure of precision for a small set of data values. ^{[15]
  X
  Research source}
- Standard deviation. The standard deviation is perhaps the most recognized measure of precision. Standard deviation may be used to calculate the precision of measurements for an entire population or a sample of the population.
3
Report your results clearly. Very often, investigators will report data by giving the mean of the measured value, followed by a statement of the precision. The precision is shown with a “±” symbol. This provides an indication of precision, but it does not clearly explain to the reader if the number following the “±” symbol is a range, standard deviation, or some other measurement. To be very clear, you should define what measure of precision you are using, either in a footnote or parenthetical note.
- For example, for one series of data, the result could be reported as 12.4±3. However, a more explanatory way to report the same data would be to say “Mean=12.4, Range=3.”

Community Q&A

Search

Add New Question

Question

How do you measure accuracy?

Community Answer

Accuracy is a measure of how close you are to the known, expected value of what you are measuring. If you have a known weight of 10 kg, for example, and you put it on a scale and the scale says "9.2," then your scale is accurate within 0.8 kg.

Thanks! We're glad this was helpful.
Thank you for your feedback.
If wikiHow has helped you, please consider a small contribution to support us in helping more readers like you. We’re committed to providing the world with free how-to resources, and even $1 helps us in our mission. Support wikiHow
Yes No

Not Helpful 6 Helpful 31
Question

How do I calculate the level of precision of an equipment? It's an electrolyte analyser

Community Answer

Use it to take several measurements and then follow the directions in this article.

Thanks! We're glad this was helpful.
Thank you for your feedback.
If wikiHow has helped you, please consider a small contribution to support us in helping more readers like you. We’re committed to providing the world with free how-to resources, and even $1 helps us in our mission. Support wikiHow
Yes No

Not Helpful 5 Helpful 27
Question

How do you know if a measurement is precise?

Community Answer

When the mean absolute deviation or the standard range is as close to zero as possible.

Thanks! We're glad this was helpful.
Thank you for your feedback.
If wikiHow has helped you, please consider a small contribution to support us in helping more readers like you. We’re committed to providing the world with free how-to resources, and even $1 helps us in our mission. Support wikiHow
Yes No

Not Helpful 2 Helpful 22

See more answers

Ask a Question

200 characters left

Include your email address to get a message when this question is answered.

Submit

Video

Tips

If one of your trial values is much higher or lower than the rest of your values, do not exclude this number from your calculations. Even if it was a mistake, it is data and should be utilized for a proper calculation.

Thanks
Helpful 1 Not Helpful 0
In this article, only five values were used for mathematical simplicity. In an actual experiment, you should perform more than five trials to achieve a more accurate calculation. The more trials you run, the closer you will get to a clear precision value.

Thanks
Helpful 1 Not Helpful 0

Submit a Tip

All tip submissions are carefully reviewed before being published

Name

Please provide your name and last initial

Submit

Thanks for submitting a tip for review!

You Might Also Like

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]

[15]

Log in

How to Calculate Precision

Steps

Calculating the Range

Calculating the Average Deviation

Calculating Standard Deviation

Deciding How to Report Precision

Community Q&A

Video

Tips

You Might Also Like

References

About This Article

Reader Success Stories

Did this article help you?

Quizzes

You Might Also Like

Featured Articles

Trending Articles

Featured Articles

Featured Articles

Watch Articles

Trending Articles

Quizzes

Explore Categories