Insufficient Precision or Accuracy of a Real Number

The program processes a real number with an implementation in which the number’s representation does not preserve required accuracy and precision in its fractional part, causing an incorrect result.


When a security decision or calculation requires highly precise, accurate numbers – such as financial calculations or prices – then small variations in the number could be exploited by an attacker.

There are multiple ways to store the fractional part of a real number in a computer. In all of these cases, there is a limit to the accuracy of recording a fraction. If the fraction can be represented in a fixed number of digits (binary or decimal), there might not be enough digits assigned to represent the number. In other cases the number cannot be represented in a fixed number of digits due to repeating in decimal or binary notation (e.g. 0.333333...) or due to a transcendental number such as Π or √2. Rounding of numbers can lead to situations where the computer results do not adequately match the result of sufficiently accurate math.


There are three major ways to store real numbers in computers. Each method is described along with the limitations of how they store their numbers.

Fixed: Some implementations use a fixed number of binary bits to represent both the integer and the fraction. In the demonstrative example about Muller's Recurrence, the fraction 108.0 - ((815.0 - 1500.0 / z) / y) cannot be represented in 8 binary digits. The numeric accuracy within languages such as PL/1, COBOL and Ada is expressed in decimal digits rather than binary digits. In SQL and most databases, the length of the integer and the fraction are specified by the programmer in decimal. In the language C, fixed reals are implemented according to ISO/IEC TR18037

Floating: The number is stored in a version of scientific notation with a fixed length for the base and the significand. This allows flexibility for more accuracy when the integer portion is smaller. When dealing with large integers, the fractional accuracy is less. Languages such as PL/1, COBOL and Ada set the accuracy by decimal digit representation rather than using binary digits. Python also implements decimal floating-point numbers using the IEEE 754-2008 encoding method.

Ratio: The number is stored as the ratio of two integers. These integers also have their limits. These integers can be stored in a fixed number of bits or in a vector of digits. While the vector of digits method provides for very large integers, they cannot truly represent a repeating or transcendental number as those numbers do not ever have a fixed length.


The following examples help to illustrate the nature of this weakness and describe methods or techniques which can be used to mitigate the risk.

Note that the examples here are by no means exhaustive and any given weakness may have many subtle varieties, each of which may require different detection methods or runtime controls.

Example One

Muller's Recurrence is a series that is supposed to converge to the number 5. When running this series with the following code, different implementations of real numbers fail at specific iterations:

fn rec_float(y: f64, z: f64) -> f64
  108.0 - ((815.0 - 1500.0 / z) / y);

fn float_calc(turns: usize) -> f64
  let mut x: Vec<f64> = vec![4.0, 4.25];
  (2..turns + 1).for_each(|number|
    x.push(rec_float(x[number - 1], x[number - 2]));


The chart below shows values for different data structures in the rust language when Muller’s recurrence is executed to 80 iterations. The data structure f64 is a 64 bit float. The data structures I<number>F<number> are fixed representations 128 bits in length that use the first number as the size of the integer and the second size as the size of the fraction (e.g. I16F112 uses 16 bits for the integer and 112 bits for the fraction). The data structure of Ratio comes in three different implementations: i32 uses a ratio of 32 bit signed integers, i64 uses a ratio of 64 bit signed integers and BigInt uses a ratio of signed integer with up to 2^32 digits of base 256. Notice how even with 112 bits of fractions or ratios of 64bit unsigned integers, this math still does not converge to an expected value of 5.

Use num_rational::BigRational;

fn rec_big(y: BigRational, z: BigRational) -> BigRational
    - ((BigRational::from_integer(BigInt::from(815))
    - BigRational::from_integer(BigInt::from(1500)) / z)
    / y)

fn big_calc(turns: usize) -> BigRational
  let mut x: Vec<BigRational> = vec![BigRational::from_float(4.0).unwrap(), BigRational::from_float(4.25).unwrap(),];

  (2..turns + 1).for_each(|number|
    x.push(rec_big(x[number - 1].clone(), x[number - 2].clone()));

Example Two

On February 25, 1991, during the eve of the of an Iraqi invasion of Saudi Arabia, a Scud missile fired from Iraqi positions hit a US Army barracks in Dhahran, Saudi Arabia. It miscalculated time and killed 28 people [REF-1190].

Example Three

Sleipner A, an offshore drilling platform in the North Sea was incorrectly constructed with an underestimate of 50% of strength in a critical cluster of buoyancy cells needed for construction. This led to a leak in buoyancy cells during lowering, causing a seismic event of 3.0 on the Richter Scale and about $700M loss [REF-1190].

See Also

Numeric Errors

Weaknesses in this category are related to improper calculation or conversion of numbers.

Comprehensive CWE Dictionary

This view (slice) covers all the elements in CWE.

Weaknesses without Software Fault Patterns

CWE identifiers in this view are weaknesses that do not have associated Software Fault Patterns (SFPs), as covered by the CWE-888 view. As such, they represent gaps in...

Weaknesses Introduced During Implementation

This view (slice) lists weaknesses that can be introduced during implementation.

Common Weakness Enumeration content on this website is copyright of The MITRE Corporation unless otherwise specified. Use of the Common Weakness Enumeration and the associated references on this website are subject to the Terms of Use as specified by The MITRE Corporation.