Deep Dive into Floating Point Numbers - Storage and Arithmetic Principles

published: 8/7/2025 and updated: 8/7/2025

Computer Systems Computer Systems , IEEE 754 , Floating Point

Preface#

In computer science, floating point numbers serve as our primary method for handling real numbers. Whether in scientific computing, graphics processing, or everyday numerical operations, floating point numbers play a central role. However, many programmers lack a deep understanding of the internal representation and arithmetic mechanisms of floating point numbers, which often leads to mysterious bugs and precision issues.

At the same time, floating point arithmetic is fundamental knowledge for the quantization process in deep learning.

This article will delve deep into floating point storage formats under the IEEE 754 standard, including single precision and double precision, as well as their arithmetic principles and common pitfalls.

Since my background focuses on low-level systems, this article will involve considerable binary representation and hardware implementation details. Please bear with me if there are any oversights.

IEEE 754 Standard Overview#

IEEE 754 is the widely adopted floating point representation standard that defines binary representation formats, arithmetic rules, and special value handling for floating point numbers. The core idea of this standard is to represent real numbers in scientific notation format:

This representation method can represent extremely large or small numerical ranges within a finite number of bits, but at the cost of precision loss.

Format Comparison#

IEEE 754 defines multiple floating point formats, with 32-bit single precision and 64-bit double precision being the most commonly used:

Single Precision Floating Point (32-bit)#

Storage Format#

Single precision floating point uses 32 bits to store a real number, with the following specific allocation:

Bit 31    Bits 30-23      Bits 22-0
 S        E E E E E E E E      M M M M M M M M M M M M M M M M M M M M M M M
Sign Bit     8-bit Exponent      23-bit Mantissa

Component Meanings#

Sign Bit#

• Bit 31: 0 represents positive, 1 represents negative
• Only determines the sign of the value

Exponent Part#

• Bits 30-23: 8-bit unsigned integer representing the exponent
• Uses biased encoding with a bias of 127
• Actual exponent = Stored value - 127
• Range: -126 to +127 (0 and 255 are reserved for special values)

Mantissa Part (Significand)#

• Bits 22-0: 23-bit fractional part
• Uses implicit leading 1 technique
• Actual mantissa = 1.M₂₂M₂₁…M₀ (binary)
• Provides approximately 7 decimal digits of precision

Numerical Calculation Example#

Let’s analyze the value represented by the 32-bit floating point number 0x42280000:

Hexadecimal: 42280000
Binary:      01000010001010000000000000000000

Bit Field Breakdown:

• Sign bit: 0 (positive)
• Exponent: 10000100₂ = 132₁₀
• Mantissa: 01010000000000000000000₂

Calculation Process:

1. Actual exponent = 132 - 127 = 5
2. Complete mantissa = 1.01010000000000000000000₂ = 1.3125₁₀
3. Final result = (+1) × 1.3125 × 2⁵ = 1.3125 × 32 = 42.0

Double Precision Floating Point (64-bit)#

Storage Format#

Double precision floating point uses 64 bits for storage, providing higher precision and larger numerical range:

Bit 63   Bits 62-52           Bits 51-0
 S       E E E E E E E E E E E    M M M M M M M M ... M M M M (52 bits)
Sign Bit    11-bit Exponent        52-bit Mantissa

Key Characteristics#

Extended Exponent Range#

• 11-bit exponent: Supports larger numerical range
• Bias: 1023
• Actual exponent range: -1022 to +1023

Higher Precision#

• 52-bit mantissa: Provides approximately 15-16 decimal digits of precision
• Implicit leading 1 technique, actual precision is 53 binary bits

Storage Advantages#

• Representation range: approximately ±1.7 × 10³⁰⁸
• Smallest normalized number: approximately 2.2 × 10⁻³⁰⁸
• Machine epsilon ε: approximately 2.22 × 10⁻¹⁶

Special Value Handling#

The IEEE 754 standard defines several special floating point values to handle exceptional situations:

Zero Values#

+0.0: S=0, E=00000000, M=00000000000000000000000 (32-bit)
-0.0: S=1, E=00000000, M=00000000000000000000000 (32-bit)

Infinity#

+∞: S=0, E=11111111, M=00000000000000000000000
-∞: S=1, E=11111111, M=00000000000000000000000

Not a Number (NaN)#

NaN: S=X, E=11111111, M≠00000000000000000000000

NaN is used to represent undefined operation results, such as:

• 0/0
• ∞ - ∞
• √(-1)

Floating Point Arithmetic Principles#

Addition Operation Steps#

Floating point addition is much more complex than integer addition, requiring the following steps:

1. Exponent Alignment: Adjust the exponent of the smaller number to match the larger one
2. Mantissa Addition: Perform addition on the aligned mantissas
3. Normalization: Adjust the result to comply with IEEE 754 format
4. Rounding: Handle excess precision bits according to rounding rules

Calculation Example#

Calculate 3.25 + 1.125:

Step 1: Convert to Binary Scientific Notation

• 3.25 = 1.101₂ × 2¹
• 1.125 = 1.001₂ × 2⁰

Step 2: Exponent Alignment

• 1.125 = 0.1001₂ × 2¹ (shift right by 1 bit)

Step 3: Mantissa Addition

• 1.101₂ + 0.1001₂ = 10.0011₂

Step 4: Normalization

• 10.0011₂ × 2¹ = 1.00011₂ × 2²

Result: 4.375

Multiplication Operation#

Floating point multiplication steps are relatively simple:

1. Sign Calculation: Result sign = XOR of operand signs
2. Exponent Addition: Result exponent = Exponent1 + Exponent2 - Bias
3. Mantissa Multiplication: Calculate the product of mantissas
4. Normalization and Rounding: Adjust result format

Precision Issues and Pitfalls#

Representation Error#

Not all decimal fractions can be precisely represented in binary floating point. For example:

float x = 0.1f;
printf("%.17f\n", x);  // Output: 0.10000000149011612

This is because the binary representation of 0.1 is infinitely repeating: 0.1₁₀ = 0.000110011001100…₂

Cumulative Computational Error#

Due to the existence of rounding errors, continuous floating point operations may lead to error accumulation:

double sum = 0.0;
for (int i = 0; i < 10; i++) {
    sum += 0.1;
}
printf("%.17f\n", sum);  // May not equal 1.0

Comparison Pitfalls#

Direct comparison of floating point numbers for equality is dangerous, because the aforementioned rounding errors cause two floating point numbers that are mathematically equal to have different binary representations in computers:

// Wrong approach
if (a == b) { ... }

// Correct approach
const double EPSILON = 1e-9;
if (fabs(a - b) < EPSILON) { ... }

Hardware Implementation Considerations#

Floating Point Unit (FPU)#

Modern processors typically include dedicated Floating Point Units (FPUs) to accelerate floating point operations:

• Pipeline Design: Multi-stage pipelines process different operation stages in parallel
• Dedicated Registers: Independent floating point register files
• SIMD Support: Single Instruction Multiple Data parallel processing

Performance Optimization#

Fast Inverse Square Root#

The famous Quake III fast inverse square root algorithm leverages the characteristics of the IEEE 754 format:

float Q_rsqrt( float number )
{
        long i;
        float x2, y;
        const float threehalfs = 1.5F;
        x2 = number * 0.5F;
        y = number;
        i = * ( long * ) &y;                       // evil floating point bit level hacking
        i = 0x5f3759df - ( i >> 1 );               // what the fuck?
        y = * ( float * ) &i;
        y = y * ( threehalfs - ( x2 * y * y ) );   // 1st iteration
//      y = y * ( threehalfs - ( x2 * y * y ) );   // 2nd iteration, this can be removed
        return y;
}

This algorithm cleverly utilizes the internal representation of floating point numbers through bit operations to achieve fast approximate calculations.

Summary#

This article has deeply explored the storage formats and arithmetic principles of IEEE 754 floating point numbers. We have learned that:

Floating point numbers are approximate representations of real numbers that provide wide-range numerical representation capabilities while introducing precision limitations and computational errors. Understanding the internal mechanisms of floating point numbers is crucial for writing high-quality numerical computation programs.

Key points include:

• The three components of IEEE 754 format: sign bit, exponent, mantissa
• Differences between single and double precision and their applicable scenarios
• Representation and handling of special values (zero, infinity, NaN)
• The complexity and error sources of floating point operations