CARD: Computer arithmetic

Integer multiplication

There is a very clear explanation of the detailed implementation of sign extension in Booth-coded multipliers in the Appendix of Gary Bewick's 1994 PhD thesis (Stanford CSL-TR-94-617). The whole thesis is also available online.

For a more in-depth comparison of Wallace and Dadda trees (including an explanation of how each type of multiplier is constructed), please consult: Whitney J. Townsend, Earl E. Swartzlander Jr., Jacob A. Abraham, "A comparison of Dadda and Wallace multiplier delays," Proc. SPIE 5205, Advanced Signal Processing Algorithms, Architectures, and Implementations XIII, (24 December 2003); doi: 10.1117/12.507012.

Floating-point arithmetic

An essential read is Goldberg's article "What Every Computer Scientist Should know About Floating-Point Arithmetic" (ACM Computing Surveys, 23 (1), 5-48, 1991).

Compressed numerical formats for AI inference

Large language models (LLMs) contain billions of parameters, each of which is a real-valued weight. To store these weights compactly, they use variants of floating-point formats that vastly reduce the precision and range from the normal FP64 or FP32 down to FP16 or smaller. The Open Compute Project published the OCP Microscaling Formats (MX) Specification version 1.0 in 2023. This formalised numerical formats for 4-bit, 6-bit and 8-bit floating point (FP4, FP6, FP8) and 8-bit integer (INT8) data elements , as well as defining the E8M0 format for values intended to scale a block of data elements.

This was further optimized by NVIDIA in their proprietary NVFP4 format. Here you can see how the NVFP4 data format is used in their Blackwell GPU architecture. Of particular interest is the structured sparsity, which not only reduces storage space for LLM models, but also reduces the computational complexity of tensor arithmetic allowing 4x4 matrix multiplication to be implemented in a single cycle with high efficiency.