Mean square quantization error (MSQE) is a figure of merit for the process of analog to digital conversion.

In this conversion process, analog signals in a continuous range of values are converted to a discrete set of values by comparing them with a sequence of thresholds. The quantization error of a signal is the difference between the original continuous value and its discretization, and the mean square quantization error (given some probability distribution on the input values) is the expected value of the square of the quantization errors.

Mathematically, suppose that the lower threshold for inputs that generate the quantized value ${\displaystyle q_{i))$ is ${\displaystyle t_{i-1))$, that the upper threshold is ${\displaystyle t_{i))$, that there are ${\displaystyle k}$ levels of quantization, and that the probability density function for the input analog values is ${\displaystyle p(x)}$. Let ${\displaystyle {\hat {x))}$ denote the quantized value corresponding to an input ${\displaystyle x}$; that is, ${\displaystyle {\hat {x))}$ is the value ${\displaystyle q_{i))$ for which ${\displaystyle t_{i}-1\leq x<t_{i))$. Then

${\displaystyle {\begin{aligned}\operatorname {MSQE} &=\operatorname {E} [(x-{\hat {x)))^{2}]\\&=\int _{t_{0))^{t_{k))(x-{\hat {x)))^{2}p(x)\,dx\\&=\sum _{i=1}^{k}\int _{t_{i-1))^{t_{i))(x-q_{i})^{2}p(x)\,dx.\end{aligned))}$