Gaussian Distribution With a Diagonal Covariance Matrix

Gaussian Distribution With a Diagonal Covariance Matrix

Often, it is convenient to use an alternative representation of a multivariate Gaussian distribution if it is known that the off-diagonals of the covariance matrix only play a minor role. In this case one can assume to have only a diagonal covariance matrix and one can estimate the mean and the variance in each dimension separately and describe the multivariate density function in terms of a product of univariate Gaussians. This is shown in the following.

Assume we have a diagonal Covariance Matrix in the following form:

\begin{align} \Sigma = \begin{pmatrix} \sigma_1^2 & 0 & \cdots & 0
0 & \sigma_2^2 & \cdots & 0
\vdots & \vdots & \ddots & \vdots
0 & 0 & \cdots & \sigma_k^2
\end{pmatrix} \end{align}

The density function of a multivariate Gaussian is defined as:

\begin{align} f_x(x_1, \ldots, x_k) = \frac{\exp \Big(-\frac{1}{2} (\vec{x}-\vec{\mu})^T \Sigma^{-1} (\vec{x}-\vec{\mu}) \Big)}{\sqrt{|2\pi\Sigma|}} \end{align} Let

\begin{align} \vec{y} = \vec{x}-\vec{\mu} \end{align} Then we get: \begin{align} f_x(x_1, \ldots, x_k) = \frac{\exp \Big(-\frac{1}{2} \vec{y}^T \Sigma^{-1} \vec{y} \Big)}{\sqrt{|2\pi\Sigma|}} \end{align}

Remember that the inverse of a diagonal matrix is as follows:

Hence, we get:

\begin{align} f_x(x_1, \ldots, x_k) = \frac{\exp \Big(-\frac{1}{2} ( \frac{y_1^2}{\sigma_1^2} + \frac{y_2^2}{\sigma_2^2} + \ldots + \frac{y_k^2}{\sigma_k^2} ) \Big)}{\sqrt{(2\pi)^{k}|\Sigma|}} \end{align}

The determinant of a diagonal matrix is:

This leads to:

Markus Thill

Markus Thill
I studied computer engineering (B.Sc.) and Automation & IT (M.Eng.). Generally, I am interested in machine learning (ML) approaches (in the broadest sense), but particularly in the fields of time series analysis, anomaly detection, Reinforcement Learning (e.g. for board games), Deep Learning (DL) and incremental (on-line) learning procedures.

Deriving a Closed-Form Solution of the Fibonacci Sequence using the Z-Transform

The Fibonacci sequence might be one of the most famous sequences in the field of mathmatics and computer science. Already high school stu...… Continue reading