Chapter 5 Statistician Tool Box

5.1 Matrix algebra

Definition 5.1 定义：矩阵\(A(t)\)的导数 如果矩阵\(A(t)=\left(a_{i j}(t)\right)_{m \times n}\) 的每一个元素\(a_{ij}(t)\)是变量t的可微函数，则称A(t)可微，其导数（微商）定义为 \[ A^{\prime}(t)=\frac{\mathrm{d}}{\mathrm{d} t} A(t)=\left(\frac{\mathrm{d}}{\mathrm{d} t} a_{i j}(t)\right)_{m \times n} \]

Definition 5.2 定义：矩阵对矩阵的导数 设 \(\boldsymbol{X}=\left(\xi_{i j}\right)_{m \times n}\), \(mn\) 元函数 \(f(\boldsymbol{X})=f\left(\xi_{11}, \xi_{12}\cdots, \xi_{1 n}, \xi_{21}, \cdots, \xi_{m n} )\right.\),定义\(f(X)\) 对矩阵\(X\)的导数为

\[\begin{equation} \frac{\mathrm{d} f}{\mathrm{d} \boldsymbol{X}}=\left(\frac{\partial f}{\partial \xi_{i j}}\right)_{m \times n}= \left[ \begin{array}{ccc} {\frac{\partial f}{\partial \xi_{11}}} & {\dots} & {\frac{\partial f}{\partial \xi_{1 n}}} \\ {\vdots} & { } & {\vdots} \\ {\frac{\partial f}{\partial \xi_{m 1}}} & {\cdots} & {\frac{\partial f}{\partial \xi_{m n}}} \end{array}\right] \end{equation}\]

根据这个定义，能够直接证明以下两个最常见的结论 \[ \frac{\partial x'a}{\partial x}=a=\frac{\partial a'x}{\partial x} \]

\[ \frac{\partial x'Ax}{\partial x}=(A+A')x \]

证明： 将矩阵乘法\(x^TAx\)打开得到 \[ \begin{aligned} f(\boldsymbol{x})=& \sum_{i=1}^{n} \sum_{j=1}^{n} a_{i j} \xi_{i} \xi_{j}=\\ & \xi_{1} \sum_{j=1}^{n} a_{1 j} \xi_{j}+\cdots+\xi_{k} \sum_{j=1}^{n} a_{k j} \xi_{j}+\cdots+\xi_{n} \sum_{j=1}^{n} a_{n j} \xi_{j} \end{aligned} \] 且有 \[ \begin{align} \frac{\partial f}{\partial \xi_{k}}=\xi_{1} a_{1 k}+\cdots+\xi_{k-1} a_{k-1, k}+\left(\sum_{j=1}^{n} a_{k j} \xi_{j}+\xi_{k} a_{k k}\right)+\\ \xi_{k+1} a_{k+1, k}+\cdots+\xi_{n} a_{\# k}=\sum_{j=1}^{n} a_{i j} \xi_{j}+\sum_{i=1}^{n} a_{k} \xi_{i} \end{align} \] 所以有 \[ \frac{d f}{d x}=\left[ \begin{array}{c}{\frac{\partial f}{\partial \xi_{1}}} \\ {\vdots} \\ {\frac{\partial f}{\partial \xi_{n}}}\end{array}\right]=\left[ \begin{array}{c}{\sum_{j=1}^{n} a_{1 j} \xi_{j}} \\ {\vdots} \\ {\sum_{j=1}^{n} a_{n j} \xi_{j}}\end{array}\right]+\left[ \begin{array}{c}{\sum_{i=1}^{n} a_{i 1} \xi_{i}} \\ {\vdots} \\ {\sum_{i=1}^{n} a_{i n} \xi_{i}}\end{array}\right]=\\ \boldsymbol{A x}+\boldsymbol{A}^{\mathrm{T}} \boldsymbol{x}=\left(\boldsymbol{A}+\boldsymbol{A}^{\mathrm{T}}\right) \boldsymbol{x} \]

5.1.1 Block diagonal matrices

\[ \mathbf{A}=\left[ \begin{array}{cccc}{\mathbf{A}_{1}} & {0} & {\cdots} & {0} \\ {0} & {\mathbf{A}_{2}} & {\cdots} & {0} \\ {\vdots} & {\vdots} & {\ddots} & {\vdots} \\ {0} & {0} & {\cdots} & {\mathbf{A}_{n}}\end{array}\right] \]

property: \[ \begin{aligned} \operatorname{det} \mathbf{A} &=\operatorname{det} \mathbf{A}_{1} \times \cdots \times \operatorname{det} \mathbf{A}_{n} \\ \operatorname{tr} \mathbf{A} &=\operatorname{tr} \mathbf{A}_{1}+\cdots+\operatorname{tr} \mathbf{A}_{n} \end{aligned} \]

It’s inverse:

\[ \left( \begin{array}{cccc}{\mathbf{A}_{1}} & {0} & {\cdots} & {0} \\ {0} & {\mathbf{A}_{2}} & {\cdots} & {0} \\ {\vdots} & {\vdots} & {\ddots} & {\vdots} \\ {0} & {0} & {\cdots} & {\mathbf{A}_{n}}\end{array}\right)=\left( \begin{array}{cccc}{\mathbf{A}_{1}^{-1}} & {0} & {\cdots} & {0} \\ {0} & {\mathbf{A}_{2}^{-1}} & {\cdots} & {0} \\ {\vdots} & {\vdots} & {\ddots} & {\vdots} \\ {0} & {0} & {\cdots} & {\mathbf{A}_{n}^{-1}}\end{array}\right) \]

5.2 两个二次型相加

In vector formulation(Assuming \(\Sigma_1\),\(\Sigma_2\) are symmetric) 有： \[ \begin{array}{c}{-\frac{1}{2}\left(\mathbf{x}-\mathbf{m}_{1}\right)^{T} \mathbf{\Sigma}_{1}^{-1}\left(\mathbf{x}-\mathbf{m}_{1}\right)} \\ {-\frac{1}{2}\left(\mathbf{x}-\mathbf{m}_{2}\right)^{T} \mathbf{\Sigma}_{2}^{-1}\left(\mathbf{x}-\mathbf{m}_{2}\right)} \\ {=-\frac{1}{2}\left(\mathbf{x}-\mathbf{m}_{c}\right)^{T} \mathbf{\Sigma}_{c}^{-1}\left(\mathbf{x}-\mathbf{m}_{c}\right)+C}\end{array} \] 其中： \[ \begin{aligned} \mathbf{\Sigma}_{c}^{-1}=& \mathbf{\Sigma}_{1}^{-1}+\mathbf{\Sigma}_{2}^{-1} \\ \mathbf{m}_{c} &=\left(\mathbf{\Sigma}_{1}^{-1}+\mathbf{\Sigma}_{2}^{-1}\right)^{-1}\left(\mathbf{\Sigma}_{1}^{-1} \mathbf{m}_{1}+\mathbf{\Sigma}_{2}^{-1} \mathbf{m}_{2}\right) \\ C &=\frac{1}{2}\left(\mathbf{m}_{1}^{T} \mathbf{\Sigma}_{1}^{-1}+\mathbf{m}_{2}^{T} \mathbf{\Sigma}_{2}^{-1}\right)\left(\mathbf{\Sigma}_{1}^{-1}+\mathbf{\Sigma}_{2}^{-1}\right)^{-1}\left(\mathbf{\Sigma}_{1}^{-1} \mathbf{m}_{1}+\mathbf{\Sigma}_{2}^{-1} \mathbf{m}_{2}\right) \\ &-\frac{1}{2}\left(\mathbf{m}_{1}^{T} \mathbf{\Sigma}_{1}^{-1} \mathbf{m}_{1}+\mathbf{m}_{2}^{T} \mathbf{\Sigma}_{2}^{-1} \mathbf{m}_{2}\right) \end{aligned} \]

而对于我需要的情况，简化\(m_1=0\),\(m_2=0\)

那么就有，只需要\(\mathbf{\Sigma}_{c}^{-1}=\mathbf{\Sigma}_{1}^{-1}+\mathbf{\Sigma}_{2}^{-1}\),同时\(m_c=0\).

5.3 正定阵的基础定理：

Theorem 1

For a \(p \times p\) symmetric matrix \(\Sigma=\left(\sigma_{i j}\right),\) the following are equivalent:

  1. \(\Sigma\) is nonnegative definite.

  2. all its leading principal minors are nonnegative definite, that is, the \(i \times i\) matrices \[ \boldsymbol{\Sigma}_{i i}=\left(\begin{array}{ccc} {\sigma_{11}} & {\cdots} & {\sigma_{1 i}} \\ {\vdots} & {\ddots} & {\vdots} \\ {\sigma_{i 1}} & {\cdots} & {\sigma_{i i}} \end{array}\right), i=1, \cdots, p \] are nonnegative definite.

  3. all eigenvalues of \(\Sigma\) are nonnegative.

  4. there exists a matrix A such that \[ \boldsymbol{\Sigma}=A A^{\prime} \]

  5. there exists a lower triangular matrix \(L\) such that \[ \boldsymbol{\Sigma}=L L^{\prime} \]

  6. there exist vectors \(\boldsymbol{u}_{1}, \cdots, \boldsymbol{u}_{p}\) in \(R^{p}\) such that \(\sigma_{i j}=\boldsymbol{u}_{i}^{\prime} \boldsymbol{u}_{j}\)

5.4 对称阵的谱分解相关定理

Theorem 4 (The Spectral Decomposition) Let \(A\) be a \(p \times p\) symmetric matrix with p pairs of eigenvalues and eigenvectors \[ \left(\lambda_{1}, \mathbf{e}_{1}\right),\left(\lambda_{2}, \mathbf{e}_{2}\right), \cdots,\left(\lambda_{p}, \mathbf{e}_{p}\right) \] Then,

  1. The eigenvalues \(\lambda_{1}, \ldots, \lambda_{p}\) are all real, and can be ordered from the largest to the smallest \[ \lambda_{1} \geq \lambda_{2} \geq \ldots \geq \lambda_{p} \]

  2. The normalized eigenvectors \(\mathbf{e}_{1}, \ldots, \mathbf{e}_{p}\) are mutually orthogonal and the matrix \[ P=\left(\mathbf{e}_{1}, \mathbf{e}_{2}, \ldots, \mathbf{e}_{p}\right) \] is an orthogonal matrix, that is, \[ P P^{\prime}=P^{\prime} P=I \]

  3. The spectral decomposition of \(A\) is the expansion \[ A=\lambda_{1} \mathbf{e}_{1} \mathbf{e}_{1}^{\prime}+\lambda_{2} \mathbf{e}_{2} \mathbf{e}_{2}^{\prime}+\cdots+\lambda_{p} \mathbf{e}_{p} \mathbf{e}_{p}^{\prime}=P \Lambda P^{\prime} \] where \(P\) is as above and \[ \Lambda=\operatorname{diag}\left(\lambda_{1}, \cdots, \lambda_{p}\right) \] is a diagonal matrix with \(\lambda_{1}, \ldots, \lambda_{p}\) as its respective diagonal entries.

  4. The matrix \(A\) is nonnegative definite, if and only if all its eigenvalues are nonnegative.

5.5 Covariance Structure

5.5.1 Compound Symmetry Covariance结构

P54 of pourahmadi covariance book

a sufficient condition for nonnegative definite is : \(1+(p-1) \rho \geq 0\) or \(-(p-1)^{-1} \leq \rho \leq 1\) \[ \boldsymbol{\Sigma}=\sigma^{2} R=\sigma^{2}\left(\begin{array}{cccc} 1 & \rho & \cdots & \rho \\ \rho & 1 & \cdots & \rho \\ \vdots & \vdots & \ddots & \vdots \\ \rho & \rho & \cdots & 1 \end{array}\right)=\sigma^{2}\left[(1-\rho) I+\rho \mathbf{1}_{n} \mathbf{1}_{n}^{\prime}\right] \]

5.5.2 Huynh-Feldt Structure

\(\Sigma=\sigma^{2}\left(\alpha I+a \mathbf{1}_{p}^{\prime}+\mathbf{1}_{p} a^{\prime}\right)\)

The \(\Sigma\) is nonnegative definite provided that \[ \alpha>\mathbf{1}_{p}^{\prime} a-\sqrt{p a^{\prime} a} \]

5.5.3 The One-Dependent Covariance Structure

\[ \Sigma=\sigma^{2}\left(\begin{array}{ccccc} {a} & {b} & {0} & {\cdots} & {0} \\ {b} & {a} & {b} & {\ddots} & {\vdots} \\ {\vdots} & {\ddots} & {\ddots} & {\ddots} & {0} \\ {\vdots} & {} & {\ddots} & {\ddots} & {b} \\ {0} & {\cdots} & {0} & {b} & {a} \end{array}\right) \]