Chapter 3 Mathematical statistic Trick
3.1 Normal distribution as exponential family
Theorem: If the density function with the form: \[ f(x) \propto \exp \left\{-\frac{1}{2} x^{T} A x+B x\right\} \]
Then, we have \(X\sim N\left(\left(B A^{-1}\right)^{T}, A^{-1}\right)\)
可以配合5.2 食用更佳
3.2 密度变换公式：
设随机变量 \(X\) 有概率密度函数 \(f(x)\), \(x\in(a, b)\) (\(a, b\) 可以为 \(\infty\)), 而 \(y = g(x)\) 在 \(x \in (a, b)\) 上是严格单调的连 续函数，存在唯一的反函数 \(x = h(y)\), \(y \in (\alpha, \beta)\) 并且 \(h′(y)\) 存在且 连续，那么 \(Y = g(X)\) 也是连续型随机变量且有概率密度函数 \[ p(y)=f(h(y))\left|h^{\prime}(y)\right|, \quad y \in(\alpha, \beta) \]
3.3 Probability mass function:
就是离散的的分布嘛，记差了还行
3.4 Vector to diagonal matrix:
(source)[https://mathoverflow.net/questions/55820/vector-to-diagonal-matrix] Let \(E_i\) be the \(n\times n\) matrix with a 1 on position (i,i) and zeros else where. Similarly let \(e_i\) be a row vector with 1 on position (1,i) and zero else where, then: \[ D=\sum_{i=1}^{n} E_{i} v e_{i} \] D is the diagonal matrix with its entry is vector \(v\).
3.5 Gaussian Integral Trick
From the blog Gaussian integral Trick , there are many aspects in this blog, here is the most useful for me part: \[ \int \exp \left(-a y^{2}+x y\right) d y=\sqrt{\frac{\pi}{a}} \exp \left(\frac{1}{4 a} x^{2}\right) \]
Using this technique, we can prove the “Objective Bayesian Methods for Model Selection: Introduction and Comparison” example under difficulty 3 that vague proper prior usually gives bad answers. Vague parameters usually have strongly influence on the Bayes factor.
“Completion of squares” trick: \[ \frac{1}{2} z^{T} A z+b^{T} z+c=\frac{1}{2}\left(z+A^{-1} b\right)^{T} A\left(z+A^{-1} b\right)+c-\frac{1}{2} b^{T} A^{-1} b \]
Then use the fact from multivariate Gaussian:
\[ \frac{1}{(2 \pi)^{n / 2}|\Sigma|^{1 / 2}} \int_{\mathbf{R}^{n}} \exp \left(-\frac{1}{2}(x-\mu)^{T} \Sigma^{-1}(x-\mu)\right)=1 \] or equivalently \[ \int_{\mathbf{R}^{n}} \exp \left(-\frac{1}{2}(x-\mu)^{T} \Sigma^{-1}(x-\mu)\right)=(2 \pi)^{n / 2}|\Sigma|^{1 / 2} \]
So we have: \[ \int_{x \in \mathbf{R}^{n}} \exp \left(-\frac{1}{2} x^{T} A x-x^{T} b-c\right) d x=\frac{(2 \pi)^{n / 2}}{|A|^{1 / 2} \exp \left(c-b^{T} A^{-1} b\right)} \]
3.6 Asymptotic issues
The case where \(\log \pi(\boldsymbol{\theta})=O_{p}(1)\), then \(n^{-1} \log \pi(\boldsymbol{\theta}) \rightarrow 0\) as \(n\rightarrow \infty\).
Las of large numbers: 为啥是这个形式 \[ n^{-1} \log \left\{f\left(\boldsymbol{X}_{n} | \boldsymbol{\theta}\right) \pi(\boldsymbol{\theta})\right\} \rightarrow \int \log \left\{f(\boldsymbol{x} | \boldsymbol{\theta}) \pi_{0}(\boldsymbol{\theta})\right\} d G(\boldsymbol{x}) \] 这个形式和大数定律联系在哪里。
3.7 Matrix inverse
\[ (A+D)^{-1}=D^{-1}\left(I+A D^{-1}\right)^{-1} \]
If \(n=k\) and \(U=V=I_{n}\) is the identity matrix, then \[ \begin{aligned} (A+B)^{-1} &=A^{-1}-A^{-1}\left(B^{-1}+A^{-1}\right)^{-1} A^{-1} \\ &=A^{-1}-A^{-1}\left(A B^{-1}+I\right)^{-1} \end{aligned} \] Continuing with the merging of the terms of the far right-hand side of the above equation results in Hua’s identity \[ (A+B)^{-1}=A^{-1}-\left(A+A B^{-1} A\right)^{-1} \] Another useful form of the same identity is \[ (A-B)^{-1}=A^{-1}+A^{-1} B(A-B)^{-1} \] which has a recursive structure that yields \[ (A-B)^{-1}=\sum_{k=0}^{\infty}\left(A^{-1} B\right)^{k} A^{-1} \] This form can be used in perturbative expansions where \(B\) is a perturbation of \(A\).
Woodbury matrix identity \[ (A+U C V)^{-1}=A^{-1}-A^{-1} U\left(C^{-1}+V A^{-1} U\right)^{-1} V A^{-1} \]