【Deep Learning】Variational Autoencoder ELBO：优美的数学推导

本文主要是介绍【Deep Learning】Variational Autoencoder ELBO：优美的数学推导，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！

Variational Autoencoder

• In this note, we talk about the generation model, where $x$ represents the given dataset, $z$ represents the latent variable, $\theta ,\varphi$ denote the parameters of models.

Latent Variable Model

• Generate $x$ by latent variable $z$: $p(x,z)=p(x)p(x|z)$
• Training: Maximum likelihood

$$\begin{array}{rlr}L(\theta )& =\sum _{x\in D}\log p(x)& \\ & =\sum _{x\in D}\log \sum _{z}p(x,z;\theta )& \\ & =\sum _{x\in D}\log \sum _{z}q(z)\frac{p(x,z;\theta )}{q(z)}& \text{Important Sampling}\\ & \ge \sum _{x\in D}\sum _{z}q(z)\log \frac{p(x,z;\theta )}{q(z)}& \text{Concavcity of log}\end{array}$$

• Assumption: ${\sum }_{z}q(z)=1$. The summation can be regarded as expectation(just for simplicity)

ELBO

• In the above deriviation, ${\sum }_{z}q(z)\log \frac{p(x,z;\theta )}{q(z)}$ is the Evidence Lower Bound of $\log p(x)$
• When $q(z)=p(z|x;\theta )$,

$$\sum _{z}q(z)\log \frac{p(x,z;\theta )}{q(z)}=\log p(x;\theta )$$

• We can set $q(z)=p(z|x;\theta )$ to optimize a tight lowerbound of $\log p(x;\theta )$

  • We call $p(z|x;\theta )$ posterior.
  • Don’t know $p(z|x;\theta )$? Use network $q(z;\varphi )$ to paratermize $p(z|x)$.
  • Optimize $q(z;\varphi )\approx p(z|x;\theta )$ and $p(x|z;\theta )$ alternatively.

• Since we use $q(z;\varphi )$ to approximate $p(z|x;\theta )$, what is the distance metric between them?

  • $KL(q||p)={\sum }_{z}q(z)\log \frac{q(z)}{p(z)}$
    • Compared to $KL(p||q)$, $KL(q||p)$ is reverse KL.
      • Empirically, We often use $KL(p||q)$, where $p$ is the groundtruth distribution, that’s why $KL(q||p)$ is ‘reverse’.
  • We call the procedure to find such $\varphi$ by Variational Inference: ${\min }_{\varphi }KL(q||p)$.

• Look at the optimization of $KL(q||p)$:
  $$\begin{array}{rl}KL(q(z;\varphi )||p(z|x))& =\sum _{z}q(z;\varphi )\log \frac{q(z;\varphi )}{p(z|x)}\\ & =\sum _{z}q(z;\varphi )\log \frac{q(z;\varphi )p(x)}{p(z,x)}\\ & =\log p(x)-\sum _{z}q(z;\varphi )\log \frac{p(z,x)}{q(z;\varphi )}\end{array}$$
  Amazing! ${\sum }_{z}q(z;\varphi )\log \frac{p(z,x)}{q(z;\varphi )}$ is just the ELBO! When we minimize $KL(q||p)$, we are also maximizing ELBO, which means the objective we alternatively trained for $p(x|z;\theta )$ and $q(z;\varphi )$ is magically the same!

  What’s more, we can also find that
  $$\log p(x)=KL(q(z;\varphi )||p(z|x))+ELBO=ApproxError+ELBO$$
  which verifies that ELBO is the lowerbound of $\log p(x)$, and there difference is exactly the approximate error between $q(z;\varphi )$ and $p(z|x)$.

• Notice: $q(z;\varphi )\approx p(z|x,\theta )$. $q$ depends on $x$, hence we can use $q(z|x;\varphi )$ instead of $q(z;\varphi )$, named Amortized Variational Inference.

• Now, only ELBO is our only joint objective. Train $\theta ,\varphi$ together!
  $$\begin{array}{rl}J(\theta ,\varphi ;x)& =\sum _{z}q(z|x;\varphi )\log \frac{p(x,z;\theta )}{q(z|x;\varphi )}\\ & =\sum _{z}q(z|x;\varphi )\left(\log p(x|z;\theta )+\log p(z;\theta )-\log q(z|x;\varphi )\right)\\ & =\sum _{z}q(z|x;\varphi )\log p(x|z;\theta )-\sum _{z}q(z|x;\varphi )\frac{\log q(z|x;\varphi )}{\log p(z;\theta )}\\ & ={\mathbb{E}}_{z\sim q(\cdot |x;\varphi )}\log p(x|z;\theta )-KL(q(z|x;\varphi )||p(z;\theta ))\end{array}$$

VAE

• Pratically, we obtain VAE from $ELBO$.

• Assume that
  $$\begin{gathered} p(z)\sim N(0,I) \\ q(z|x;\varphi )\sim N({\mu }_{\varphi }(x),{\sigma }_{\varphi }(x)) \\ p(x|z;\theta )\sim N({\mu }_{\theta }(z),{\sigma }_{\mu }(z)) \end{gathered}$$
  They are all Gaussian, where the mean and variance are from net work.

• Let $q(z|x;\varphi )$ be the encoder, $p(x|z;\theta )$ be the decoder, then ${\mathbb{E}}_{z\sim q(\cdot |x;\varphi )}\log p(x|z;\theta )$ represents reconstruction error:

  • The error after encoding into latent space, then decoding into the original space.
  • We wish this term big, so that the original data can be recovered with high probability.

• Re-parameterization trick:

  • In ${\mathbb{E}}_{z\sim q(\cdot |x;\varphi )}\log p(x|z;\theta )$ term, $\varphi$ is the sampling parameters, whose gradient can’t be computed.
  • Sample $z^{\prime }\sim N(0,I)$, then compute $z=\mu +z^{\prime }\cdot \sigma$.

Conclusion

• The amazing and elegent mathematical deviation behind VAE inspires me to write down this blog.
• Furthermore, VAE shows its great stability through many tasks, compared to GAN. There are still more Pro and Cons to talk about.

这篇关于【Deep Learning】Variational Autoencoder ELBO：优美的数学推导的文章就介绍到这儿，希望我们推荐的文章对编程师们有所帮助！

---