Lecture 9 VAE variants 40mins - GitHub Pages

VAE variantsHao Dong

Peking University

• Convolutional VAE• Conditional VAE• β-VAE• IWAE• Ladder VAE• Progressive + Fade-in VAE• VAE in speech• Temporal Difference VAE (TD-VAE)

VAE variants

Hierarchical representation learning

Temporal representa2on learning

Representation learning

• Convolutional VAE• Conditional VAE• IWAE• β-VAE• Ladder VAE• Progressive + Fade-in VAE• VAE in speech• Temporal Difference VAE (TD-VAE)

VAE variants

Temporal representation learning

Convolutional Variational Autoencoder

• Limitations of vanilla VAE• The size of weight of fully connected layer == input size x output size• If VAE uses fully connected layers only, will lead to curse of dimensionality

when the input dimension is large (e.g., image).

• Solution

Image is modified from: Deep Clustering with Convolutional Autoencoder. NIPS 2017.

Fully connected layers here(independent variables)

VAE variants

Hierarchical representa2on learning

Conditional Variational Autoencoder

• Train and inference with labelled data.

Learning structured output representation using deep conditional generative models. NeurIPS 2015.

Recap: Variational Autoencoder

Auto-Encoding Variational Bayes. Diederik P. Kingma, Max Welling. ICLR 2013

• Recap: Se>ng up the objecAve• Maximise P(X)• Set Q(z) to be an arbitrary distribuGon

Recap: Variational Autoencoder

Auto-Encoding Variational Bayes. Diederik P. Kingma, Max Welling. ICLR 2013

• Recap: Setting up the objective

encoder ideal reconstruction KLD

Condi6onal Varia6onal Autoencoder

• Setting up the objective with labels• Maximise P(Y|X)• Set Q(z) to be an arbitrary distribution

• Setting up the objective

encoder ideal

reconstruction KLD

• Train and inference without labelled data i.e., vanilla VAE

• Train and inference with labeled data.

VAE variants

Before we start

• Disentangled / Factorised representation• Each variable in the inferred latent representation is only sensitive to one

single generative factor and relatively invariant to other factors• Good interpretability and easy generalization to a variety of tasks

β-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. ICLR 2017.

Before we start

• Unsupervised hierarchical representation learning

Rethinking Style and Content Disentanglement in Variational Autoencoders. ICLR 2018 Workshops.

β-VAE

• Unsupervised representation learning• Augment the original VAE framework with a single hyper-parameter β

that modulates the learning constraints• Impose a limit on the capacity of the latent information channel

β-VAE

• Discussion: It is really unsupervised?

• It is unsupervised/self-supervised learning, because it does not need any label data

• It is not fully unsupervised learning, it works because of the inductive bias of the neural network model, the hierarchical design introduces prior knowledge about the data

VAE variants

IWAE（Importance Weighted Autoencoder）

Importance Weighted Autoencoders. ICLR 2016.

• Optimise a tighter lower bound than VAE• VAE just optimises a lower bound of log P(X)

encoder ideal reconstruction KLD

• Optimise a tighter lower bound than VAE

• Why “Importance weighted”

VAE variants

Ladder VAE

• To learn hierarchical latent representation• Deep models with several layers of dependent stochastic variables are difficult to train

• Limiting the improvements obtained using these highly expressive models

LVAE: Ladder Variational Autoencoder. Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. NIPS 2016.

Ladder VAE

encode decode encode decode

Hierarchical VAE Ladder VAE

Ladder VAE

VAE variants

Progressive + Fade-in VAE

• Discussion

Progressive Learning and Disentanglement of Hierarchical Representations. ICLR 2020.

encoders decoders

high-level

middle-level

low-level

Can we directly train a hierarchical VAE with ladder structure like that?

• Discussion

encoders decoders

high-level

middle-level

low-level

Information SHORTCUT problemall information go through the low-level path,other paths are ignored.model is lazy…

• Progressive + Fade-in

Stage 1: enable high-level path Stage 2: enable high-level and middle paths

Stage 3: enable all paths …

𝛼 increase from 0 to 1 gradually

• Results

High-level: background and foreground colors

Middle-level: shape

Low-level: …

• Results

High-level Middle-level Low-level

VAE variants

VAE in speech

• Learning latent representations for style control and transfer in end-to-end speech synthesis

• RNN as encoder

Learning latent representations for style control and transfer in end-to-end speech synthesis. ICASSP 2019.

VAE variants

TD-VAE

• State-space model as a Markov Chain model

Temporal Difference VAE. ICLR 2019.

TD-VAE

Summary

Thanks

Lecture 9 VAE variants 40mins - GitHub Pages

Documents

Transcript of Lecture 9 VAE variants 40mins - GitHub Pages

EFFECTS OF MILK PROTEIN VARIANTS ON THE PROTEIN RECOVERY ...old.eaap.org/Previous_Annual_Meetings/2004Bled/papers/GPH1.8_H... · EFFECTS OF MILK PROTEIN VARIANTS ON THE PROTEIN RECOVERY

Deep Neural Networks - GitHub Pages

The Teaching of the Apostles (Didache) A Critical Greek Edition with footnotes covering textual variants Compiled by David Robert Palmer

18.175: Lecture 16 limit theorem variants · 18.175: Lecture 16 Central limit theorem variants Scott Sheﬃeld. MIT. 18.175. Lecture 16. 1. Outline. CLT idea. CLT variants. 18.175

Lecture 9 VAE variants 50mins - Deep Generative Models

Σεμινάριο Git & GitLab · –Διορθωμένο CVS για project-wide management •git – 2005 –Distributed version control system •GitHub – 2008 (open source

Molecular Fingerprint VAE - icml-compbio.github.io

Geometric Topology and Geometry of Banach Spacesarkady/Eilat_2017_conference/booklet... · Geometric Topology and Geometry of Banach Spaces ... Variants of decomposition complexity

Abhishek Kumar, Ben Pooleno... · Abhishek Kumar, Ben Poole Google Research, Brain Team ICML 2020. Conﬁdential & Proprietary β-VAE Encoder Decoder How does variational family regularize

VAE: Encoding stochastic process priors with variational ... · In theory, they provide ﬂexible priors over function classes that can encode a wide range of interesting as- ...

Status Report - GitHub Pages

Informatique tronc commun - GitHub Pages · Informatique tronc commun Problème no1 22 mai 2018 I. Analysepréliminaire:ﬁltragenumérique I.1. Filtrenumériquepassebasdupremierordre

Effects of cow’s milk beta-casein variants on …...Background β-casein is a major protein component of cow’s milk, and numerous variants have been described, including the A1

2018.10.09 From Autoencoders to β-VAE Emir Konuk

OnSite FOB Rapid Test. Hemoglobin Variants Hemoglobin variantsHemoglobin variants are a part of the normal embryonicembryonic and fetal development, but.

Cross-connections and variants of TX · Variants of T XCross-connectionsSubsets and partitions Reg(T ) Biordered setSummary Cross-connections and variants of T X Azeef Muhammed P.

ARTICULATED VEHICLE MODEL - GitHub Pagesandresmendes.github.io/Vehicle-Dynamics-Lateral/theory/vehicleArticulated.pdf · VEHICLE DYNAMICS - LATERAL ANDRÉ DE SOUZA MENDES ARTICULATED

Propiedades globales de curvas planas - GitHub Pages

Genotyping and protein profiling of milk β-casein variants ...

Chapter 8 - db.inf.uni-tuebingen.de · Operator Pipelining Volcano Iterator Model 3 Operator Variants Specic operator variants may be tailored to exploit physical properties of its