A Spectral Regularizer for Unsupervised Disentanglement

Aditya Ramesh, Youngduck Choi, Yann LeCun

Abstract

A generative model with a disentangled representation allows for control over independent aspects of the output. Learning disentangled representations has been a recent topic of great interest, but it remains poorly understood. We show that even forGANs that do not possess disentangled representations, one can find curved trajectories in latent space over which local disentanglement occurs. These trajectories are found by iteratively following the leading right-singular vectors of the Jacobian of the generator with respect to its input. Based on this insight, we describe an efficient regularizer that aligns these vectors with the coordinate axes, and show that it can be used to induce disentangled representations inGANs, in a completely unsupervised manner.

Additional GAN Disentanglement Results

Aditya Ramesh 1 2 Youngduck Choi 3 Yann LeCun 4

A generative model with a disentangled representation allows for control over independent aspects of the output. Learning disentangled representations has been a recent topic of great interest, but it remains poorly understood. We show that even for GANs that do not possess disentangled representations, one can find curved trajectories in latent space over which local disentanglement occurs. These trajectories are found by iteratively following the leading right-singular vectors of the Jacobian of the generator with respect to its input. Based on this insight, we describe an efficient regularizer that aligns these vectors with the coordinate axes, and show that it can be used to induce disentangled representations in GANs, in a completely unsupervised manner.

Introduction

Rapid progress has been made in the development of generative models capable of producing realistic samples from distributions over complex, high-resolution natural images (Karras et al. (2017), Brock et al. (2018)). Despite the success of these models, it is unclear how one can achieve control over the data-generation process when labeled instances are not available. Perturbing an individual component of a latent variable typically results in an unpredictable change to the output. When the model has a disentangled representation, these changes become interpretable, and each component of the latent variable affects a distinct attribute. So far, the problem of learning disentangled reprersentations remains poorly understood. Most approaches for doing this have focused on VAEs (Kingma & Welling (2013), Rezende et al. (2014)), which produce blurry samples in practice. The few approaches that have been developed for GANs, such as InfoGAN (Chen et al., 2016), have had comparably limited

1 OpenAI 2 Continuation of work done at New York University 3 Department of Computer Science, Yale University 4 Department of Computer Science, New York University. Correspondence to: Aditya Ramesh aramesh@openai.com.

success.

Learning disentangled representations is of interest, because they allow us gauge the ability of generative models to benefit downstream tasks. For example, a robotics application may require a vision model that reliably determines the orientation of an object, subject to changes in viewpoint, lighting conditions, and intra-class variation in appearance. A generative model with independent control over the full extent of each of these attributes may help to achieve the level of robustness that is required for these applications. Moreover, the ability to synthesize concepts in ways that are not present in the original data (e.g., rotation of an object to a position not encoutered during training) is a useful benchmark for reasoning and out-of-distribution generalization. Often times, the goal of training a generative model is to observe that this type of generalization occurs.

Following (Higgins et al., 2016), we say a generative model G : Z → X , where Z ⊂ R m and X ⊂ R n , possesses a disentangled representation when it satisfies two important properties. The first property is independence of the components of the latent variable. Roughly speaking, this means that a perturbation to any component of a latent variable should result in a change to the output that is distinct, in some sense, from the changes resulting from perturbations to the other components. For example, if the first component of the latent variable controls hair length, then the other components should not influence this attribute. The

second property is interpretability of the changes in X resulting from these perturbations. Suppose that the process of sampling from the distribution p data modeled by G can be realized by a simulator that is parameterized over a list of independent, scalar-valued factors or attributes. These attributes might correspond to concepts such as lighting, azimuth, gender, age, and so on. This property is met when G ( z + αe i ) results in a change along exactly one of these attributes, for each i ∈ [1 , m ] , where e i is the i th standard basis vector and α > 0 . Quantifying the extent to which this property holds is generally challenging.

Classical dimensionality reduction techniques, such as PCA and metric MDS, can be used to obtain a latent representation satisfying the first property. For instance, the former ensures that changes to the output resulting from changes to distinct components of the latent variable are orthogonal, which is a particularly restrictive form of distinctness. These techniques are guaranteed to faithfully represent the underlying structure only when the data occupy a linear subspace. For many applications of interest, such as image modeling, the data manifold could be a highly curved and twisted surface that does not satisfy this assumption. The Swiss Roll dataset is a simple example for which this is the case: two points with small euclidean distance could have large geodesic distance on the data manifold. Since both techniques measure similarity between each pair of points using euclidean distance, distant outputs in X could get mapped to nearby embeddings in Z . Thus, the second property often fails to hold in practice.

Deep generative models have the potential to learn representations that satisfy both of the required properties for disentanglement. Our work focuses on the case in which G is a GAN generator. Let J G ( z 0 ) ∈ R n × m be the Jacobian of G evaluated at the latent variable z 0 ∈ Z . Our main contributions are as follows:

We show that the leading right-singular vectors of J G ( z 0 ) can be used to obtain a local disentangled representation about a neighborhood of z 0 ;
We show that following the path determined by a leading right-singular vector in the latent space of a GAN yields a trajectory along which local changes to the output are interpretable; and
We formulate a regularizer that induces disentangled representations in GANs, by aligning the top k rightsingular vectors v 1 , . . . , v k with the first k coordinate directions, e 1 , . . . , e k .

To date, the two most successful approaches for unsupervised learning of disentangled representations are the β -

VAE (Higgins et al., 2016) and InfoGAN (Chen et al., 2016). The former proposes increasing the weight β for the KLdivergence term in the objective of the VAE (Kingma & Welling (2013), Rezende et al. (2014)), which is normally set to one. This weight controls the tradeoff between minimizing reconstruction error, and minimizing the KL-divergence between the approximate posterior and the prior of the decoder. The authors observe that as the KL-divergence is reduced, more coordinate directions in the latent space of the decoder end up corresponding to disentangled factors. To measure the extent to which this occurs, they describe a disentanglement metric score, which we use in Section 6. Follow-up work (Burgess et al., 2018) analyzes β -VAE from the perspective of information channel capacity, and describes principled way of controlling the tradeoff between reconstruction error and disentanglement. Several variants of the β -VAE have also since been developed (Kim & Mnih (2018), Chen et al. (2018), Esmaeili et al. (2018)).

InfoGAN Chen et al. (2016) augments the GAN objective (Goodfellow et al., 2014) with a term that maximizes the mutual information between the generated samples and small subset of latent variables. This is done by means of an auxiliary classifier that is trained along with the generator and the discriminator. Adversarial training offers the potential for a high degree of realism that is difficult to achieve with models like VAEs, which are based on reconstruction error. However, InfoGAN finds fewer disentangled factors than VAE-based approaches on datasets such as CelebA (Liu et al., 2015) and 3DChairs (Aubry et al., 2014), and offers limited control over each latent factor (e.g., maximum rotation angle for azimuth). The mutual information regularizer also detriments sample quality. Hence, the development of an unsupervised method for learning disentangled representations with GANs, while meeting or exceeding the quality of those found by VAE-based approaches, remains an open problem.

Our work makes an important step in this direction. In contrast to previous approaches, which are based on information theory, we leverage the spectral properties of the Jacobian of the generator. This new perspective not only allows us to induce high-quality disentangled representations in GANs, but also to find curved paths over which disentanglement occurs, for GANs that do not possess disentangled representations.

Identifying Local Disentangled Factors

We begin by describing how the singular value decomposition of J G ( z 0 ) can be used to define a local generative model about z 0 that satisfies the first property of disentangled representations. In particular, we show that the left-singular vectors of J G ( z 0 ) form an orthonormal set of directions from G ( z 0 ) along which the magnitudes of the instantaneous changes in G are maximized. Since perturbations

from z 0 along the right-singular vectors result in changes from G ( z 0 ) along the left-singular vectors, the right-singular vectors result in distinct changes to the output of G . This simple relationship allows us to define a local generative model that satisfies the independence property.

First, we show that the perturbations from z 0 along the right-singular vectors maximize the magnitude of the instantaneous change in G ( z 0 ) . Given an arbitrary vector v ∈ R m , the directional derivative

measures the instantaneous change in G resulting from a perturbation along v from z 0 . The magnitude of this change is given by

which is a seminorm involving the positive semidefinite matrix M z ( z 0 ) := J G ( z 0 ) t J G ( z 0 ) ∈ R m × m . The unit-norm perturbation from z 0 that maximizes n z ( v, z 0 ) is given by

where S m -1 ⊂ R m is the unit sphere. This is the first eigenvector of M z ( z 0 ) , which coincides with the first rightsingular vector of J G ( z 0 ) . It follows from the singular value decomposition of J G ( z 0 ) that the first left-singular vector is given by u 1 = σ -1 1 J G ( z 0 ) v 1 , where σ 1 is the first singular value. Hence, a perturbation from z 0 along v 1 maximizes the magnitude of the instantaneous change in G ( z 0 ) , and this change occurs along u 1 .

This is the second eigenvector of M z ( v, z 0 ) , which coincides with the second right-singular vector of J G ( z 0 ) . As before, we get u 2 = σ -1 2 J G ( z 0 ) v 2 , where σ 2 is the second singular value. So a perturbation from z 0 along v 2 results in an instantaneous change in G ( z 0 ) along u 2 . Continuing in this way, we consider the k th unit-norm perturbation orthogonal to v 1 , . . . , v k -1 that maximizes n z ( v, z 0 ) , for each k ∈ [2 , r ] , where r := rank( M z ( z 0 )) . This shows that the right-singular vectors of J G ( z 0 ) maximize the magnitude of the instantaneous change in magnitude of G ( z 0 ) , and these changes occur along the corresponding left-singular vectors.

Now, we use the right-singular vectors of J G ( z 0 ) to define a local generative model about z 0 . Consider the func-

The components of α control perturbations along orthonormal directions, and these directions also result in orthonormal changes to G ( z 0 ) . Hence, ¯ G z 0 satisfies the first property for a generative model to possess a disentangled representation, but only about a neighborhood of z 0 . Figure 13 in Appendix F investigates whether ¯ G z 0 also satisfies the second property: interpretability of changes to individual components of α . We can see that perturbations along the leading eigenvectors M z ( z 0 ) , especially the principal eigenvector, often result in the most drastic changes. These changes are interpretable, and tend to make modifications to isolated attributes of the face. To see this in more detail, we consider the top two rows of subfigure (g). Movement along the first two eigenvectors changes hair length and facial orientation; movement along the third eigenvector decreases the length of the bangs; movement along the fourth and fifth eigenvectors changes background color; and movement along the sixth and seventh eigenvectors changes hair color.

Finding Quasi-Disentangled Paths

Generative models known to possess disentangled representations, such as β -VAEs (Higgins et al., 2016), allow for

continuous manipulation of attributes via perturbations to individual coordinates. Starting from a latent variable z 0 ∈ Z , we can move along the path γ : t ↦→ z 0 + te i in order to vary a single attribute of G ( z 0 ) , while keeping the others held fixed. GANs are not known to learn disentangled representations automatically. Nonetheless, the previous section shows that the local generative model ¯ G z 0 does possess a disentangled representation, but only about a neighborhood of the base point z 0 . We explore whether it is possible to extend this local model to obtain disentanglement along a continuous path from z 0 . To do this, we construct a trajectory γ k : R → R n , t ↦→ G ( γ k ( t )) by repeatedly following the k th leading eigenvector of M z ( γ k ( t )) , where γ (0) := z 0 . The procedure used to do this is given by Algorithm 1.

We first test the procedure on a toy example for which it is possible to explicitly plot the trajectories. We use the dSprites dataset (Matthey et al., 2017), which consists of 64 × 64 white shapes on black backgrounds. Each shape is completely determined by six attributes: symbol (square, ellipse, or heart), scale (6 values), rotation angle from 0 to 2 π (40 values), and x - and y -positions (30 values each). We trained a GAN on this dataset using a latent variable size of three, fewer than the six latent factors that determine each shape. Figure 2 shows that outputs of the generator along these trajectories vary locally along only one or two attributes at a time. Along the first trajectory γ 1 , the generator first decreases the scale of the square, then morphs it into a heart, increases the scale again, and finally begins to rotate it. Similar comments apply to the other two trajectories, γ 2 and γ 3 .

Next, we test the procedure on a GAN trained on the CelebA dataset (Liu et al., 2015) at 64 × 64 resolution. Figure 3(a) shows the trajectories γ 1 starting at four fixed embeddings z 1 , . . . , z 4 . Although the association between the ordinal k of the eigenvector v k and the attribute of the image being varied is not consistent throughout latent space, local changes still tend to occur along only one or two attributes at a time. Figure 3(b) shows the trajectories γ 1 , γ 2 , γ 3 and γ 5 , all starting from the same fixed embedding z 5 . As is the case for the dSprites dataset, we can see that trajectories γ k for distinct k tend to effect changes to G ( z 0 ) along distinct attributes. These results suggest that, along trajectories from z 0 determined by a leading eigenvector of M z ( z 0 ) , changes in the output tend to occur along isolated attributes.

Aligning the Local Disentangled Factors

The β -VAE (Higgins et al., 2016) is known to learn disentangled representations, so that traveling along paths of the form

for certain coordinate directions e j , produces changes to isolated attributes of G ( z 0 ) . In the previous section, we saw

Algorithm 1 Procedure to trace path determined by k th leading eigenvector. Require: mv : R m × R m → R d , z, v ↦→ M z ( z ) v is a function that computes matrixvector products with the implicitly-defined matrix M z ( z ) ∈ R m × m . Require: z ∈ R m is the embedding from which to begin the trajectory. Require: k ∈ [1 , m ] is ordinal of the eigenvector to trace, with k = 1 corresponding to the leading eigenvector. Require: α > 0 is the step size. Require: ρ ∈ [0 , 1) is the decay factor. Require: N glyph[greaterorequalslant] 1 is the required number of steps in the trajectory. 1 2 procedure TRACEEIGENPATH( mv , z, k, α, ρ, N ) 3 z 0 ← z 4 for i ∈ [1 , N ] do 5 M z ( z ) ← EVALUATENORMALJACOBIAN (mv , z ) glyph[triangleright] Details in Appendix A 6 V, D, V t ← SVD ( M z ( z )) 7 w i ← v k glyph[triangleright] Take the k th eigenvector 8 if i glyph[greaterorequalslant] 2 then 9 if 〈 w i -1 , w i 〉 < 0 then 10 w i ←-w i glyph[triangleright] Prevent backtracking by ensuring that ∠ ( w i -1 , w i ) glyph[lessorequalslant] π/ 2 11 w i ← ρw i -1 +(1 -ρ ) w i glyph[triangleright] Apply decay to smoothen the trajectory 12 z i ← z i -1 -αw i 13 return { z 0 , . . . , z N }

that such paths still exist for GANs, but they are not simply straight lines oriented along the coordinate axes (see Figure 2). We develop an efficient regularizer that encourages these paths to take the form of Equation 5, based on alignment of the top k eigenvectors of M z ( z 0 ) with the first k coordinate directions e 1 , . . . , e k . Before proceeding, we describe a useful visualization technique to help measure the extent to which this happens. Let M z ( z 0 ) = V DV t be the eigendecomposition of M z ( z 0 ) , where V is an orthogonal matrix whose columns are eigenvectors, and D the diagonal matrix of nonnegative eigenvalues, sorted in descending order. Now we define ˜ V : z ↦→ V to be the function that maps z to the corresponding eigenvector matrix V , and let

where p z is the prior over Z , and ' ◦ ' denotes Hadamard product. If the k th column is close to a one-hot vector, with values close to zero everywhere except at entry j , then we know that on average, the k th leading eigenvector of M z ( z ) is aligned with e j . A heatmap generated from this matrix therefore allows us to gauge the extent to which each eigenvector v k is aligned with the coordinate direction e k . Figure 5 shows that this does not happen automatically for a GAN, even when it is trained with a small latent variable size. Interestingly, eigenvector alignment does occur for β -VAEs. Appendix B explores this connection in more detail.

We begin by considering the case where we only seek to align the leading eigenvector v 1 ∈ R m with the first coordinate direction e 1 ∈ R m . A simple way to do this is to obtain an estimate ˆ v 1 for v 1 using T power iterations, and then renormalize ˆ v 1 ∈ R m to a unit vector. We can then maximize the value of the first component of the elementwise squared vector ˆ v 1 ◦ ˆ v 1 , and minimize values of the remaining components. Using the mask s 1 := ( -1 , 1 , . . . , 1) ∈ R m , we define the regular-

(a) Trajectories corresponding to the principal eigenvector for embeddings z 1 , . . . , z 4 , from top to bottom, respectively. For embeddings z 1 , z 2 , and z 3 , we show iterates 0 , 20 , . . . , 180 , and for embedding z 4 , we show iterates 0 , 10 , . . . , 90 . The first trajectory varies azimuth; the second, gender; the third, hair length; and the fourth, hair color.

Figure 3: Trajectories found by following leading eigenvectors from five fixed embeddings z 1 , . . . , z 5 , using Algorithm 1 ( α = 1 . 5 · 10 -2 , ρ = 0 ). Details regarding the model architecture are given in Appendix C.

izer R 1 : R m → R , (7)

Since ˆ v 1 is constrained to unit norm, this regularizer is bounded. It can be incorporated into the loss for the generator using an appropriate penalty weight λ > 0 .

Next, we consider the case where we would like to align the first two leading eigenvectors v 1 , v 2 ∈ R m with the first two coordinate directions e 1 , e 2 ∈ R m . One potential approach is to first compute an estimate ˆ v 1 to v 1 using T power iterations, as before. Then, we could apply a modified version of the power method to obtain an estimate ˆ v 2 for v 2 , in which we project the result of each power iteration onto span(ˆ v 1 ) ⊥ using the projection P 1 := I -ˆ v 1 ˆ v t 1 . There are two problems with this approach. Firstly, it could be inaccurate: unless ‖ v 1 -ˆ v 1 ‖ < τ for sufficiently small τ > 0 , which may require a large number of power iterations, P 1 will not be an accurate projection onto span( v 1 ) ⊥ . Error in approximating v 1 would then jeopardize the approximation to v 2 . Second, the approach is inefficient. We can only run the power method to estimate v 2 after the we have already obtained an estimate for v 1 . If we use T power iterations to estimate each eigenvector, then estimating the first k eigenvectors will require a total of kT power iterations. This is too slow to be practical.

Algorithm 2 Procedure to estimate the top k eigenpairs of M z ( z ) . Require: mv : R m → R m is a function that computes matrix-vector products with the implicitly-defined matrix M z ( z ) ∈ R m × m . Require: V ∈ R m × k is a matrix whose columns are the initial estimates for the eigenvectors. Require: T glyph[greaterorequalslant] 1 is the required number of power iterations.

1 2 glyph[epsilon1] ← 10 -8 glyph[triangleright] Guards against division by numbers close to zero 3 4 procedure ESTIMATELEADINGEIGENPAIRS( mv , V, T ) 5 Let M k ∈ R m × k be given by Equation 11 6 V 0 ← M k ◦ V glyph[triangleright] ' ◦ ' denotes Hadamard product 7 8 for i ∈ [1 , T ] do 9 V i ← M ◦ mv( V i -1 ) 10 Λ i ← diag( COLUMNNORMS ( V i )) 11 V i ← V i (Λ i + glyph[epsilon1]I ) -1 glyph[triangleright] Renormalize columns 12 return Λ T , V T 13 14 procedure COLUMNNORMS( A ) glyph[triangleright]A ∈ R p × q 15 return ( ‖ a 1 ‖ , . . . , ‖ a q ‖ ) glyph[triangleright] a i ∈ R p is the i th column of A

Our specific application of the power method enables an optimization that allows for the k power iterations to be run simultaneously. Once we apply the regularizer to the GANtraining procedure, the first eigenvector v 1 will quickly align with the first coordinate direction e 1 . We therefore assume that v 1 = e 1 . This assumption would imply that span( v 1 ) ⊥ = span( e 2 , . . . , e m ) , so applying P 1 would amount to zeroing out the first component of ˆ v 2 after each power iteration. Since P 1 would no longer depend on ˆ v 1 , we can run the power iterations for v 1 and v 2 in parallel. To formally describe this, we let c p := 1 ∈ R p be the constant vector of ones, and let

Given a matrix ˆ V (2) t ∈ R m × 2 whose columns are the current estimates for v 1 and v 2 , respectively, we can describe the power iterations for v 1 and v 2 using the recurrence

Now, let ˆ V (2) ∈ R m × 2 be the final estimate for v 1 and v 2 . To implement the regularizer, we let s 2 := (1 , -1 , 1 , . . . , 1) ∈ R m , S 2 := ( s 1 s 2 ) ∈ R m × 2 , and define

It is straightforward to generalize this approach to the case where we seek to align the first k eigenvectors v 1 , . . . , v k with e 1 , . . . , e k . For each eigenvector v i , with i ∈ [2 , k ] , we assume that eigenvectors v 1 , . . . v i -1 are already aligned with e 1 , . . . , e i -1 . The projections onto span( e 1 ) ⊥ , span( e 1 , e 2 ) ⊥ , . . . , span( e 1 , . . . , e i -1 ) ⊥ can be implemented using columns 2 , 3 , . . . , i , respectively,

Algorithm 3 Procedure to evaluate the alignment penalty.

Require: k ∈ [1 , m ] is number of leading eigenvectors to align with e 1 , . . . , e k . Require: mv , T are as defined in Algorithm 2. 1 2 procedure EVALUATEALIGNMENTREGULARIZER( k, mv , T ) 3 Let S k be given by Equation 12 4 5 α ← 2 / ( k ( k +1)) 6 A ← diag( α · ( k, k -1 , . . . , 1)) 7 S k ← S k A glyph[triangleright] Reweight columns to prioritize alignment of leading eigenvectors 8 9 V 0 ← RANDOMRADEMACHER ( m,k ) 10 ˆ Λ , ˆ V ← ESTIMATELEADINGEIGENPAIRS (mv , V 0 , t ) 11 return SUM ( S k ◦ ˆ V ◦ ˆ V ) 12 13 procedure RANDOMRADEMACHER( p, q ) 14 return A ∈ R p × q , where a ij = 1 with probability 1 / 2 and -1 with probability 1 / 2

of the mask M k ∈ R m × k . This mask, which is a generalization of the one defined by Equation 8, is given by R m × k glyph[owner] M k :=

The resulting procedure to estimate the leading k eigenvectors is described by Algorithm 2. Figure 4 shows that the runtime of Algorithm 2 scales linearly with respect to the number of eigenvectors k and the number of power iterations T . Next, we generalize Equation 10, in order to describe how to evaluate the regularizer R k . Let s p ∈ R m be given by ( s p ) i = -1 if i = p and 1 otherwise, and define

Algorithm 3 shows how S k is used with the result of Algorithm 2 to evaluate R k . Finally, Figure 2(c) shows that incorporating the alignment regularizer into the generator loss successfully aligns the trajectories produced by Algorithm 1 with the coordinate axes.

If the alignment regularizer is implemented exactly as described by Equation 10, it will fail to have the intended effect. The reason for this has to do with the assumption behind the optimization used to run the k power iterations in parallel. Before attempting to align v i with e i , we assume that that v 1 , . . . , v i -1 are already aligned with e 1 , . . . , e i -1 . When this assumption fails to hold, the projections computed using the columns of M k will no longer be valid. Figure 5(a) shows that the matrix F does not have a diagonal in its top-left corner, which is what we would expect to see if v 1 , . . . , v k were aligned with e 1 , . . . , e k . Fortunately, there is a simple fix that remedies the situation. We would like to encourage the optimizer to prioritize alignment of v i with e i over alignment of v i +1 , . . . , v k with e i +1 , . . . , e k ,

for all i ∈ [1 , k -1] . A simple way to do this is to multiply the i th column of M k by a weight of ( k -i +1) α . We choose α based on the condition that these weights sum to one, i.e.,

This reweighting scheme is implemented in lines 5-7 of Algorithm 3. Figure 5(b) confirms that this modification induces the desired structure in the top-left corner of F .

Rapid progress has been made in the development of generative models capable of producing realistic samples from distributions over complex, high-resolution natural images (ProgressiveGAN, BigGAN). Despite the success of these models, it is unclear how one can achieve control over the data-generation process when labeled instances are not available. Perturbing an individual component of a latent variable typically results in an unpredictable change to the output. When the model has a disentangled representation, these changes become interpretable, and each component of the latent variable affects a distinct attribute. So far, the problem of learning disentangled reprersentations remains poorly understood. Most approaches for doing this have focused on VAEs (AEVB, StochasticBackprop), which produce blurry samples in practice. The few approaches that have been developed for GANs, such as InfoGAN (InfoGAN), have had comparably limited success.

Following (betaVAE), we say a generative model G:Z→XfragmentsG:Z→XG:Z\to X, where Z⊂ℝmfragmentsZR𝑚{Z\subset\mathbb{R}^{m}} and X⊂ℝnfragmentsXR𝑛{X\subset\mathbb{R}^{n}}, possesses a disentangled representation when it satisfies two important properties. The first property is independence of the components of the latent variable. Roughly speaking, this means that a perturbation to any component of a latent variable should result in a change to the output that is distinct, in some sense, from the changes resulting from perturbations to the other components. For example, if the first component of the latent variable controls hair length, then the other components should not influence this attribute. The second property is interpretability of the changes in X𝑋X resulting from these perturbations. Suppose that the process of sampling from the distribution pdatafragmentspdatap_{\text{data}} modeled by G𝐺G can be realized by a simulator that is parameterized over a list of independent, scalar-valued factors or attributes. These attributes might correspond to concepts such as lighting, azimuth, gender, age, and so on. This property is met when G(z+αei)fragmentsG(zαe𝑖)G(z+\alpha e_{i}) results in a change along exactly one of these attributes, for each i∈[1,m]fragmentsi[1,m]i\in[1,m], where eifragmentse𝑖e_{i} is the i𝑖ith standard basis vector and α>0fragmentsα0\alpha>0. Quantifying the extent to which this property holds is generally challenging.

Classical dimensionality reduction techniques, such as PCA and metric MDS, can be used to obtain a latent representation satisfying the first property. For instance, the former ensures that changes to the output resulting from changes to distinct components of the latent variable are orthogonal, which is a particularly restrictive form of distinctness. These techniques are guaranteed to faithfully represent the underlying structure only when the data occupy a linear subspace. For many applications of interest, such as image modeling, the data manifold could be a highly curved and twisted surface that does not satisfy this assumption. The Swiss Roll dataset is a simple example for which this is the case: two points with small euclidean distance could have large geodesic distance on the data manifold. Since both techniques measure similarity between each pair of points using euclidean distance, distant outputs in X𝑋X could get mapped to nearby embeddings in Z𝑍Z. Thus, the second property often fails to hold in practice.

Deep generative models have the potential to learn representations that satisfy both of the required properties for disentanglement. Our work focuses on the case in which G𝐺G is a GAN generator. Let JG(z0)∈ℝn×mfragmentsJ𝐺(z0)Rfragmentsnm{J_{G}(z_{0})\in\mathbb{R}^{n\times m}} be the Jacobian of G𝐺G evaluated at the latent variable z0∈Zfragmentsz0Z{z_{0}\in Z}. Our main contributions are as follows:

We show that the leading right-singular vectors of JG(z0)fragmentsJ𝐺(z0)J_{G}(z_{0}) can be used to obtain a local disentangled representation about a neighborhood of z0fragmentsz0z_{0};

We formulate a regularizer that induces disentangled representations in GANs, by aligning the top k𝑘k right-singular vectors v1,…,vkfragmentsv1,…,v𝑘{v_{1},\ldots,v_{k}} with the first k𝑘k coordinate directions, e1,…,ekfragmentse1,…,e𝑘{e_{1},\ldots,e_{k}}.

To date, the two most successful approaches for unsupervised learning of disentangled representations are the β𝛽\beta-VAE (betaVAE) and InfoGAN (InfoGAN). The former proposes increasing the weight β𝛽\beta for the KL-divergence term in the objective of the VAE (AEVB, StochasticBackprop), which is normally set to one. This weight controls the tradeoff between minimizing reconstruction error, and minimizing the KL-divergence between the approximate posterior and the prior of the decoder. The authors observe that as the KL-divergence is reduced, more coordinate directions in the latent space of the decoder end up corresponding to disentangled factors. To measure the extent to which this occurs, they describe a disentanglement metric score, which we use in Section LABEL:sec:results. Follow-up work (UnderstandingDisentangling) analyzes β𝛽\beta-VAE from the perspective of information channel capacity, and describes principled way of controlling the tradeoff between reconstruction error and disentanglement. Several variants of the β𝛽\beta-VAE have also since been developed (FactorVAE, TCVAE, HFVAE).

InfoGAN InfoGAN augments the GAN objective (GAN) with a term that maximizes the mutual information between the generated samples and small subset of latent variables. This is done by means of an auxiliary classifier that is trained along with the generator and the discriminator. Adversarial training offers the potential for a high degree of realism that is difficult to achieve with models like VAEs, which are based on reconstruction error. However, InfoGAN finds fewer disentangled factors than VAE-based approaches on datasets such as CelebA (CelebA) and 3DChairs (3DChairs), and offers limited control over each latent factor (e.g., maximum rotation angle for azimuth). The mutual information regularizer also detriments sample quality. Hence, the development of an unsupervised method for learning disentangled representations with GANs, while meeting or exceeding the quality of those found by VAE-based approaches, remains an open problem.

We begin by describing how the singular value decomposition of JG(z0)fragmentsJ𝐺(z0)J_{G}(z_{0}) can be used to define a local generative model about z0fragmentsz0z_{0} that satisfies the first property of disentangled representations. In particular, we show that the left-singular vectors of JG(z0)fragmentsJ𝐺(z0)J_{G}(z_{0}) form an orthonormal set of directions from G(z0)fragmentsG(z0)G(z_{0}) along which the magnitudes of the instantaneous changes in G𝐺G are maximized. Since perturbations from z0fragmentsz0z_{0} along the right-singular vectors result in changes from G(z0)fragmentsG(z0)G(z_{0}) along the left-singular vectors, the right-singular vectors result in distinct changes to the output of G𝐺G. This simple relationship allows us to define a local generative model that satisfies the independence property.

First, we show that the perturbations from z0fragmentsz0z_{0} along the right-singular vectors maximize the magnitude of the instantaneous change in G(z0)fragmentsG(z0)G(z_{0}). Given an arbitrary vector v∈ℝmfragmentsvR𝑚{v\in\mathbb{R}^{m}}, the directional derivative

measures the instantaneous change in G𝐺G resulting from a perturbation along v𝑣v from z0fragmentsz0z_{0}. The magnitude of this change is given by

which is a seminorm involving the positive semidefinite matrix Mz(z0)≔JG(z0)tJG(z0)∈ℝm×mfragmentsM𝑧(z0)≔J𝐺(z0)𝑡J𝐺(z0)Rfragmentsmm{M_{z}(z_{0})\coloneqq J_{G}(z_{0})^{t}J_{G}(z_{0})\in\mathbb{R}^{m\times m}}. The unit-norm perturbation from z0fragmentsz0z_{0} that maximizes nz(v,z0)fragmentsn𝑧(v,z0)n_{z}(v,z_{0}) is given by

where 𝕊m−1⊂ℝmfragmentsSfragmentsm1R𝑚\mathbb{S}^{m-1}\subset\mathbb{R}^{m} is the unit sphere. This is the first eigenvector of Mz(z0)fragmentsM𝑧(z0)M_{z}(z_{0}), which coincides with the first right-singular vector of JG(z0)fragmentsJ𝐺(z0)J_{G}(z_{0}). It follows from the singular value decomposition of JG(z0)fragmentsJ𝐺(z0)J_{G}(z_{0}) that the first left-singular vector is given by u1=σ1−1JG(z0)v1fragmentsu1σ1fragments1J𝐺(z0)v1{u_{1}=\sigma_{1}^{-1}J_{G}(z_{0})v_{1}}, where σ1fragmentsσ1\sigma_{1} is the first singular value. Hence, a perturbation from z0fragmentsz0z_{0} along v1fragmentsv1v_{1} maximizes the magnitude of the instantaneous change in G(z0)fragmentsG(z0)G(z_{0}), and this change occurs along u1fragmentsu1u_{1}.

This is the second eigenvector of Mz(v,z0)fragmentsM𝑧(v,z0)M_{z}(v,z_{0}), which coincides with the second right-singular vector of JG(z0)fragmentsJ𝐺(z0)J_{G}(z_{0}). As before, we get u2=σ2−1JG(z0)v2fragmentsu2σ2fragments1J𝐺(z0)v2{u_{2}=\sigma_{2}^{-1}J_{G}(z_{0})v_{2}}, where σ2fragmentsσ2\sigma_{2} is the second singular value. So a perturbation from z0fragmentsz0z_{0} along v2fragmentsv2v_{2} results in an instantaneous change in G(z0)fragmentsG(z0)G(z_{0}) along u2fragmentsu2u_{2}. Continuing in this way, we consider the k𝑘kth unit-norm perturbation orthogonal to v1,…,vk−1fragmentsv1,…,vfragmentsk1{v_{1},\ldots,v_{k-1}} that maximizes nz(v,z0)fragmentsn𝑧(v,z0)n_{z}(v,z_{0}), for each k∈[2,r]fragmentsk[2,r]k\in[2,r], where r≔rank(Mz(z0))fragmentsr≔rank(M𝑧(z0))r\coloneqq\operatorname{rank}(M_{z}(z_{0})). This shows that the right-singular vectors of JG(z0)fragmentsJ𝐺(z0)J_{G}(z_{0}) maximize the magnitude of the instantaneous change in magnitude of G(z0)fragmentsG(z0)G(z_{0}), and these changes occur along the corresponding left-singular vectors.

Now, we use the right-singular vectors of JG(z0)fragmentsJ𝐺(z0)J_{G}(z_{0}) to define a local generative model about z0fragmentsz0z_{0}. Consider the function G¯z0:ℝr→ℝnfragments¯𝐺fragmentsz0:R𝑟→R𝑛{\bar{G}{z{0}}:\mathbb{R}^{r}\to\mathbb{R}^{n}},

The components of α𝛼\alpha control perturbations along orthonormal directions, and these directions also result in orthonormal changes to G(z0)fragmentsG(z0)G(z_{0}). Hence, G¯z0fragments¯𝐺fragmentsz0\bar{G}{z{0}} satisfies the first property for a generative model to possess a disentangled representation, but only about a neighborhood of z0fragmentsz0z_{0}. Figure LABEL:fig:latent_perturbations in Appendix LABEL:sec:supplementary_figures investigates whether G¯z0fragments¯𝐺fragmentsz0\bar{G}{z{0}} also satisfies the second property: interpretability of changes to individual components of α𝛼\alpha. We can see that perturbations along the leading eigenvectors Mz(z0)fragmentsM𝑧(z0)M_{z}(z_{0}), especially the principal eigenvector, often result in the most drastic changes. These changes are interpretable, and tend to make modifications to isolated attributes of the face. To see this in more detail, we consider the top two rows of subfigure (g). Movement along the first two eigenvectors changes hair length and facial orientation; movement along the third eigenvector decreases the length of the bangs; movement along the fourth and fifth eigenvectors changes background color; and movement along the sixth and seventh eigenvectors changes hair color.

Generative models known to possess disentangled representations, such as β𝛽\beta-VAEs (betaVAE), allow for continuous manipulation of attributes via perturbations to individual coordinates. Starting from a latent variable z0∈Zfragmentsz0Zz_{0}\in Z, we can move along the path γ:t↦z0+teifragmentsγ:tmaps-toz0te𝑖{\gamma:t\mapsto z_{0}+te_{i}} in order to vary a single attribute of G(z0)fragmentsG(z0)G(z_{0}), while keeping the others held fixed. GANs are not known to learn disentangled representations automatically. Nonetheless, the previous section shows that the local generative model G¯z0fragments¯𝐺fragmentsz0\bar{G}{z{0}} does possess a disentangled representation, but only about a neighborhood of the base point z0fragmentsz0z_{0}. We explore whether it is possible to extend this local model to obtain disentanglement along a continuous path from z0fragmentsz0z_{0}. To do this, we construct a trajectory γk:ℝ→ℝnfragmentsγ𝑘:R→R𝑛{\gamma_{k}:\mathbb{R}\to\mathbb{R}^{n}}, t↦G(γk(t))fragmentstmaps-toG(γ𝑘(t)){t\mapsto G(\gamma_{k}(t))} by repeatedly following the k𝑘kth leading eigenvector of Mz(γk(t))fragmentsM𝑧(γ𝑘(t))M_{z}(\gamma_{k}(t)), where γ(0)≔z0fragmentsγ(0)≔z0\gamma(0)\coloneqq z_{0}. The procedure used to do this is given by Algorithm 1.

We first test the procedure on a toy example for which it is possible to explicitly plot the trajectories. We use the dSprites dataset (dSprites), which consists of 64×64fragments646464\times 64 white shapes on black backgrounds. Each shape is completely determined by six attributes: symbol (square, ellipse, or heart), scale (6 values), rotation angle from 00 to 2πfragments2π2\pi (40 values), and x𝑥x- and y𝑦y-positions (30 values each). We trained a GAN on this dataset using a latent variable size of three, fewer than the six latent factors that determine each shape. Figure 2 shows that outputs of the generator along these trajectories vary locally along only one or two attributes at a time. Along the first trajectory γ1fragmentsγ1\gamma_{1}, the generator first decreases the scale of the square, then morphs it into a heart, increases the scale again, and finally begins to rotate it. Similar comments apply to the other two trajectories, γ2fragmentsγ2\gamma_{2} and γ3fragmentsγ3\gamma_{3}.

Next, we test the procedure on a GAN trained on the CelebA dataset (CelebA) at 64×64fragments646464\times 64 resolution. Figure 3(a) shows the trajectories γ1fragmentsγ1\gamma_{1} starting at four fixed embeddings z1,…,z4fragmentsz1,…,z4{z_{1},\ldots,z_{4}}. Although the association between the ordinal k𝑘k of the eigenvector vkfragmentsv𝑘v_{k} and the attribute of the image being varied is not consistent throughout latent space, local changes still tend to occur along only one or two attributes at a time. Figure 3(b) shows the trajectories γ1,γ2,γ3fragmentsγ1,γ2,γ3{\gamma_{1},\gamma_{2},\gamma_{3}} and γ5fragmentsγ5\gamma_{5}, all starting from the same fixed embedding z5fragmentsz5z_{5}. As is the case for the dSprites dataset, we can see that trajectories γkfragmentsγ𝑘\gamma_{k} for distinct k𝑘k tend to effect changes to G(z0)fragmentsG(z0)G(z_{0}) along distinct attributes. These results suggest that, along trajectories from z0fragmentsz0z_{0} determined by a leading eigenvector of Mz(z0)fragmentsM𝑧(z0)M_{z}(z_{0}), changes in the output tend to occur along isolated attributes.

for certain coordinate directions ejfragmentse𝑗e_{j}, produces changes to isolated attributes of G(z0)fragmentsG(z0)G(z_{0}). In the previous section, we saw that such paths still exist for GANs, but they are not simply straight lines oriented along the coordinate axes (see Figure 2). We develop an efficient regularizer that encourages these paths to take the form of Equation 5, based on alignment of the top k𝑘k eigenvectors of Mz(z0)fragmentsM𝑧(z0)M_{z}(z_{0}) with the first k𝑘k coordinate directions e1,…,ekfragmentse1,…,e𝑘{e_{1},\ldots,e_{k}}. Before proceeding, we describe a useful visualization technique to help measure the extent to which this happens. Let Mz(z0)=VDVtfragmentsM𝑧(z0)VDV𝑡{M_{z}(z_{0})=VDV^{t}} be the eigendecomposition of Mz(z0)fragmentsM𝑧(z0)M_{z}(z_{0}), where V𝑉V is an orthogonal matrix whose columns are eigenvectors, and D𝐷D the diagonal matrix of nonnegative eigenvalues, sorted in descending order. Now we define V~:z↦Vfragments~𝑉:zmaps-toV{\tilde{V}:z\mapsto V} to be the function that maps z𝑧z to the corresponding eigenvector matrix V𝑉V, and let

Now, let V^(2)∈ℝm×2fragments^𝑉fragments(2)Rfragmentsm2\hat{V}^{(2)}\in\mathbb{R}^{m\times 2} be the final estimate for v1fragmentsv1v_{1} and v2fragmentsv2v_{2}. To implement the regularizer, we let s2≔(1,−1,1,…,1)∈ℝmfragmentss2≔(1,1,1,…,1)R𝑚s_{2}\coloneqq(1,-1,1,\ldots,1)\in\mathbb{R}^{m}, S2≔(s1s2)∈ℝm×2fragmentsS2≔matrixfragmentss1fragmentss2Rfragmentsm2S_{2}\coloneqq\begin{pmatrix}s_{1}&s_{2}\end{pmatrix}\in\mathbb{R}^{m\times 2}, and define

Refer to caption Trajectory obtained by following the leading eigenvector of Mz(z0)fragmentsM𝑧(z0)M_{z}(z_{0}) from a fixed embedding z0∈ℝ256fragmentsz0R256z_{0}\in\mathbb{R}^{256} in the latent space of a GAN trained on the LSUN Bedrooms dataset at 256×256fragments256256{256\times 256} resolution. The camera angle is varied smoothly, while the content of the bedroom is preserved. The parameters used for Algorithm 1 are given by α,ρ,N=1.5⋅10−2,0.99,2000fragmentsα,ρ,N1.5⋅10fragments2,0.99,2000{\alpha,\rho,N}=1.5\cdot 10^{-2},0.99,2000; we show iterates 0,20,…,180fragments0,20,…,1800,20,\ldots,180. The architecture and training procedure are as described in (GANConvergence). Videos of the trajectories for the first four eigenvectors can be viewed at this URL: https://drive.google.com/open?id=1_SxlvcjakzkL8vNY4hporwtvg8mLQw4S.

Refer to caption (a)

$$ \lim_{\epsilon\to 0}\frac{G(z_{0}+\epsilon v)-G(z_{0})}{\epsilon}=J_{G}(z_{0})v. $$ \tag{S3.E1}

$$ v_{1}\coloneqq\max_{v\in\mathbb{S}^{m-1}}n_{z}(v,z_{0})=\max_{v\in\mathbb{S}^{m-1}}v^{t}J_{G}(z_{0})^{t}J_{G}(z_{0})v, $$ \tag{S3.E3}

$$ \bar{G}{z{0}}:\alpha\mapsto G\left(z_{0}+\sum_{i\in[1,r]}\alpha_{i}v_{i}\right). $$ \tag{S3.Ex2}

$$ \gamma:t\mapsto z_{0}+te_{j}, $$ \tag{S5.E5}

$$ F\coloneqq\operatorname{E}{z\sim p{z}}\tilde{V}(z)\circ\tilde{V}(z), $$ \tag{S5.E6}

$$ R_{1}(z)\coloneqq\sum_{i\in[1,m]}(s_{1}\circ\hat{v}{1}\circ\hat{v}{1})_{i}. $$ \tag{S5.E7}

$$ M_{2}\coloneqq\begin{pmatrix}c_{m}&\begin{matrix}0\ c_{m-1}\end{matrix}\end{pmatrix}. $$ \tag{S5.E8}

$$ \displaystyle|J_{G}(z_{0})v| $$

Results

We first make a quantitative comparison between our approach and previous methods that have been used to obtain disentangled representations. This requires us to measure the extent to which the second property of disentangled representations - namely, interpretability of changes resulting from perturbations to individual coordinates of a latent variable - holds. Suppose that we had knowledge of the ground truth latent factors for the dataset, along with a simulator

Table 1: Comparison of disentanglement metric scores for our method to those reported in (Higgins et al., 2016). We use an alignment-regularized GAN ( k, λ = 6 , 0 . 6 ) with latent variable size 10 . Details regarding the model architecture and training procedure are given in Appendix C.

that can synthesize new outputs given assignments to these latent factors. Then we could generate pairs of outputs, such that the outputs in each pair differ only along a single attribute. Let ( x 0 , x 1 ) be one such pair, and z 0 := G -1 ( x 0 ) and z 1 := G -1 ( x 1 ) the corresponding latent variables for the generator G : R m → R n . If G satisfies the second property, then we would expect z 0 and z 1 to be approximately equal along all components, except the one corresponding to the attribute that was varied. Hence, | z 0 -z 1 | should be a one-hot vector in expectation. (Higgins et al., 2016) propose an evaluation metric based on this idea. It involves training a linear classifier to predict the index of the latent factor that was varied in order to generate each pair. At each step of training for the classifier, we sample a batch of input-target pairs in accordance with the procedure described in (Higgins et al., 2016), and update the classifier using the cross-entropy loss.

Application of the evaluation metric to GANs is complicated by the fact that a direct procedure to invert the generator is usually not available. Models such as the β -VAE consist of an encoder that effectively functions as an inverse for the generative model, so this is not an issue. For the purpose of evaluation, we also train an encoder to invert the fixed generator, after GAN training has finished. This additional training procedure uses a standard autoencoding loss, and the details are specified in Appendix D. Table 1 compares our approach to the ones evaluated in (Higgins et al., 2016) on the dSprites dataset (Matthey et al., 2017), which we describe in Section 4. As stated in Appendix C, we added noise to both the real and fake inputs of the discriminator in order to stabilize GAN training on this dataset, for which the pixels are binary-valued. Incorporating this modification into the training procedure for InfoGAN may improve the score reported by (Higgins et al., 2016). Our approach achieves a score competitive to that of DC-IGN, which makes use of supervised information. We note that the encoder trained to invert the generator does not always succeed in finding a latent variable that yields an accurate reconstruction of the input. This may lead to a reduction in the disentanglement score, a problem that does not affect the other approaches for which an accurate inference procedure is available.

Next, we make a qualitative comparison between our approach and β -VAE. We train a series of GANs on the CelebA dataset (Liu et al., 2015), with k ∈ { 8 , 16 , 32 } for Algorithm 3 and varying values for the alignment regularizer weight λ . We also train a series of β -VAEs with β ∈ { 1 , 2 , 4 , 8 , 16 } . The results for our method with k, λ = 32 , 0 . 5 are shown in Figure 7, and the results for β -VAE with β = 8 in Figure 8. Figure 6 shows the top-left corners of the matrix F given by Equation 6, for three configurations of our approach with k ∈ { 8 , 16 , 32 } . A penalty weight of λ = 0 . 5 was not sufficient to result in a clear diagonal structure in the top-left corner of the matrix F for the configuration with k = 32 . Nonetheless, this configuration resulted in the largest number of disentangled factors, and the results are shown in Figure 7. The best results for the configurations with k ∈ { 8 , 16 } are shown in Appendix E. We found that the β -VAE configuration with β = 8 resulted in the largest number of disentangled factors, while still maintaining sufficiently low reconstruction error so as to keep the sample quality acceptable. These results are shown in Figure 8. In addition to better sample quality, our approach is able to learn concepts such as different kinds of hair styles (coordinates 9 and 10 in Figure 7) that are not modeled by the β -VAE.

Even with relatively large values for the penalty weight, the heatmaps shown in Figure 6 contain nonzero entries above and below the diagonal. This can result in some degree of leakage of the attribute controlled by one coordinate into the next. To see this in more detail, we examine the disentangled factors found by the configuration with k = 16 , which are shown in Figure 12. Coordinates { 1 , 2 , 3 } all involve change in hair darkness; coordinates { 4 , 5 , 6 } all involve change in gender; coordinates { 6 , 7 , 8 , 9 } all involve smiling; and coordinates { 12 , 13 } all involve change in the location of the hair partition. This is a limitation of our current approach, and modifications to the implementation of the alignment regularizer in Algorithm 3 may help to mitigate this leakage. We plan to investigate such improvements in future work.

Conclusion

Our work approaches the problem of learning disentangled representations from the perspective of eigenvector alignment. We develop a novel regularizer which, when incorporated into the GAN objective, induces disentangled representations of quality comparable to those obtained by VAE-based approaches (Higgins et al. (2016), Kim & Mnih

Figure 6: Comparison of the top-left corners of the matrix F (Equation 6) for alignment-regularized GANs with k ∈ { 8 , 16 , 32 } . All GANs were trained with latent variable size 128 . Details regarding the model architecture and training procedure are given in Appendix C.

(2018), Chen et al. (2018), Esmaeili et al. (2018)), without the need to introduce auxiliary models into the training procedure (Chen et al., 2016). This approach is also not specific to the GAN framework (Goodfellow et al., 2014), and could potentially be applied to autoregressive models, such as those used to generate text and audio. We believe this is an important direction for future investigation. So far, two different perspectives for viewing disentanglement have been proposed: maximizing mutual information and eigenvector alignment. An investigation into the relationship between the two could further our understanding of what disentanglement is and why it occurs.

Acknowledgements

We would like to thank for Ryan-Chris Moreno Vasquez and Emily Denton for early discussions that led us to think carefully about the paths determined by the leading eigenvectors of M z ( z 0 ) . We are also grateful for the suggestions provided by Mikael Henaff, Alec Radford, Yura Burda, and Harri Edwards, which improved the quality of our presentation.

Figure 7: Disentanglement results for alignment-regularized GAN ( k, λ = 32 , 0 . 5 ) with latent variable size 128 on the CelebA dataset, at 64 × 64 resolution. Details regarding the model architecture and training procedure are given in Appendix C. Results with additional samples for each coordinate can be found at this URL: https://drive.google.com/open?id= 1E2TneLSAdyFYgN4GUzYKnHDYABo-G8RK .

References

Forward- and reverse-mode automatic differentiation

The entries of J G are not explicitly stored in memory: it is an implicit-defined matrix. As such, information about J G must be accessed via automatic differentiation (AD), of which there are two kinds: forward-mode and reverse-mode. We briefly describe them here, and refer the reader to (Baydin et al., 2017) for a more comprehensive survey.

Suppose that we are given a differentiable function f : R m → R n corresponding to a feedforward neural network with L layers, and let G := ( V, E ) be its representation as a computation graph. For each k ∈ [1 , L ] , let R k be the set of nodes corresponding to the k th layer, and let r k : R α k -1 → R α k be the function computed by the nodes in R k , with α 0 := m and α L := n . Then f = r L ◦ r L -1 ◦ · · · ◦ r 2 ◦ r 1 . Finally, let ¯ x ∈ R m denote the variable for the input to f , and ¯ b k -1 ∈ R α k -1 the variable for the input to r k , with ¯ b 0 := ¯ x . Given a vector v ∈ R m , forward-mode AD computes Jv . It works according to the recurrence

where k ∈ [1 , L ] and v 0 := v . After step k , we will have obtained the product of the Jacobian of r k with respect to x , and v ; after step L , we will have obtained the desired product Jv .

Forward-mode AD computes Jv far more efficiently than the approach of first evaluating J , and subsequently multiplying by v . Suppose for simplicity that α k = n for all k ∈ [0 , L ] , so that J ∈ R n × n . To compute J independently, we would use the chain rule, which gives

Each layer r k , for k ∈ [1 , L ] , must compute its n × n Jacobian, and each layer after the first must multiply its Jacobian with the n × n Jacobian of the preceding layer with respect to x . This process is described by the recurrence

Assuming that Θ(1) operations are required to compute each element of the Jacobian, the total number of operations required is proportional to

despite the fact that J has only n 2 elements. On the other hand, the k th step of the forward-mode recurrence requires Θ( n ) operations, so computing Jv only requires Θ( Ln ) operations. By running forward-mode with v = e i for each i ∈ [1 , n ] , we can form J column-by-column in only Θ( Ln 2 ) operations. Henceforth, we will measure operation count in terms of invocations to the AD engine, rather than by counting elementary operations.

Of the two kinds of AD, it is reverse-mode that finds the most use in machine learning. Given a vector w ∈ R n , it computes J t w . Most applications involve minimizing a scalar-valued loss function with respect to a vector of parameters, which corresponds to the case where n = 1 . This special case is otherwise known as backpropagation. Unlike forward-mode, which begins at the first layer and ends at the last, reverse-mode begins at the last layer and ends at the first. It works according to the recurrence

where k ranges from L to 1 , and w L := w . After L steps, we will have obtained the desired product J t w . Machine learning frameworks typically expose the full interface to the reverse-mode AD engine, rather than specializing to the case of backpropagation. E.g., in TensorFlow (Abadi et al., 2016), one can change the value of w for tf.gradients from a vector of ones to a custom value specified by the grad_ys parameter.

If desired, we can compute the entire Jacobian using either type of AD. By running forward-mode with v = e i for each i ∈ [1 , m ] , we can form J column-by-column, using m total invocations to AD. Similarly, by running reverse-mode

1 import tensorflow as tf 2 3 def forward_gradients(ys, xs, d_xs): 4 v = tf.placeholder_with_default(tf.ones_like(ys), shape=ys.get_shape()) 5 g = tf.gradients(ys, xs, grad_ys=v) 6 return tf.gradients(g, v, grad_ys=d_xs) 7 8 def j_v(ys, xs, vs): 9 return forward_gradients(ys, xs, vs) 10 11 def jt_v(ys, xs, vs): 12 return tf.gradients(ys, xs, vs) 13 14 def jt_j_v(ys, xs, vs): 15 jv = j_v(ys, xs, vs) 16 return tf.gradients(ys, xs, jv) 17 18 def j_jt_v(ys, xs, vs): 19 jt_v_ = jt_v(ys, xs, vs) 20 return forward_gradients(ys, xs, jt_v_)

Listing 1: TensorFlow implementation of Jacobian-vector operations.

with w = e i for each i ∈ [1 , n ] , we can form J row-by-row using n total invocations to AD. If n > m , the former is typically faster; otherwise, the latter is preferable. We note that while AD offers a relatively efficient approach for evaluating J , doing so at each iteration of optimization becomes impractical.

Many popular machine learning frameworks (e.g., TensorFlow) do not implement forward-mode natively, since it is seldom used for machine learning. Surprisingly, this is not a limitation: reverse-mode can be used to implement forward-mode. Given a differentiable function f : R m → R n , we can compute w t J for a given vector w ∈ R n , using reverse-mode AD. Treating the input to f as a constant, we can regard w t J as a function g : R n → R m , w ↦→ w t J . The derivative of g with respect to w is given by J t , and so another application of reverse-mode AD allows us to compute v t J t = ( Jv ) t . Hence, reverse-mode AD can be used to implement forward-mode AD. This trick was first described by (Townsend, 2017). We provide a TensorFlow implementation of the procedures to compute Jv, J t v, J t Jv , and JJ t v in Listing 1.

Eigenvector Alignment for~$ beta$-VAEs

The results from Section 4 suggest that alignment of the eigenvectors of M z ( z 0 ) with the coordinate axes might be sufficient to induce disentanglement in the latent representation of a generative model. The β -VAE is known to learn such a representation when β is increased, so that the KL-divergence between the approximate posterior and the prior of the decoder is made sufficiently small. Suppose that a β -VAE exhibits disentanglement along the j th coordinate direction in

Similarity with Coordinate Axis

Figure 10: Investigation of whether the matrix F can be used to determine which directions in latent space correspond to disentangled factors for a β -VAE. For each i ∈ [1 , 32] , we compute the maximum value along the i th row of matrix F shown in Figure 9(b). Then, we plot a point indicating whether or not disentanglement occurs along this direction, as determined by visual inspection. The directions that do result in disentanglement are shown in Figure 8.

Table 2: GAN and β -VAE architectures used for all figures and tables. Note 1: weight normalization (Salimans & Kingma, 2016) was applied both to the generator and the discriminator, with the scale g fixed to one for the discriminator. Note 2: spectral weight normalization (Miyato et al., 2018) was applied both to the encoder and the decoder, with learned scales for both. Note 3: gaussian noise with standard deviation 0 . 6 was added to both real and fake inputs to the discriminator.

latent space. Then paths of the form γ : t ↦→ z 0 + te j will produce changes to G ( z 0 ) along isolated factors of variation. If such a path coincides with the trajectory found by Algorithm 1 for the k th leading eigenvector throughout latent space, then this eigenvector must be aligned with the j th coordinate axis. Hence, we would expect the k th column of the matrix F given by Equation 6 to be a one-hot vector close to e j . Figure 9 shows that, in fact, several columns of F have high similarity to coordinate directions.

Next, we investigate whether a column of F having high similarity with a coordinate direction e j implies that disentanglement actually occurs along γ : t ↦→ z 0 + te j . Figure 10 shows that with the exception of three coordinates having similarity greater than 0.35, this turns out not to be the case. In other words, several of the eigenvectors naturally align with coordinate directions during training, but disentanglement does not reliably occur along all of them. More work needs to be done in order to better understand the relationship between eigenvector alignment and disentangled representations.

Generator Architectures and Training Procedure

All models in this work are based on the DCGAN (Radford et al., 2015) architecture. Table 2 describes the architectures of the generator and discriminator (in the case of GANs) and the encoder and decoder (in the case of V AEs) associated with each figure and table. All models use the translated PReLU activation function (Xiang & Li, 2017); the ReLU leaks are learned, and clipped to the interval [0 , 1] after each parameter update. The only data preprocessing we applied was to scale the pixel values to the interval [0 , 1] . The GANs were trained using the original, non-saturating GAN loss described in (Goodfellow et al., 2014) with multivariate normal prior. Both the generator and the discriminator were trained using RMSProp (Tieleman &Hinton, 2012) with step size 10 -4 , decay factor 0 . 9 , and glyph[epsilon1] = 10 -6 . Each parameter update was made using a batch size of 32 , and the models were trained for a total of 750 , 000 updates. The VAEs were trained using a gaussian likelihood model for the decoder, and the log-diagonal covariance parameterization for the encoder. The fixed, per-pixel standard deviation of the decoder was chosen such that, disregarding constant terms, the log-likelihood corresponds to the reconstruction error, normalized by the product of the number of channels and the number of pixels. To optimize the evidence lower bound, we used Adam (Kingma & Ba, 2014) with β 1 , β 2 , glyph[epsilon1] = 0 . 5 , 0 . 99 , 10 -8 . Each parameter update was made using a batch size of 32 , and the models were trained for a total of 1 , 500 , 000 updates. The KL-divergence weight β was increased linearly

Algorithm 4 Procedure to generate a batch for disentanglement metric classifier. Require: n inst glyph[greaterorequalslant] 1 is the number of aligned pairs to use, in order to create each instance in the batch. Require: n batch glyph[greaterorequalslant] 1 is the batch size. Require: Enc : R n → R m is the encoder functioning as the inverse of the generator G . c ← (3 , 6 , 40 , 30 , 30) glyph[triangleright] Ranges for the five attributes that determine each shape procedure SAMPLESHAPE( i, v ) glyph[triangleright] Sample shape with attribute i fixed to value v for j ∈ [1 , 5] do u j ← RANDOMUNIFORM (1 , c j ) u ← ( u 1 , u 2 , u 3 , u 4 , u 5 ) u i ← v return MAKESHAPE ( u ) procedure MAKEINSTANCE( n inst , Enc ) i ← RANDOMUNIFORM (2 , 5) glyph[triangleright] Sample index of attribute to fix v ← RANDOMUNIFORM (1 , c i ) glyph[triangleright] Sample value for fixed attribute z ← 0 ∈ R m for k ∈ [1 , n inst ] do z 0 ← Enc( SAMPLESHAPE ( i, v )) z 1 ← Enc( SAMPLESHAPE ( i, v )) z ← z + | z 0 -z 1 | return z/n inst , i procedure MAKEBATCH( n inst , n batch , Enc ) inputs ← ∅ targets ← ∅ for k ∈ [1 , n batch ] do x, y ← MAKEINSTANCE ( n inst , Enc) inputs ← inputs ∪ { x } targets ← targets ∪ { y } return inputs , targets

from 0 to the final value over the first 1 , 000 , 000 updates. We applied the function x ↦→ 2(tanh( x ) + 1 / 2) to the outputs of both the GAN generators and the VAE decoders; this ensures that the output pixel values are in the interval [0 , 1] . To predict the mean and log-diagonal covariance with the VAE encoders, we apply the translated ReLU activation function (Xiang & Li, 2017) to the final convolutional features, followed by two fully-connected layers, one for each statistic.

Implementation Details for Disentanglement Metric Score

Algorithm 4 describes the procedure used to generate the batches to train and evaluate the classifier for the disentanglement metric (Higgins et al., 2016). We use n inst , n batch = 64 , 32 , and update the classifier using SGD with nesterov accelerated gradient (step size 10 -2 , momentum 0 . 99 ). We train the classifier using a total of 10 , 000 parameter updates, and evaluate its performance using 5000 instances, as reported by (Higgins et al., 2016). To invert the pretrained GAN generator, we train a VAE decoder with twice the base feature map count for the generator, using 30 , 000 parameter updates. The details for this training procedure are identical to those for regular V AE training described in Appendix C, except that the generator is not updated, and the KL-divergence weight β is set to zero. In other words, we use a standard autoencoding loss with a fixed decoder.

Additional GAN Disentanglement Results

Figure 11: Disentanglement results for alignment-regularized GAN ( k, λ = 8 , 2 . 3 ). Details regarding the model architecture and training procedure are given in Appendix C. Results with additional samples for each coordinate can be found at this URL: https://drive. google.com/open?id=1Wnd5jIxopFsBRlylMUN2HfWhICXGiU3u .

Supplementary Figures

jet

(a) Log spectra for CelebA models with n f = 64 and m varied.

Figure 13: Effect of perturbing a latent variable z along the leading eigenvectors of M z ( z ) . Subfigures (a) and (b) show the top eigenvalues of M z ( z i ) , where { z 1 , . . . , z 12 } are fixed embeddings; small eigenvalues are omitted. Subfigures (c)-(d) compare the effects of perturbations along random directions to perturbations along leading eigenvectors. Each subfigure consists of two stacked two-row grids. The leftmost images of each grid are both identical and equal to G ( z i ) , for some i ∈ [1 , 12] . The first row shows the effect of perturbing z i along 13 directions sampled uniformly from the sphere of radius glyph[epsilon1] , while the second row shows the effect of perturbing z i along the first 13 leading eigenvectors of M z ( z i ) , by the same distance glyph[epsilon1] . Details regarding the model architectures are given in Appendix C.

Model	Disentanglement Score
Ground truth	100
Raw pixels	45.75 ± 0.8
PCA	84.90 ± 0.4
ICA	42.03 ± 10.6
DC-IGN	99.3 ± 0.1
InfoGAN 1	73.5 ± 0.9
VAE (untrained)	44.14 ± 2.5
VAE	61.58 ± 0.5
β -VAE	99.23 ± 0.1
Ours	92.34 ± 0.4

Coordinate	Description	Example
1	Background darkness
2	Azimuth
3	Bangs
4	Gender, bangs, and smiling
5	Smiling
6	Hair color
7	Hair color and hair style
8	Lighting color and bangs
9	Hair color and hair style
10	Hair color and hair style
11	Jawline
12	Smiling and bangs
13	Background color
14	Age and hairline
15	Age
17	Location of hair partition
18	Lighting and skin tone
20	Mouth open
21	Mouth open

Coordinate	Description	Example
1	Azimuth
2	Background color
3	Location of hair partition
4	Skin color
5	Hair direction
8	Background color
12	Hair length
14	Jawline
19	Skin tone
20	Hair style
22	Sunglasses
23	Gender
25	Background darkness
26	Hairline and skin tone
28	Lighting direction and rotation
32	Hair size, color, and smiling

Figure	DCGAN Base Feature Map Count	Latent Variable Size	Notes
13(c)	64	32	1
13(e)	64	64	1
13(g)	64	128	1
13(d),5, 4, 6(a), 6(b), 7	64	128	1
3, 11, 12	128	128	1
13(f)	64	256	1
13(h)	64	512	1
2	64	3	1, 3
1	64	10	1, 3
9(a)	64	16	1
9(b), 10	64	32	2

Coordinate	Description	Example
1	Background darkness and hair color
2	Azimuth, lighting, and hair color
3	Hairline and hair color
4	Azimuth
5	Shadow
6	Smiling, age, skin tone, gender
7	Smiling, age, jawline
8	Jawline and hairstyle

Coordinate	Description	Example
1	Background and hair darkness
2	Azimuth and hair darkness
3	Hair length, hair darkness, and lighting
4	Smiling and gender
5	Hair length, hair darkness, and gender
6	Smiling, bangs, and gender
7	Smiling and hairstyle
8	Smiling, jawline, and glaring expression
9	Smiling, bangs, and mouth open
10	Hairline
11	Raised eyebrows and skin tone
12	Raised eyebrows and location of hair partition
13	Raised eyebrows and location of hair partition
14	Lighting color

$$ \lim_{\epsilon \to 0} \frac{G(z_0 + \epsilon v) - G(z_0)}{\epsilon} = \jg(z_0) v. $$

$$ v_2 \coloneqq \max_{v \in \mathbb{S}^{d - 1} ;\cap; \vspan(v_1)^\perp} v^t \jg(z_0)^t \jg(z_0) v. $$

$$ \gamma : t \mapsto z_0 + t e_j, \label{eq:aligned_path} $$ \tag{eq:aligned_path}

$$ F \coloneqq \E_{z \sim p_z} \tilde{V}(z) \circ \tilde{V}(z), \label{eq:heatmap_matrix} $$ \tag{eq:heatmap_matrix}

$$ R_1(z) \coloneqq \sum_{i \in [1, m]} (s_1 \circ \hat{v}_1 \circ \hat{v}_1)_i. $$

$$ M_2 \coloneqq \begin{pmatrix*} c_m & \begin{matrix*} 0 \ c_{m - 1} \end{matrix*} \end{pmatrix*}. \label{eq:eigvec_mask_2} $$ \tag{eq:eigvec_mask_2}

$$ \smallmath{\begin{pmatrix*} c_k & c_k - e_1 & c_k - (e_1 + e_2) & \cdots & c_k - (e_1 + \cdots + e_{k - 1}) \ c_{m - k} & c_{m - k} & c_{m - k} & \cdots & c_{m - k} \end{pmatrix*}}. \label{eq:eigvec_mask} $$ \tag{eq:eigvec_mask}

$$ S_k \coloneqq \smallmath{\begin{pmatrix*}[l] -1 & \phantom{-}1 & \phantom{-}1 & \cdots & \phantom{-}1 \ \phantom{-}1 & -1 & \phantom{-}1 & \cdots & \phantom{-}1 \ \phantom{-}1 & \phantom{-}1 & -1 & \cdots & \phantom{-}1 \ \phantom{-}\vdots & \phantom{-}\vdots & \phantom{-}\vdots & \ddots & \phantom{-}\vdots \ \phantom{-}1 & \phantom{-}1 & \phantom{-}1 & \cdots & -1 \ \phantom{-}\vdots & \phantom{-}\vdots & \phantom{-}\vdots & \ddots & \phantom{-}\vdots \ \phantom{-}1 & \phantom{-}1 & \phantom{-}1 & \cdots & \phantom{-}1 \end{pmatrix*}} = \begin{pmatrix*} s_1 & \cdots & s_k \end{pmatrix*} \in \reals^{m \times k}. \label{eq:reg_mask} $$ \tag{eq:reg_mask}

$$ \sum_{i \in [1, k]} i \alpha = 1, \quad\text{implying that}\quad \alpha = \frac{2}{k (k + 1)}. $$

$$ | \jg(z_0) v | &= (v^t \jg(z_0)^t \jg(z_0) v)^{1 / 2} \notag\ &\eqqcolon (v^t M_z(z_0) v)^{1 / 2} \eqqcolon n_z(v, z_0), $$

$$ v_k \coloneqq \left( D_{\bar{x}} r_k \big|{\bar{x} = x} \right) v &= \left( D{\bar{b}{k - 1}} r_k \big|{\bar{b}{k - 1} = b{k - 1}} \right) \left( D_{\bar{x}}; r_{k - 1} \circ \ldots \circ r_1 \big|{\bar{x} = x} \right) v \ &= \left( D{\bar{b}{k - 1}} r_k \big|{\bar{b}{k - 1} = b{k - 1}} \right) v_{k - 1}, $$

$$ w_k^t \coloneqq w^t \left( D_{\bar{b}{k - 1}} r_k \big|{\bar{b}{k - 1} = b{k - 1}} \right) &= w^t \left( D_{\bar{b}k}; r_L \circ \cdots \circ r{k + 1} \big|{\bar{b}k = b_k} \right) \left( D{\bar{b}{k - 1}} r_k \big|{\bar{b}{k - 1} = b_{k - 1}} \right) \ &= w_{k + 1}^t \left( D_{\bar{b}{k - 1}} r_k \big|{\bar{b}{k - 1} = b{k - 1}} \right), \ $$

$$ N \coloneqq n^2 + (L - 1) (n^2 + n^3) \in \Theta(n^3), $$

Algorithm: algorithm
[t]\tiny
\caption{Procedure to trace path determined by $k$th~leading
eigenvector.\label{alg:trace_eigenpath}}
\newcommand{\mv}{\operatorname{mv}}

\begin{algorithmic}[1]
\Require{${\mv : \reals^m \times \reals^m \to \reals^d}$, ${z, v \mapsto M_z(z) v}$ is a function
that computes matrix-vector products with the implicitly-defined matrix~$M_z(z) \in \reals^{m \times
m}$.}
\Require{$z \in \reals^m$ is the embedding from which to begin the trajectory.}
\Require{$k \in [1, m]$ is ordinal of the eigenvector to trace, with~$k = 1$ corresponding to the
leading eigenvector.}
\Require{$\alpha > 0$ is the step size.}
\Require{$\rho \in [0, 1)$ is the decay factor.}
\Require{$N \geq 1$ is the required number of steps in the trajectory.}
\State

\Procedure{TraceEigenpath}{$\mv, z, k, \alpha, \rho, N$}
\State $z_0 \gets z$
\For{$i \in [1, N]$}
	\State $M_z(z) \gets \Call{EvaluateNormalJacobian}{\mv, z}$ \Comment{Details in Appendix~\ref{sec:autodiff}}
	\State $V, D, V^t \gets \Call{SVD}{M_z(z)}$
	\State $w_i \gets v_k$ \Comment{Take the~$k$th eigenvector}
	\If{$i \geq 2$}
		\If{$\inner{w_{i - 1}}{w_i} < 0$}
			\State $w_i \gets -w_i$ \Comment{Prevent backtracking by ensuring that
			${\angle(w_{i - 1}, w_i) \leq \pi / 2}$}
		\EndIf
		\State $w_i \gets \rho w_{i - 1} + (1 - \rho) w_i$ \Comment{Apply decay to smoothen
		the trajectory}
	\EndIf
	\State $z_i \gets z_{i - 1} - \alpha w_i$
\EndFor
\State \Return $\{z_0, \ldots, z_N\}$
\EndProcedure
\end{algorithmic}

Algorithm: algorithm
[t]\scriptsize
\caption{Procedure to estimate the top~$k$ eigenpairs of~$M_z(z)$.\label{alg:est_dom_eigenpairs}}
\newcommand{\mv}{\operatorname{mv}}

\begin{algorithmic}[1]
\Require{$\mv : \reals^m \to \reals^m$ is a function that computes matrix-vector products with the
implicitly-defined matrix~$M_z(z) \in \reals^{m \times m}$.}
\Require{$V \in \reals^{m \times k}$ is a matrix whose columns are the initial estimates for the
eigenvectors.}
\Require{$T \geq 1$ is the required number of power iterations.}
\State

\State $\epsilon \gets 10^{-8}$ \Comment{Guards against division by numbers close to zero}
\State

\Procedure{EstimateLeadingEigenpairs}{$\mv, V, T$}
\State Let~$M_k \in \reals^{m \times k}$ be given by Equation~\ref{eq:eigvec_mask}
\State $V_0 \gets M_k \circ V$ \Comment{`$\circ$' denotes Hadamard product}
\State

\For{$i \in [1, T]$}
	\State $V_i \gets M \circ \mv(V_{i - 1})$
	\State $\Lambda_i \gets \diag(\Call{ColumnNorms}{V_i})$
	\State $V_i \gets V_i\, (\Lambda_i + \epsilon I)^{-1}$ \Comment{Renormalize columns}
\EndFor

\State \Return $\Lambda_T, V_T$
\EndProcedure
\State

\Procedure{ColumnNorms}{$A$} \Comment{$A \in \reals^{p \times q}$}
	\State \Return $(\|a_1\|, \ldots, \|a_q\|)$ \Comment{$a_i \in \reals^p$ is the $i$th column
	of $A$}
\EndProcedure
\end{algorithmic}

Algorithm: algorithm
[t]\scriptsize
\caption{Procedure to evaluate the alignment penalty.\label{alg:alignment_reg}}
\newcommand{\mv}{\operatorname{mv}}

\begin{algorithmic}[1]
\Require{$k \in [1, m]$ is number of leading eigenvectors to align with~$e_1, \ldots, e_k$.}
\Require{$\mv, T$ are as defined in Algorithm~\ref{alg:est_dom_eigenpairs}.}
\State

\Procedure{EvaluateAlignmentRegularizer}{$k, \mv, T$}
\State Let~$S_k$ be given by Equation~\ref{eq:reg_mask}
\State

\State $\alpha \gets 2 / (k (k + 1))$
\State $A \gets \diag(\alpha \cdot (k, k - 1, \ldots, 1))$
\State $S_k \gets S_k A$ \Comment{Reweight columns to prioritize alignment of leading eigenvectors}
\State

\State $V_0 \gets \Call{RandomRademacher}{m, k}$
\State $\hat{\Lambda}, \hat{V} \gets \Call{EstimateLeadingEigenpairs}{\mv, V_0, t}$
\State \Return $\Call{Sum}{S_k \circ \hat{V} \circ \hat{V}}$
\State
\EndProcedure

\Procedure{RandomRademacher}{$p, q$}
\State \Return $A \in \reals^{p \times q}$, where~$a_{ij} = 1$ with probability~$1 / 2$ and~$-1$
with probability~$1 / 2$
\EndProcedure
\end{algorithmic}

Algorithm: algorithm
[t]\tiny
\caption{Procedure to generate a batch for disentanglement metric
classifier.\label{alg:classifier_batch}}
\begin{algorithmic}
\Require{$n_\text{inst} \geq 1$ is the number of aligned pairs to use, in order to create each
instance in the batch.}
\Require{$n_\text{batch} \geq 1$ is the batch size.}
\Require{$\enc : \reals^n \to \reals^m$ is the encoder functioning as the inverse of the
generator~$G$.}
\State

\State{$c \gets (3, 6, 40, 30, 30)$} \Comment{Ranges for the five attributes that determine each
shape}
\State

\Procedure{SampleShape}{$i, v$} \Comment{Sample shape with attribute~$i$ fixed to value~$v$}
	\For{$j \in [1, 5]$}
		\State{$u_j \gets \Call{RandomUniform}{1, c_j}$}
	\EndFor
	\State{$u \gets (u_1, u_2, u_3, u_4, u_5)$}
	\State{$u_i \gets v$}
	\State \Return $\Call{MakeShape}{u}$
\EndProcedure
\State

\Procedure{MakeInstance}{$n_\text{inst}, \enc$}
	\State{$i \gets \Call{RandomUniform}{2, 5}$} \Comment{Sample index of attribute to fix}
	\State{$v \gets \Call{RandomUniform}{1, c_i}$} \Comment{Sample value for fixed attribute}
	\State{$z \gets 0 \in \reals^m$}
	\For{$k \in [1, n_\text{inst}]$}
		\State{$z_0 \gets \enc(\Call{SampleShape}{i, v})$}
		\State{$z_1 \gets \enc(\Call{SampleShape}{i, v})$}
		\State{$z \gets z + |z_0 - z_1|$}
	\EndFor
	\State \Return $z / n_\text{inst}, i$
\EndProcedure
\State

\Procedure{MakeBatch}{$n_\text{inst}, n_\text{batch}, \enc$}
	\State{$\text{inputs} \gets \varnothing$}
	\State{$\text{targets} \gets \varnothing$}
	\For{$k \in [1, n_\text{batch}]$}
		\State{$x, y \gets \Call{MakeInstance}{n_\text{inst}, \enc}$}
		\State{$\text{inputs} \gets \text{inputs} \cup \{x\}$}
		\State{$\text{targets} \gets \text{targets} \cup \{y\}$}
	\EndFor
	\State \Return $\text{inputs}, \text{targets}$
\EndProcedure

\end{algorithmic}

Algorithm: algorithm
[1]
\Require{$k \in [1, m]$ is number of leading eigenvectors to align with~$e_1, \ldots, e_k$.}
\Require{$\mv, T$ are as defined in Algorithm~\ref{alg:est_dom_eigenpairs}.}
\State

\Procedure{EvaluateAlignmentRegularizer}{$k, \mv, T$}
\State Let~$S_k$ be given by Equation~\ref{eq:reg_mask}
\State

\State $\alpha \gets 2 / (k (k + 1))$
\State $A \gets \diag(\alpha \cdot (k, k - 1, \ldots, 1))$
\State $S_k \gets S_k A$ \Comment{Reweight columns to prioritize alignment of leading eigenvectors}
\State

\State $V_0 \gets \Call{RandomRademacher}{m, k}$
\State $\hat{\Lambda}, \hat{V} \gets \Call{EstimateLeadingEigenpairs}{\mv, V_0, t}$
\State \Return $\Call{Sum}{S_k \circ \hat{V} \circ \hat{V}}$
\State
\EndProcedure

\Procedure{RandomRademacher}{$p, q$}
\State \Return $A \in \reals^{p \times q}$, where~$a_{ij} = 1$ with probability~$1 / 2$ and~$-1$
with probability~$1 / 2$
\EndProcedure

Model	Disentanglement Score
Ground truth	100
Raw pixels	45.75 ± 0.8
PCA	84.90 ± 0.4
ICA	42.03 ± 10.6
DC-IGN	99.3 ± 0.1
InfoGAN 1	73.5 ± 0.9
VAE (untrained)	44.14 ± 2.5
VAE	61.58 ± 0.5
β -VAE	99.23 ± 0.1
Ours	92.34 ± 0.4

Coordinate	Description	Example
1	Background darkness
2	Azimuth
3	Bangs
4	Gender, bangs, and smiling
5	Smiling
6	Hair color
7	Hair color and hair style
8	Lighting color and bangs
9	Hair color and hair style
10	Hair color and hair style
11	Jawline
12	Smiling and bangs
13	Background color
14	Age and hairline
15	Age
17	Location of hair partition
18	Lighting and skin tone
20	Mouth open
21	Mouth open

Coordinate	Description	Example
1	Azimuth
2	Background color
3	Location of hair partition
4	Skin color
5	Hair direction
8	Background color
12	Hair length
14	Jawline
19	Skin tone
20	Hair style
22	Sunglasses
23	Gender
25	Background darkness
26	Hairline and skin tone
28	Lighting direction and rotation
32	Hair size, color, and smiling

Figure	DCGAN Base Feature Map Count	Latent Variable Size	Notes
13(c)	64	32	1
13(e)	64	64	1
13(g)	64	128	1
13(d),5, 4, 6(a), 6(b), 7	64	128	1
3, 11, 12	128	128	1
13(f)	64	256	1
13(h)	64	512	1
2	64	3	1, 3
1	64	10	1, 3
9(a)	64	16	1
9(b), 10	64	32	2

Coordinate	Description	Example
1	Background darkness and hair color
2	Azimuth, lighting, and hair color
3	Hairline and hair color
4	Azimuth
5	Shadow
6	Smiling, age, skin tone, gender
7	Smiling, age, jawline
8	Jawline and hairstyle

Coordinate	Description	Example
1	Background and hair darkness
2	Azimuth and hair darkness
3	Hair length, hair darkness, and lighting
4	Smiling and gender
5	Hair length, hair darkness, and gender
6	Smiling, bangs, and gender
7	Smiling and hairstyle
8	Smiling, jawline, and glaring expression
9	Smiling, bangs, and mouth open
10	Hairline
11	Raised eyebrows and skin tone
12	Raised eyebrows and location of hair partition
13	Raised eyebrows and location of hair partition
14	Lighting color

$$ D_{\bar{x}} f \big|{\bar{x} = x} = \left( D{\bar{b}{L - 1}} r_L \big|{\bar{b}{L - 1} = b{L - 1}} \right) \left( D_{\bar{b}{L - 2}} r{L - 1} \big|{\bar{b}{L - 2} = b_{p - 2}} \right) \cdots \ \left( D_{\bar{b}0} r_1 \big|{\bar{b}_0 = x} \right). $$

References

[TF] Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et~~al. Tensorflow: A system for large-scale machine learning. In OSDI, volume~~16, pp.\ 265--283, 2016.

[3DChairs] Aubry, M., Maturana, D., Efros, A.~A., Russell, B.~C., and Sivic, J. Seeing 3d chairs: exemplar part-based 2d-3d alignment using a large dataset of cad models. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 3762--3769, 2014.

[ADSurvey] Baydin, A.~G., Pearlmutter, B.~A., Radul, A.~A., and Siskind, J.~M. Automatic differentiation in machine learning: a survey. Journal of Machine Learning Research, 18\penalty0 (153):\penalty0 1--153, 2017.

[BigGAN] Brock, A., Donahue, J., and Simonyan, K. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096, 2018.

[UnderstandingDisentangling] Burgess, C.~P., Higgins, I., Pal, A., Matthey, L., Watters, N., Desjardins, G., and Lerchner, A. Understanding disentangling in $\beta$-vae. arXiv preprint arXiv:1804.03599, 2018.

[TCVAE] Chen, T.~Q., Li, X., Grosse, R., and Duvenaud, D. Isolating sources of disentanglement in variational autoencoders. arXiv preprint arXiv:1802.04942, 2018.

[InfoGAN] Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., and Abbeel, P. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp.\ 2172--2180, 2016.

[HFVAE] Esmaeili, B., Wu, H., Jain, S., Bozkurt, A., Siddharth, N., Paige, B., Brooks, D.~~H., Dy, J., and van~~de Meent, J.-W. Structured disentangled representations. stat, 1050:\penalty0 29, 2018.

[GAN] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Advances in neural information processing systems, pp.\ 2672--2680, 2014.

[betaVAE] Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. beta-vae: Learning basic visual concepts with a constrained variational framework. 2016.

[ProgressiveGAN] Karras, T., Aila, T., Laine, S., and Lehtinen, J. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017.

[FactorVAE] Kim, H. and Mnih, A. Disentangling by factorising. arXiv preprint arXiv:1802.05983, 2018.

[Adam] Kingma, D.~P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.

[AEVB] Kingma, D.~P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.

[CelebA] Liu, Z., Luo, P., Wang, X., and Tang, X. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), 2015.

[dSprites] Matthey, L., Higgins, I., Hassabis, D., and Lerchner, A. dsprites: Disentanglement testing sprites dataset. https://github.com/deepmind/dsprites-dataset/, 2017.

[GANConvergence] Mescheder, L., Geiger, A., and Nowozin, S. Which training methods for gans do actually converge? In International Conference on Machine Learning, pp.\ 3478--3487, 2018.

[SpectralWeightNorm] Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957, 2018.

[DCGAN] Radford, A., Metz, L., and Chintala, S. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.

[StochasticBackprop] Rezende, D.~J., Mohamed, S., and Wierstra, D. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4082, 2014.

[WeightNorm] Salimans, T. and Kingma, D.~P. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, pp.\ 901--909, 2016.

[RMSProp] Tieleman, T. and Hinton, G. Lecture 6.5---RmsProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.

[jvp_from_vjp] Townsend, J. A new trick for calculating jacobian vector products. https://j-towns.github.io/2017/06/12/A-new-trick.html, 2017.

[TPReLU] Xiang, S. and Li, H. On the effects of batch and weight normalization in generative adversarial networks. stat, 1050:\penalty0 22, 2017.

Additional GAN Disentanglement Results​

Introduction​

success.

Related Work​

Identifying Local Disentangled Factors​

Finding Quasi-Disentangled Paths​

Aligning the Local Disentangled Factors​

Results​

Conclusion​

Acknowledgements​

References

Forward- and reverse-mode automatic differentiation​

Eigenvector Alignment for~$ beta$-VAEs​

Generator Architectures and Training Procedure​

Implementation Details for Disentanglement Metric Score​

Additional GAN Disentanglement Results​

Supplementary Figures​

References​

Additional GAN Disentanglement Results

Introduction

Related Work

Identifying Local Disentangled Factors

Finding Quasi-Disentangled Paths

Aligning the Local Disentangled Factors

Results

Conclusion

Acknowledgements

Forward- and reverse-mode automatic differentiation

Eigenvector Alignment for~$ beta$-VAEs

Generator Architectures and Training Procedure

Implementation Details for Disentanglement Metric Score

Additional GAN Disentanglement Results

Supplementary Figures

References