Why SVD Should Only Appear at Inference: A Gradient Analysis of Rotation Representations

Predicting 3D rotations from neural networks sounds straightforward—output a rotation matrix and move on. But rotations live on $\mathrm{SO}(3)$, a curved 3-dimensional manifold inside $\mathbb{R}^{3 \times 3}$ with the constraints $R^\top R = I$ and $\det(R) = 1$. How you enforce these constraints during training turns out to matter a lot.

In this post, I’ll walk through the key results of our new paper, Training Without Orthogonalization, Inference With SVD, which provides the theoretical foundation for a surprisingly effective recipe:

Train with direct 9D regression (no orthogonalization). Apply SVD only at inference.

The Rotation Representation Zoo

Before diving in, let’s quickly review the landscape. Neural networks can output rotations in several ways:

The last row is the sweet spot. But why? The answer lies in what happens to gradients when you backpropagate through orthogonalization.

The Core Problem: SVD Gradients Are Broken During Training

When you put SVD inside the training loop, you backpropagate through the mapping $M \mapsto R = UV^\top$ where $M = U\Sigma V^\top$. The resulting Jacobian has a very specific structure that creates three problems.

Problem 1: Gradient Explosion and Poor Conditioning

Theorem (SVD Jacobian Spectrum). The Jacobian $J_{\text{SVD}}$ of the mapping $M \mapsto R = UV^\top$ has rank 3 with a 6-dimensional null space. Its three nonzero singular values are:

with spectral norm $2/(s_2 + s_3)$ and condition number $\kappa = (s_1 + s_2)/(s_2 + s_3)$.

Proof sketch. The key is to work in the SVD’s own coordinate system. Given the SVD $M = U\Sigma V^\top$, any perturbation $\mathrm{d}M$ can be rotated into the frame $P = U^\top \mathrm{d}M\, V$. Since $U$ and $V$ are orthogonal, this preserves norms ($\lVert \mathrm{d}M \rVert_F = \lVert P \rVert_F$).

Differentiating $M = U\Sigma V^\top$ and using the orthogonality of $U, V$ (which forces $U^\top \mathrm{d}U$ and $V^\top \mathrm{d}V$ to be antisymmetric), we arrive at a $2\times 2$ system for each off-diagonal pair $(i,j)$:

where $A = U^\top \mathrm{d}U$ and $\Omega = V^\top \mathrm{d}V$ encode how the singular vectors rotate in response to the perturbation. The determinant of this system is $s_j^2 - s_i^2$—and here lies the source of all SVD gradient problems.

Solving and substituting into $\mathrm{d}R = U(A - \Omega)V^\top$, the rotation differential $\Phi = A - \Omega$ has entries:

Now the 9 entries of $P$ decompose into three orthogonal subspaces:

1. Diagonal entries $P_{ii}$ (3 dims): map to $\Phi_{ii} = 0$ — null space
2. Symmetric off-diagonal $(P_{ij} + P_{ji})/\sqrt{2}$ (3 dims): cancel in $P_{ji} - P_{ij}$ — null space
3. Antisymmetric off-diagonal $(P_{ji} - P_{ij})/\sqrt{2}$ (3 dims): each maps with gain $2/(s_i + s_j)$

Since these subspaces are orthogonal and map to orthogonal outputs, the three nonzero singular values of $J_{\text{SVD}}$ are exactly ${2/(s_i + s_j)}_{i < j}$. $\square$

The spectral norm $2/(s_2 + s_3)$ blows up as $O(1/s_3)$ when the smallest singular value approaches zero—exactly what happens early in training when the network hasn’t learned near-orthogonal outputs.

Even when gradients aren’t exploding, the condition number $\kappa = (s_1 + s_2)/(s_2 + s_3)$ creates unequal scaling across rotation axes. Standard optimizers use isotropic step sizes, so this anisotropy means some directions converge fast while others lag behind.

Under a Gaussian noise model $M = I + \sigma N$, we can derive the expected condition number in closed form using random matrix theory. The singular values of $M$ are approximately $s_k \approx 1 + \sigma \lambda_k(W)$ where $W = (N + N^\top)/2$ is a GOE matrix. The expected ordered eigenvalues of the $3 \times 3$ GOE are $\mathbb{E}[\lambda_1] = c_3 \approx 1.466$, $\mathbb{E}[\lambda_2] = 0$, $\mathbb{E}[\lambda_3] = -c_3$, giving:

Even at moderate noise ($\sigma = 0.3$, typical of early-to-mid training), $\kappa \approx 1.57$—a nontrivial anisotropy. Direct regression maintains $\kappa = 1$ for all $\sigma$.

Problem 2: Two-Thirds of Gradient Information Is Lost

Here’s perhaps the most striking result. The SVD Jacobian has rank 3 with a 6-dimensional null space. Those 6 “lost” dimensions correspond to the normal space of $\mathrm{SO}(3)$—they encode how far the predicted matrix is from being a valid rotation.

Proposition (Gradient Information Retention). For an isotropic random gradient $\nabla_R \mathcal{L} \sim \mathcal{N}(0, I_9)$, define the gradient information retention as $\eta(J) = \mathbb{E}[\lVert J^\top \nabla_R \mathcal{L} \rVert^2] / \mathbb{E}[\lVert \nabla_R \mathcal{L} \rVert^2]$. Then:

• SVD-Train near $\mathrm{SO}(3)$: $\eta(J_{\text{SVD}}) = 1/3$
• Direct 9D regression: $\eta(J_{\text{id}}) = 1$

Proof sketch. For $\mathbf{g} \sim \mathcal{N}(0, I_9)$, we have $\mathbb{E}[\lVert J^\top \mathbf{g} \rVert^2] = \text{tr}(JJ^\top)$. From the spectrum theorem above, $J_{\text{SVD}}$ has nonzero singular values $\lbrace 2/(s_i + s_j) \rbrace_{i \lt j}$, so:

Near $\mathrm{SO}(3)$ where $s_1, s_2, s_3 \approx 1$, each term equals $4/4 = 1$, giving $\eta = 3/9 = 1/3$. For the identity Jacobian, $\text{tr}(I_9) = 9$ and $\eta = 9/9 = 1$. $\square$

The 6-dimensional null space decomposes neatly into two geometric pieces, both living in the normal space of $\mathrm{SO}(3)$ at $R$:

• Diagonal entries (3 dims): perturbations that change singular values, i.e., how far $M$ is from having unit singular values
• Symmetric off-diagonal (3 dims): shearing/anisotropic scaling deformations

SVD discards both, so the optimizer receives no signal pushing $M$ toward orthogonality. The loss landscape is completely flat in these 6 directions. Direct regression retains this signal—and this is exactly why networks trained with MSE naturally converge to near-orthogonal outputs without any explicit constraint.

Seeing It in the Gradients

The gradient scatter plot below makes these problems viscerally clear:

Gradient of loss w.r.t. $M_{11}$ vs. the error $M_{11} - R^_{11}$ (10,000 samples, $\sigma = 0.5$). Left: Direct 9D gradients lie cleanly on the diagonal—each parameter depends only on its own error. Center: SVD-Train gradients scatter wildly; red points (small singular value gap) are most erratic. Right: Gram-Schmidt also produces ambiguous, scattered gradients.*

The left panel is what you want: a clean, diagonal relationship where gradient magnitude is proportional to error. The center and right panels show the chaos that orthogonalization introduces.

Can We Fix SVD Gradients? No.

A natural question: can we stabilize the SVD backward pass rather than avoid it? Wang et al. (2021) proposed a Taylor expansion approximation that clamps the problematic $1/(s_i^2 - s_j^2)$ terms.

Our analysis shows this is a dead end. Any stabilization that bounds the gradient magnitude must introduce gradient direction error. The tradeoff is quantifiable: stabilized SVD gradients can deviate by up to 5.71 degrees from the true gradient direction. You’re trading explosion for inaccuracy.

The better solution is to simply not backpropagate through SVD at all.

Why 9D Beats 6D: Gram-Schmidt’s Asymmetry Problem

The 6D representation (Gram-Schmidt orthogonalization) avoids some of SVD’s problems but introduces its own: asymmetric gradient flow.

Gram-Schmidt is inherently sequential:

1. Normalize $\mathbf{t}_1’$ to get $\mathbf{r}_1’$
2. Orthogonalize $\mathbf{t}_2’$ against $\mathbf{r}_1’$, then normalize to get $\mathbf{r}_2’$
3. Compute $\mathbf{r}_3’ = \mathbf{r}_1’ \times \mathbf{r}_2’$ (no free parameters)

Theorem (Gram-Schmidt Jacobian Asymmetry). The GS Jacobian $J_{\text{GS}} \in \mathbb{R}^{9 \times 6}$ has the following structure:

1. $\partial \mathbf{r}_1’ / \partial \mathbf{t}_1’$ depends only on $\mathbf{t}_1’$ (rank 2 due to radial degeneracy)
2. $\partial \mathbf{r}_2’ / \partial \mathbf{t}_1’ \neq 0$ but $\partial \mathbf{r}_1’ / \partial \mathbf{t}_2’ = 0$ (one-directional coupling)
3. $\mathbf{r}_3’ = \mathbf{r}_1’ \times \mathbf{r}_2’$ has no dedicated parameters
4. $\kappa(J_{\text{GS}}) \geq \lVert \mathbf{t}_1’ \rVert / \lVert \mathbf{r}_2’’ \rVert$ where $\mathbf{r}_2’’ = \mathbf{t}_2’ - (\mathbf{r}_1’ \cdot \mathbf{t}_2’)\mathbf{r}_1’$

Proof sketch. The key insight is that GS is a chain of normalizations and projections, each with a known Jacobian. Normalization $\mathbf{x} \mapsto \mathbf{x}/\lVert \mathbf{x} \rVert$ has Jacobian $(I - \hat{\mathbf{x}}\hat{\mathbf{x}}^\top)/\lVert \mathbf{x} \rVert$—rank 2, with a null space along $\hat{\mathbf{x}}$ (stretching a unit vector doesn’t change its direction).

Since $\mathbf{r}_1’ = \mathbf{t}_1’/\lVert \mathbf{t}_1’ \rVert$ depends only on $\mathbf{t}_1’$, we immediately get $\partial \mathbf{r}_1’/\partial \mathbf{t}_2’ = 0$. But the reverse isn’t true: $\mathbf{r}_2’$ depends on $\mathbf{r}_1’$ through the orthogonalization step $\mathbf{r}_2’’ = \mathbf{t}_2’ - (\mathbf{r}_1’ \cdot \mathbf{t}_2’)\mathbf{r}_1’$, so $\partial \mathbf{r}_2’/\partial \mathbf{t}_1’ \neq 0$ via the chain rule through $\mathbf{r}_1’$.

This creates a strict gradient hierarchy:

• $\mathbf{t}_1’$ receives gradient feedback from all 3 columns of the rotation
• $\mathbf{t}_2’$ is blind to column 1 errors—it only gets feedback from columns 2 and 3
• Column 3 has no parameters at all—it’s entirely derived

The condition number bound follows from the normalization step for $\mathbf{r}_2’$: its Jacobian scales as $1/\lVert \mathbf{r}_2’’ \rVert$, which diverges as $\mathbf{t}_1’$ and $\mathbf{t}_2’$ become parallel (making the orthogonalization residual $\mathbf{r}_2’’ \to 0$). $\square$

The per-column projection error plot below shows the consequence:

Per-column squared error after projection. Left: Gram-Schmidt produces monotonically increasing error across columns—column 1 is privileged, column 3 (derived from the cross product) has the most error. Right: SVD distributes error equally across all columns.

SVD treats all columns democratically. Gram-Schmidt doesn’t. This asymmetry means the 6D representation creates an uneven optimization landscape where some rotation axes are systematically harder to learn than others.

Why SVD Wins at Inference: 3x Error Reduction

So if we skip orthogonalization during training, what should we use at inference? SVD, and it’s not close.

SVD projection is the closest rotation matrix in Frobenius norm:

Proposition (SVD Projection Reduces Error). Let $M = R^\ast + \sigma N$ with $R^\ast \in \mathrm{SO}(3)$ and $\sigma$ small. Decompose the perturbation in the body frame: ${R^\ast}^\top N = A + S$ where $A$ is antisymmetric (tangent to $\mathrm{SO}(3)$) and $S$ is symmetric (normal to $\mathrm{SO}(3)$). Then:

Proof sketch. Write $M = R^\ast(I + \sigma A + \sigma S)$. For small $\sigma$, the polar decomposition of $I + \sigma A + \sigma S$ separates into an orthogonal factor $(I + \sigma A + O(\sigma^2))$ and a positive-definite factor $(I + \sigma S + O(\sigma^2))$—you can verify this by multiplying out and matching terms order by order.

Since $\text{SVDO}^+$ extracts the orthogonal factor of the polar decomposition (this is what makes it the nearest rotation), and using left-equivariance $\text{SVDO}^+(R^\ast X) = R^\ast \cdot \text{SVDO}^+(X)$:

The error is then $\sigma^2 \lVert A \rVert_F^2 + O(\sigma^3)$. Since $A$ and $S$ are orthogonal under the Frobenius inner product ($\text{tr}(A^\top S) = -\text{tr}(AS) = \text{tr}(AS) = 0$ by antisymmetry) and $\lVert N \rVert_F^2 = \lVert A \rVert_F^2 + \lVert S \rVert_F^2$, the error reduction follows. $\square$

Corollary (Factor-of-3 Error Reduction). Under isotropic Gaussian noise ($N_{ij} \sim \mathcal{N}(0,1)$), SVD achieves 3x lower projection error than the raw matrix:

The proof is pure dimension counting: let $P = {R^\ast}^\top N$ (also i.i.d. Gaussian since $R^\ast$ is orthogonal). Its antisymmetric part $A_{ij} = (P_{ij} - P_{ji})/2$ has variance $1/2$ per entry, and $\lVert A \rVert_F^2 = 2\sum_{i \lt j} A_{ij}^2$, giving $\mathbb{E}[\lVert A \rVert_F^2] = 2 \cdot 3 \cdot 1/2 = 3$. Meanwhile $\mathbb{E}[\lVert N \rVert_F^2] = 9$. So the ratio is $3/9 = 1/3$: out of 9 dimensions, 3 are tangential to $\mathrm{SO}(3)$ and survive projection, while 6 are normal and get removed.

Compared to Gram-Schmidt ($\mathbb{E} \approx 6\sigma^2$), SVD ($\mathbb{E} \approx 3\sigma^2$) is 2x better:

Gram-Schmidt’s sequential orthogonalization only removes 3 of the 6 normal-space dimensions, while SVD removes all 6.

Expected squared projection error vs. noise level $\sigma$. SVD (blue) consistently achieves roughly half the error of Gram-Schmidt (orange). The right panel shows the error ratio stays near the theoretical prediction of 2.

SVD Is Also Coordinate-Independent

There’s another important advantage. SVD projection is globally equivariant: rotating the input rotates the output. Gram-Schmidt is only equivariant under rotations of its first basis vector.

Coordinate dependence test. Left: SVD produces zero inconsistency (spike at 0 degrees), confirming $\text{SVDO}^+(MR_2)R_2^\top = \text{SVDO}^+(M)$ for any rotation $R_2$. Right: Gram-Schmidt spreads over 10–60 degrees of inconsistency, showing strong coordinate dependence.

This means SVD gives you the same answer regardless of which coordinate frame you work in. Gram-Schmidt’s answer depends on which two columns you chose to parameterize.

The Convergence Story

The convergence analysis seals the deal.

Proposition (Convergence Rates). Consider gradient descent on the Frobenius loss with step size $\eta$:

(A) Direct 9D regression. The update $M_{t+1} = (1 - 2\eta)M_t + 2\eta R^\ast$ converges linearly:

The rate $\rho = \lvert 1 - 2\eta \rvert$ is state-independent and achieves one-step convergence at $\eta = 1/2$.

(B) SVD-Train. The effective Hessian $H = 2J_{\text{SVD}}^\top J_{\text{SVD}}$ has eigenvalues $4/(s_i + s_j)^2$ on the column space. The per-component contraction rate is $\rho_{ij} = \lvert 1 - 4\eta/(s_i + s_j)^2 \rvert$, giving a slowest rate of:

Proof sketch. For direct regression, $\nabla_M \mathcal{L} = 2(M - R^\ast)$, so the error $E_t = M_t - R^\ast$ satisfies $E_{t+1} = (1 - 2\eta) E_t$. The Hessian is $2I_9$, confirming state-independence.

For SVD-Train, the gradient is

Near convergence, the effective Hessian inherits the spectrum of $J_{\text{SVD}}^\top J_{\text{SVD}}$, whose eigenvalues are $\lbrace 4/(s_i+s_j)^2 \rbrace_{i \lt j}$ (squared singular values of $J_{\text{SVD}}$). Each eigendirection contracts at its own rate. The step size must satisfy $\eta \lt (s_2 + s_3)^2/2$ to prevent divergence in the fastest direction, while the slowest direction barely moves. $\square$

The practical upshot:

The $2\times$ slowdown near $\mathrm{SO}(3)$ has a clean geometric interpretation: the SVD Jacobian is a rank-3 projector onto the tangent space, which “sees” only 3 of 9 degrees of freedom. Direct regression sees all 9, giving twice the effective curvature per step. Far from $\mathrm{SO}(3)$, the anisotropy creates the classic ill-conditioning bottleneck: the step size must be small (to control the fast direction) while the slow direction barely moves.

When Are Quaternions Fine?

Given all the advantages of 9D, you might wonder: is there ever a reason to use quaternions? Actually, yes.

Quaternion normalization shares $\kappa = 1$ with direct 9D regression and requires only 4 output dimensions instead of 9. Their sole theoretical disadvantage is the double cover ($q \equiv -q$): the same rotation maps to two antipodal points on the unit sphere. This forces either a non-smooth loss like $\min(\lVert q - q^\ast \rVert^2, \lVert q + q^\ast \rVert^2)$ or a hemisphere constraint with its own discontinuity.

The key insight: this discontinuity only matters when your data includes rotations near the 180-degree boundary. When the rotation distribution is concentrated—small perturbations around a reference pose, or a narrow viewpoint range—the topological obstruction falls outside the data support and quaternions can outperform higher-dimensional representations.

Use quaternions when:

• Rotations are small perturbations around a reference pose
• Viewpoint range is narrow and known in advance
• Model size or efficiency is critical (4D vs. 9D)

Use 9D + SVD-Inference when:

• Rotations span a large portion of $\mathrm{SO}(3)$
• The rotation distribution is unknown a priori
• You want clean theoretical guarantees without tuning

The Recipe

Putting it all together, the evidence points to a clean, simple pipeline:

During training:

• Output all 9 elements of a $3 \times 3$ matrix $M$
• Use MSE loss directly: $\mathcal{L} = \lVert M - R^\ast \rVert_F^2$
• No orthogonalization, no special layers, no hyperparameters
• The Jacobian is the identity ($\kappa = 1$), all gradient information is preserved

At inference:

• Project to $\mathrm{SO}(3)$ via SVD: $\hat{R} = U \text{diag}(1, 1, \det(UV^\top)) V^\top$
• Optimal projection with 3x error reduction
• Coordinate-independent

Why this is self-consistent. The training loss $\epsilon^2 = \mathbb{E}[\lVert M - R^\ast \rVert_F^2]$ serves as a conservative upper bound on inference error: $\mathbb{E}[\lVert \hat{R} - R^\ast \rVert_F^2] \leq \epsilon^2$, with typical value $\approx \epsilon^2/3$ under isotropic residuals. Minimizing MSE simultaneously minimizes the tangential error (which persists after SVD projection) and the normal error (which SVD removes). No training signal is wasted: the normal-error gradients implicitly regularize the network toward $\mathrm{SO}(3)$, ensuring the perturbative regime where SVD projection is most effective.

This approach was empirically validated by Gu et al. (2024) under the name PRoM (Pseudo Rotation Matrices), achieving 54.8 PA-MPJPE on human pose estimation compared to 55.6 for 9D+GS and 56.7 for 6D+GS. Our paper provides the why behind these numbers.

Key Takeaways

1. SVD during training is harmful because it creates gradient explosion ($O(1/s_3)$), poor conditioning, and discards 2/3 of gradient information.

2. Stabilizing SVD gradients doesn’t help—you trade explosion for direction error. Avoidance beats mitigation.

3. 6D (Gram-Schmidt) has asymmetric gradients—different parameters get different amounts of feedback, making 9D strictly preferable.

4. SVD at inference is optimal—it’s the closest rotation in Frobenius norm, achieves 3x lower projection error than GS, and is coordinate-independent.

5. The simplest approach wins: direct 9D regression with MSE loss during training, SVD projection at inference.

Sometimes the best thing you can do with a sophisticated mathematical operation is keep it out of the backward pass.

Citation

@article{choy2026rotation,
  title={Training Without Orthogonalization, Inference With SVD: A Gradient Analysis of Rotation Representations},
  author={Choy, Chris},
  year={2026}
}

References

• Zhou et al., “On the Continuity of Rotation Representations in Neural Networks,” CVPR 2019 ¹
• Levinson et al., “An Analysis of SVD for Deep Rotation Estimation,” NeurIPS 2020 ²
• Gu et al., “Pseudo Rotation Matrices (PRoM),” 2024 ³
• Wang et al., “Robust Differentiable SVD,” TPAMI 2021 ⁴
• Geist et al., “Learning SO(3) Equivariant Representations,” 2024 ⁵