LiteFFN: How We Made LTX-2 19B Run Faster with Low-Rank Magic

Mar 5, 2026
LiteFFN MoonLite

The Problem: Attention Got Fast, FFN Didn't

If you've been following the AI video generation space, you've probably heard of models like LTX-2 and OpenAI's Sora. These models can generate stunning video from text descriptions, but they are computationally expensive.

LTX-2 is a 19-billion parameter Diffusion Transformer that can generate native 4K video without upscaling, up to 50 FPS, synchronized audio in the same forward pass, and up to 20 seconds of high-fidelity video.

That power comes at a cost: even on an H100 GPU, generation can take [X] seconds and requires [X] GB of memory (or 12 GB minimum with FP8 quantization). This is manageable for research, but a major blocker for scaled production deployment.

Where Does the Time Go?

Transformer-based models have two main computational blocks:

• Attention layers: capture relationships between different parts of the input
• Feed-forward networks (FFN): process each position independently

The community has heavily optimized attention (Flash Attention 1-3, Sage Attention, LiteAttention), but FFN layers have been far less optimized.

As attention gets faster, FFN becomes the bottleneck. On LTX-2 inference, FFN is nearly 50% of transformer runtime, and this is the gap we targeted.

The Solution: Compress the FFN Smartly

Our approach combines two ideas:

• Low-rank decomposition: approximate large weight matrices with smaller ones
• Quantization: store the remaining error in fewer bits

The Intuition

In LTX-2, FFN matrices can be as large as 8192x2048 with tens of millions of parameters. Not all parameters carry equal signal, and many capture overlapping structure.

Low-rank decomposition represents one large matrix as the product of two smaller matrices, preserving most useful behavior at much lower compute and memory cost. Quantization then captures the leftover correction terms efficiently.^[1]

How It Works

Step 1: Collect Calibration Data

We run the model on a sample of real prompts and collect per-layer input statistics to characterize typical activation structure.

For each FFN layer, we compute:

R = E[x * x^T]

For 2048-dimensional inputs, each autocorrelation matrix has 4 million entries. Across many layers, memory grows quickly, so we compute R incrementally:

# Pseudocode for incremental autocorrelation
R_sum = zeros((dim, dim))
N = 0

for batch in calibration_data:
    x = get_activations(batch)  # Shape: [batch, seq_len, dim]
    x_flat = x.reshape(-1, dim) # Flatten to [N_samples, dim]
    R_sum += x_flat.T @ x_flat  # Accumulate outer product
    N += x_flat.shape[0]

R = R_sum / N

Step 2: Compute Effective Weights

We use calibration statistics to transform the original weight matrix into a form that is easier to compress:

W_effective = W @ R^(1/2)

This reweights columns by practical usage, improving how efficiently SVD can approximate the matrix for real workloads.

Step 3: Decompose

W_effective = U @ S @ V^T

We keep the top-r singular components:

W_low_rank_eff = U[:, :r] @ S[:r] @ V[:r, :]^T

Step 4: Handle the Remainder

Remainder = W - W_low_rank

This residual is quantized to 4-bit or 8-bit formats.

Step 5: Inference

def forward(x):
    # Low-rank path (full precision, small matrices)
    y_lowrank = A @ (B @ x)

    # Remainder path (quantized)
    y_remainder = Q_quantized @ x

    return y_lowrank + y_remainder + bias

Quantization Options

Current production format on Hopper:

FP8 E4M3

• Bits: 8 (4 exponent, 3 mantissa)
• Range: approximately +/-448
• Pros: native H100 tensor core support, strong accuracy
• Cons: 2x memory versus FP4

Planned formats for LiteFFN on Blackwell and future Hopper support include NVFP4 E2M1 and MXFP4.

The Fun Part: Multiply-Free FP4

For non-native FP4 support, NVFP4's limited value set allows replacing floating-point multiplies with shift/add operations.

Results

Average Runtime

Allocated VRAM Reduction

Video Samples (baseline vs LiteFFN r=32 / r=64 / r=512)

Click a thumbnail to play the video.

Baseline	LiteFFN r=32	LiteFFN r=64	LiteFFN r=512

Quality Measurements

PSNR average over different seeds, per prompt (dB)^[2] (Higher is better; 20 dB corresponds to a mean squared error of roughly 10⁻²):

Prompt PSNR Summary

Performance Benchmark on Shapes Captured from LTX-Video

Multiplier columns are speedup ratios vs baseline linear (>1 faster, <1 slower).

Future Work

• Attention projection decomposition (Q/K/V/O)
• Adaptive per-layer rank selection from singular value decay
• Dynamic rank by denoising timestep
• More optimized FP4 kernels with warp-level tuning
• Extension to other video models, world models, and VLMs

Acknowledgments

This work would not exist without SVDQuant. We also thank the Lightricks team for open-sourcing LTX-2 and the Flash Attention authors for pushing efficient attention forward.

References

• LiteFFN repository: https://github.com/moonmath-ai/LiteFFN
• SVDQuant Paper: https://arxiv.org/abs/2411.05007
• LTX-2: https://huggingface.co/Lightricks/LTX-2