• Main points: LayerNorm and RMSNorm have replaced batch normalization in deep learning. Implementing a fast LayerNorm kernel is needed for running big models in the browser on the GPU. GPU programming involves understanding reductions. One-pass and two-pass algorithms have pros and cons. Welford's algorithm offers the best of both worlds with high performance and numerical precision using WebGPU Subgroups Proposal.
• Key information: PyTorch defines LayerNorm with specific operations. Reductions are a core GPU programming primitive. One-pass algorithm uses shared memory. Two-pass algorithm avoids subtraction but is slower. Welford's algorithm uses recurrence relations and subgroups. Performance benchmarks show different algorithms' results.
• Important details: In one-pass algorithm, each workgroup computes mean and variance of a single row. Two-pass algorithm computes mean first and subtracts from elements for variance. Welford's algorithm updates mean and variance with specific formulas. WebGPU Subgroups Proposal is used for peak performance. Benchmarking repo shows more results. References are provided for more information.