Neural Network Learning Rate Calculator

The learning rate (LR) is a hyperparameter that controls how much model weights change in response to the estimated error during training. Too high and the model diverges or oscillates; too low and training is painfully slow and can get stuck in local minima. It is arguably the most impactful single hyperparameter to tune — the right LR can mean the difference between a model that converges in hours versus one that never converges at all.

The LR Range Test, introduced by Leslie Smith in 2015, involves starting training with a very small LR (e.g., 1e-7) and exponentially increasing it each mini-batch. You plot the loss against the LR — the loss will initially decrease, then start rising steeply. The optimal LR is typically 1–2 orders of magnitude below the point of steepest increase. This test takes only a few hundred iterations and is far more efficient than a full grid search.

Warmup gradually increases the LR from a small value (near zero) to the target LR over the first N steps. It prevents early instability when model weights are random and gradients are noisy. Warmup is strongly recommended for: (1) transformer/attention-based models (BERT, GPT, ViT), (2) large batch training (batch size ≥ 512), (3) Adam and its variants, (4) fine-tuning pretrained models. A common rule is warmup for 5–10% of total training steps.

Step decay reduces the LR by a fixed factor (e.g., ÷10) every N epochs — simple and predictable. Cosine annealing smoothly decays the LR following a cosine curve from max to min, which often leads to better final loss because the smooth transitions avoid jarring changes. Cyclical LR (CLR) oscillates between a minimum and maximum LR, allowing the model to periodically escape local minima — effective for training from scratch. For modern deep learning, cosine annealing with warmup (used by most SOTA papers) is the default safe choice.

Larger batches produce more accurate gradient estimates, which means you can use a proportionally higher LR without instability. The 'linear scaling rule' (Goyal et al., 2017) suggests scaling LR linearly with batch size: if you double the batch size, double the LR. However, this rule breaks down for very large batches (> 8K) where warmup and careful tuning become essential. Smaller batches introduce noise that acts as a regularizer, allowing smaller models to sometimes generalize better with lower LRs.