PyTorch Layer Calculator

Parameters are the learnable weights and biases in a neural network model. During training, these values are updated via backpropagation to minimize the loss. A Linear(in=128, out=64) layer has 128×64=8,192 weights plus 64 biases = 8,256 parameters. The total parameter count determines model complexity, memory usage, and training time.

Model memory = total parameters × bytes per parameter. float32 (default) uses 4 bytes per parameter; float16/bfloat16 uses 2 bytes; int8 uses 1 byte. A model with 1M float32 parameters uses ~4 MB for weights alone. During training, you also need memory for gradients (~same as weights) and optimizer states (Adam uses ~2× more). Total training memory is roughly 3-4× the model size.

Linear (fully connected) layer connects every input to every output. Parameters = in_features × out_features + bias. Conv2d applies convolutional filters over spatial data (images). Parameters = out_channels × in_channels × kernel_h × kernel_w + bias. Conv2d is much more parameter-efficient for spatial data because filters share weights across positions.

An Embedding layer maps integer token IDs to dense vectors. It's like a lookup table: vocab_size × embedding_dim. For example, Embedding(50000, 768) maps 50K token IDs to 768-dimensional vectors — used in transformers and NLP models. Parameters = vocab_size × embedding_dim (no bias).

BatchNormalization (BatchNorm1d/2d) normalizes layer activations across the batch dimension, stabilizing training and allowing higher learning rates. It has 2 learnable parameters per feature (gamma and beta): BatchNorm1d(64) has 128 parameters. LayerNorm normalizes across feature dimensions instead — preferred in Transformers.

LSTM (Long Short-Term Memory) is a recurrent network layer that processes sequences. It maintains a cell state and hidden state to capture long-range dependencies. Parameters = 4 × (hidden × (input + hidden) + hidden). Use LSTM for time series, text sequences, and audio when order matters. For modern NLP, Transformer-based models (attention) often outperform LSTMs but require more memory.