SyncBatchNorm¶
- class SyncBatchNorm(num_features, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True, freeze=False, group=WORLD, **kwargs)[source]¶
Applies Synchronized Batch Normalization for distributed training.
- Parameters
num_features – usually \(C\) from an input of shape \((N, C, H, W)\) or the highest ranked dimension of an input less than 4D.
eps – a value added to the denominator for numerical stability. Default: 1e-5
momentum – the value used for the
running_mean
andrunning_var
computation. Default: 0.9affine – a boolean value that when set to True, this module has learnable affine parameters. Default: True
track_running_stats – when set to True, this module tracks the running mean and variance. When set to False, this module does not track such statistics and always uses batch statistics in both training and eval modes. Default: True
freeze – when set to True, this module does not update the running mean and variance, and uses the running mean and variance instead of the batch mean and batch variance to normalize the input. The parameter takes effect only when the module is initilized with track_running_stats as True. Default: False
group (
Optional
[Group
]) – communication group, caculate mean and variance between this group. Default:WORLD