MapSampler

class MapSampler(dataset, batch_size=1, drop_last=False, num_samples=None, world_size=None, rank=None, seed=None)

Sampler for map dataset.

Parameters

• dataset – dataset to sample from.

• batch_size – batch size for batch method.

• drop_last – set True to drop the last incomplete batch, if the dataset size is not divisible by the batch size. If False and the size of dataset is not divisible by the batch_size, then the last batch will be smaller. Default: False

• num_samples – number of samples assigned to one rank.

• world_size – number of ranks.

• rank – rank id, non-negative interger within 0 and world_size.

• seed – seed for random operators.

batch()

Batch method provides a batch indices generator.

Return type

Iterator[List[Any]]

sample()

Return a list contains all sample indices.

scatter(indices)

Scatter method is used for splitting indices into subset, each subset will be assigned to a rank. Indices are evenly splitted by default. If customized indices assignment method is needed, please rewrite this method.

Return type

List

previous

Sampler

next

StreamSampler