ConcatDataset¶

class ConcatDataset(datasets)[source]¶

ConcatDataset is a concatenation of multiple datasets.

This dataset is used for assembleing multiple map-style datasets.

Parameters: datasets (list of Dataset) – list of datasets to be composed.
Returns: A Dataset which composes fields of multiple datasets.
Return type: Dataset

Examples

from megengine.data.dataset import ArrayDataset, ConcatDataset

data1 = np.random.randint(0, 255, size=(2, 1, 32, 32), dtype=np.uint8)
data2 = np.random.randint(0, 255, size=(2, 1, 32, 32), dtype=np.uint8)
label = np.random.randint(0, 10, size=(2,), dtype=int)
labe2 = np.random.randint(0, 10, size=(2,), dtype=int)
dataset1 = ArrayDataset(data1, label1)
dataset2 = ArrayDataset(data2, label2)
dataset = ConcatDataset([dataset1, dataset2])
seque_sampler = SequentialSampler(dataset, batch_size=2)

dataloader = DataLoader(
    dataset,
    sampler = seque_sampler,
    num_workers=3,
)

for step, data in enumerate(dataloader):
    print(data)

StreamDataset

VisionDataset