How to get entire dataset from dataloader in PyTorch

python pytorch dataloader
Aakanksha W.S picture Aakanksha W.S · Aug 7, 2019 · Viewed 13.5k times · Source

How to load entire dataset from the DataLoader? I am getting only one batch of dataset.

This is my code

dataloader = torch.utils.data.DataLoader(dataset=dataset, batch_size=64)
images, labels = next(iter(dataloader))

Answer

asymptote picture asymptote · Aug 7, 2019

You can set batch_size=dataset.__len__() in case dataset is torch Dataset, else something like batch_szie=len(dataset) should work.

Beware, this might require a lot of memory depending upon your dataset.

Related questions

What does next() and iter() do in PyTorch's DataLoader()

I have the following code: import torch import numpy as np import pandas as pd from torch.utils.data import TensorDataset, DataLoader # Load dataset df = pd.read_csv(r'../iris.csv') # Extract features and target data = df.drop('target',axis=1).…

Read More
How does the __getitem__'s idx work within PyTorch's DataLoader?

I'm currently trying to use PyTorch's DataLoader to process data to feed into my deep learning model, but am facing some difficulty. The data that I need is of shape (minibatch_size=32, rows=100, columns=41). The __getitem__ coad that I have …

Read More
How to check if pytorch is using the GPU?

I would like to know if pytorch is using my GPU. It's possible to detect with nvidia-smi if there is any activity from the GPU during the process, but I want something written in a python script. Is there a …

Read More