Memory error in amazon sagemaker

amazon-sagemaker
VaRun Sabu picture VaRun Sabu · Jul 23, 2018 · Viewed 7.6k times · Source

Memory error occurs in amazon sagemaker when preprocessing 2 gb of data which is stored in s3. No problem in loading the data. Dimension of data is 7 million rows and 64 columns. One hot encoding is also not possible. Doing so results in memory error. Notebook instance is ml.t2.medium. How to solve this issue?

Answer

Julien Simon picture Julien Simon · Jul 23, 2018

I assume you're processing on the data on the notebook instance, right? t2.medium has only 4GB of RAM, so it's quite possible you're simply running out of memory.

Have you tried a larger instance? The specs are here: https://aws.amazon.com/sagemaker/pricing/instance-types/

Related questions

Load S3 Data into AWS SageMaker Notebook

I've just started to experiment with AWS SageMaker and would like to load data from an S3 bucket into a pandas dataframe in my SageMaker python jupyter notebook for analysis. I could use boto to grab the data from S3, …

Read More
How to schedule tasks on SageMaker

I have a notebook on SageMaker I would like to run every night. What's the best way to schedule this task. Is there a way to run a bash script and schedule Cron job from SageMaker?

Read More
Reading a large csv from a S3 bucket using python pandas in AWS Sagemaker

I'm trying to load a large CSV (~5GB) into pandas from S3 bucket. Following is the code I tried for a small CSV of 1.4 kb : client = boto3.client('s3') obj = client.get_object(Bucket='grocery', Key='stores.csv') body = …

Read More