AWS Glue Crawler Not Creating Table

amazon-web-services aws-glue
Vince picture Vince · Nov 1, 2017 · Viewed 11.2k times · Source

I have a crawler I created in AWS Glue that does not create a table in the Data Catalog after it successfully completes.

The crawler takes roughly 20 seconds to run and the logs show it successfully completed. CloudWatch log shows:

  • Benchmark: Running Start Crawl for Crawler
  • Benchmark: Classification Complete, writing results to DB
  • Benchmark: Finished writing to Catalog
  • Benchmark: Crawler has finished running and is in ready state

I am at a loss as to why the tables in the data catalog are not being created. AWS Docs are not of much help debugging.

Answer

Ray picture Ray · Jan 10, 2018

check the IAM role associated with the crawler. Most likely you don't have correct permission.

When you create the crawler, if you choose to create an IAM role(the default setting), then it will create a policy for S3 object you specified only. if later you edit the crawler and change the S3 path only. The role associated with the crawler won't have permission to the new S3 path.

Related questions

AWS Glue to Redshift: Is it possible to replace, update or delete data?

Here are some bullet points in terms of how I have things setup: I have CSV files uploaded to S3 and a Glue crawler setup to create the table and schema. I have a Glue job setup that writes the …

Read More
Can we consider AWS Glue as a replacement for EMR?

Just a quick question to clarify from Masters, since AWS Glue as an ETL tool, can provide companies with benefits such as, minimal or no server maintenance, cost savings by avoiding over-provisioning or under-provisioning resources, besides running on spark, I …

Read More
Use AWS Glue Python with NumPy and Pandas Python Packages

What is the easiest way to use packages such as NumPy and Pandas within the new ETL tool on AWS called Glue? I have a completed script within Python I would like to run in AWS Glue that utilizes NumPy …

Read More