What is the difference between Google Cloud Dataflow and Google Cloud Dataproc?

google-cloud-platform google-cloud-dataflow google-cloud-dataproc
KosiB picture KosiB · Sep 27, 2017 · Viewed 37.4k times · Source

I am using Google Data Flow to implement an ETL data ware house solution.

Looking into google cloud offering, it seems DataProc can also do the same thing.

It also seems DataProc is little bit cheaper than DataFlow.

Does anybody know the pros / cons of DataFlow over DataProc

Why does google offer both?

Answer

Andrew Mo picture Andrew Mo · Sep 27, 2017

Yes, Cloud Dataflow and Cloud Dataproc can both be used to implement ETL data warehousing solutions.

An overview of why each of these products exist can be found in the Google Cloud Platform Big Data Solutions Articles

Quick takeaways:

  • Cloud Dataproc provides you with a Hadoop cluster, on GCP, and access to Hadoop-ecosystem tools (e.g. Apache Pig, Hive, and Spark); this has strong appeal if you are already familiar with Hadoop tools and have Hadoop jobs
  • Cloud Dataflow provides you with a place to run Apache Beam based jobs, on GCP, and you do not need to address common aspects of running jobs on a cluster (e.g. Balancing work, or Scaling the number of workers for a job; by default, this is automatically managed for you, and applies to both batch and streaming) -- this can be very time consuming on other systems
    • Apache Beam is an important consideration; Beam jobs are intended to be portable across "runners," which include Cloud Dataflow, and enable you to focus on your logical computation, rather than how a "runner" works -- In comparison, when authoring a Spark job, your code is bound to the runner, Spark, and how that runner works
    • Cloud Dataflow also offers the ability to create jobs based on "templates," which can help simplify common tasks where the differences are parameter values

Related questions

FTP to Google Storage

Some files get uploaded on a daily basis to an FTP server and I need those files under Google Cloud Storage. I don't want to bug the users that upload the files to install any additional software and just let …

Read More
Google server putty connect 'Disconnected: No supported authentication methods available (server sent: publickey)

I'm trying to connect to my Debian Google Compute Engine server through PuTTy (I've tried other alternatives too) but when I do I get the error "Disconnected: No supported authentication methods available (server sent: publickey) The google server came without …

Read More
How to open a specific port such as 9090 in Google Compute Engine

I have 2 Google Compute Engine instances and I want to open port 9090 in both the instances. I think we need to add some firewall rules. Can you tell me how can I do that?

Read More