How to configure Apache Spark random worker ports for tight firewalls?

configuration apache-spark worker ports
Isma Khan picture Isma Khan · Jan 1, 2015 · Viewed 14.6k times · Source

I am using Apache Spark to run machine learning algorithms and other big data tasks. Previously, I was using spark cluster standalone mode running spark master and worker on the same machine. Now, I added multiple worker machines and due to a tight firewall, I have to edit the random port of worker. Can anyone help how to change random spark ports and tell me exactly what configuration file needs to be edited? I read the spark documentation and it says spark-defaults.conf should be configured but I don't know how I can configure this file for particularly changing random ports of spark.

Answer

Samson Scharfrichter picture Samson Scharfrichter · Aug 19, 2017

Update for Spark 2.x

Some libraries have been rewritten from scratch and many legacy *.port properties are now obsolete (cf. SPARK-10997 / SPARK-20605 / SPARK-12588 / SPARK-17678 / etc)

For Spark 2.1, for instance, the port ranges on which the driver will listen for executor traffic are

  • between spark.driver.port and spark.driver.port+spark.port.maxRetries
  • between spark.driver.blockManager.port and spark.driver.blockManager.port+spark.port.maxRetries

And the port range on which the executors will listen for driver traffic and/or other executors traffic is

  • between spark.blockManager.port and spark.blockManager.port+spark.port.maxRetries

The "maxRetries" property allows for running several Spark jobs in parallel; if the base port is already used, then the new job will try the next one, etc, unless the whole range is already used.

Source:
   https://spark.apache.org/docs/2.1.1/configuration.html#networking
   https://spark.apache.org/docs/2.1.1/security.html under "Configuring ports"

Related questions

What is spark.driver.maxResultSize?

The ref says: Limit of total size of serialized results of all partitions for each Spark action (e.g. collect). Should be at least 1M, or 0 for unlimited. Jobs will be aborted if the total size is above this limit. …

Read More
Error message "Forbidden You don't have permission to access / on this server"

I have configured my Apache by myself and have tried to load phpMyAdmin on a virtual host, but I received: 403 Forbidden You don't have permission to access / on this server My httpd.conf # # This is the main Apache HTTP server …

Read More
Reading settings from app.config or web.config in .NET

I'm working on a C# class library that needs to be able to read settings from the web.config or app.config file (depending on whether the DLL is referenced from an ASP.NET web application or a Windows Forms …

Read More