Could not parse Master URL: 'spark:http://localhost:18080'

Anas picture Anas · Dec 2, 2014 · Viewed 23.5k times · Source

When I'm trying to run my code it throws this Exception:

Exception in thread "main" org.apache.spark.SparkException: Could not parse Master URL:spark:http://localhost:18080

This is my code:

SparkConf conf = new SparkConf().setAppName("App_Name").setMaster("spark:http://localhost:18080").set("spark.ui.port","18080");
JavaStreamingContext ssc = new JavaStreamingContext(sc, new Duration(1000));
String[] filet=new String[]{"Obama","ISI"};

JavaReceiverInputDStream<Status> reciverStream=TwitterUtils.createStream(ssc,filet);
JavaDStream<String> statuses = reciverStream.map(new Function<Status, String>() {
     public String call(Status status) { return status.getText(); }
     }
      );
ssc.start();
ssc.awaitTermination();}}

Any idea how can I fix this problem?

Answer

icza picture icza · Dec 2, 2014

The problem is that you specify 2 schemas in the URL you pass to SparkConf.setMaster().

The spark is the schema, so you don't need to add http after spark. See the javadoc of SparkConf.setMaster() for more examples.

So the master URL you should be using is "spark://localhost:18080". Change this line:

SparkConf conf = new SparkConf().setAppName("App_Name")
    .setMaster("spark://localhost:18080").set("spark.ui.port","18080");