I am trying to write JavaPairRDD into file in local system. Code below:
JavaPairDStream<String, Integer> wordCounts = words.mapToPair(
new PairFunction<String, String, Integer>() {
@Override
public Tuple2<String, Integer> call(String s) {
return new Tuple2<String, Integer>(s, 1);
}
}).reduceByKey(new Function2<Integer, Integer, Integer>() {
@Override
public Integer call(Integer i1, Integer i2) {
return i1 + i2;
}
});
wordCounts.dstream().saveAsTextFiles("/home/laxmikant/Desktop/teppppp", "txt");
I am trying to save the logs or the wordcount in file. But it is not able to save in a local file (NOT HDFS).
I also tried to save on HDFS using
saveAsHadoopFiles("hdfs://10.42.0.1:54310/stream","txt")
The above line does not write to file. Can anybody tell the solution? Various solutions on stackoverflow dont work.
Try to write output as an absolute path:
saveAsTextFiles("file:///home/laxmikant/Desktop/teppppp", "txt");