Apache Spark Moving Average

time-series hdfs moving-average apache-spark
Ahmed Shabib picture Ahmed Shabib · May 1, 2014 · Viewed 24.3k times · Source

I have a huge file in HDFS having Time Series data points (Yahoo Stock prices).

I want to find the moving average of the Time Series how do I go about writing the Apache Spark job to do that .

Answer

Arvind picture Arvind · Mar 4, 2015

You can use the sliding function from MLLIB which probably does the same thing as Daniel's answer. You will have to sort the data by time before using the sliding function.

import org.apache.spark.mllib.rdd.RDDFunctions._

sc.parallelize(1 to 100, 10)
  .sliding(3)
  .map(curSlice => (curSlice.sum / curSlice.size))
  .collect()

Related questions

Plotting two variables as lines using ggplot2 on the same graph

A very newbish question, but say I have data like this: test_data <- data.frame( var0 = 100 + c(0, cumsum(runif(49, -20, 20))), var1 = 150 + c(0, cumsum(runif(49, -10, 10))), date = seq(as.Date("2002-01-01"), by="1 month", length.out=100) ) How can I …

Read More
How to calculate rolling / moving average using NumPy / SciPy?

There seems to be no function that simply calculates the moving average on numpy/scipy, leading to convoluted solutions. My question is two-fold: What's the easiest way to (correctly) implement a moving average with numpy? Since this seems non-trivial and …

Read More
Peak signal detection in realtime timeseries data

Update: The best performing algorithm so far is this one. This question explores robust algorithms for detecting sudden peaks in real-time timeseries data. Consider the following dataset: Example of this data in Matlab format (but this question is not about …

Read More