How to detect significant change / trend in a time series data?

Question

So I have an array of say 25 samples and I would want to be able to note the trends of whether it's decreasing n or increasing from those 25 sample time interval(basically 25 samples array is my buffer that is being filled by every say 1 ms).

Note that it is general trend that I am looking for, not the individual derivative(as I would have obtained using finite difference or other numerical differentiation techniques).

Basically I expect my data to be noisy so there might be ups and downs even after doing filtering and so on. But it's the general trend of increasing or decreasing behaviour that I am looking for.

I want to integrate the increasing/decreasing behaviour in every ms to trigger some event which is more of a user interface event (blinking a LED) so it does not have to very delay of processing as long as I can detect the general trend.

Thanks in advance!

Answer 1

As has been pointed out already, you're not looking for the derivative. You're really looking for a "significant change" detection algorithm for a time series.

You'll certainly want a smoothing filter (and the moving average filter is fine -- see Bjorn's answer for this part).

But in addition to the smoothing filter, you will also need a decision criteria or threshold selector beyond which you will decide whether or not the filtered changes are significant.

If the underlying statistics of your time series is stable (stationary time series), then you can use a fixed statistical threshold, in the sense of standard deviations from the mean. For example, you might choose 2 standard deviations if you want a fairly strong "alarm" threshold (think alarming only on the strongest 5% of returns).

If there is nothing in the underlying problem that suggests that your time series is stable, i.e. if series could have a trend in it, or the underlying process generating the time series can go through fundmantal changes while you're monitoring it, then you'll need to use a dynamic, or adaptive threshold, in the sense of signal-to-noise (mu/sigma). You might then choose to detect all "meaningful" elements that pass the signal to noise test.