iOS Heart rate detection Algorithm
I'm trying to implement heart beat recording functionality in an app i'm developing.
The preferred method of doing this is by using the iPhone's camera with the light on, having the user place their finger on the lens, and detecting fluctuations in the video feed, which correspond to the user's heart.
I found a very good starting point with the following stack overflow question here
The question provides useful code to plot a heart beat time graph.
It shows how to start an AVCaptureSession and turn the camera's light on like so:
session = [[AVCaptureSession alloc] init];
AVCaptureDevice* camera = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
if([camera isTorchModeSupported:AVCaptureTorchModeOn]) {
[camera lockForConfiguration:nil];
camera.torchMode=AVCaptureTorchModeOn;
// camera.exposureMode=AVCaptureExposureModeLocked;
[camera unlockForConfiguration];
}
// Create a AVCaptureInput with the camera device
NSError *error=nil;
AVCaptureInput* cameraInput = [[AVCaptureDeviceInput alloc] initWithDevice:camera error:&error];
if (cameraInput == nil) {
NSLog(@"Error to create camera capture:%@",error);
}
// Set the output
AVCaptureVideoDataOutput* videoOutput = [[AVCaptureVideoDataOutput alloc] init];
// create a queue to run the capture on
dispatch_queue_t captureQueue=dispatch_queue_create("catpureQueue", NULL);
// setup our delegate
[videoOutput setSampleBufferDelegate:self queue:captureQueue];
// configure the pixel format
videoOutput.videoSettings = [NSDictionary dictionaryWithObjectsAndKeys:[NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA], (id)kCVPixelBufferPixelFormatTypeKey,
nil];
videoOutput.minFrameDuration=CMTimeMake(1, 10);
// and the size of the frames we want
[session setSessionPreset:AVCaptureSessionPresetLow];
// Add the input and output
[session addInput:cameraInput];
[session addOutput:videoOutput];
// Start the session
[session startRunning];
Self in this example must be an <AVCaptureVideoDataOutputSampleBufferDelegate>
And will therefore have to implement the following method to obtain raw camera data:
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {
static int count=0;
count++;
// only run if we're not already processing an image
// this is the image buffer
CVImageBufferRef cvimgRef = CMSampleBufferGetImageBuffer(sampleBuffer);
// Lock the image buffer
CVPixelBufferLockBaseAddress(cvimgRef,0);
// access the data
int width=CVPixelBufferGetWidth(cvimgRef);
int height=CVPixelBufferGetHeight(cvimgRef);
// get the raw image bytes
uint8_t *buf=(uint8_t *) CVPixelBufferGetBaseAddress(cvimgRef);
size_t bprow=CVPixelBufferGetBytesPerRow(cvimgRef);
float r=0,g=0,b=0;
for(int y=0; y<height; y++) {
for(int x=0; x<width*4; x+=4) {
b+=buf[x];
g+=buf[x+1];
r+=buf[x+2];
// a+=buf[x+3];
}
buf+=bprow;
}
r/=255*(float) (width*height);
g/=255*(float) (width*height);
b/=255*(float) (width*height);
float h,s,v;
RGBtoHSV(r, g, b, &h, &s, &v);
// simple highpass and lowpass filter
static float lastH=0;
float highPassValue=h-lastH;
lastH=h;
float lastHighPassValue=0;
float lowPassValue=(lastHighPassValue+highPassValue)/2;
lastHighPassValue=highPassValue;
//low pass value can now be used for basic heart beat detection
}
RGB is converted to HSV and it is Hue that is monitored for fluctuations.
And RGB to HSV is implemented as follows
void RGBtoHSV( float r, float g, float b, float *h, float *s, float *v ) {
float min, max, delta;
min = MIN( r, MIN(g, b ));
max = MAX( r, MAX(g, b ));
*v = max;
delta = max - min;
if( max != 0 )
*s = delta / max;
else {
// r = g = b = 0
*s = 0;
*h = -1;
return;
}
if( r == max )
*h = ( g - b ) / delta;
else if( g == max )
*h=2+(b-r)/delta;
else
*h=4+(r-g)/delta;
*h *= 60;
if( *h < 0 )
*h += 360;
}
The low pass value calculated in capureOutput: initially provides erratic data, but then stabilises to the following:
2013-11-04 16:18:13.619 SampleHeartRateApp[1743:1803] -0.071218
2013-11-04 16:18:13.719 SampleHeartRateApp[1743:1803] -0.050072
2013-11-04 16:18:13.819 SampleHeartRateApp[1743:1803] -0.011375
2013-11-04 16:18:13.918 SampleHeartRateApp[1743:1803] 0.018456
2013-11-04 16:18:14.019 SampleHeartRateApp[1743:1803] 0.059024
2013-11-04 16:18:14.118 SampleHeartRateApp[1743:1803] 0.052198
2013-11-04 16:18:14.219 SampleHeartRateApp[1743:1803] 0.078189
2013-11-04 16:18:14.318 SampleHeartRateApp[1743:1803] 0.046035
2013-11-04 16:18:14.419 SampleHeartRateApp[1743:1803] -0.113153
2013-11-04 16:18:14.519 SampleHeartRateApp[1743:1803] -0.079792
2013-11-04 16:18:14.618 SampleHeartRateApp[1743:1803] -0.027654
2013-11-04 16:18:14.719 SampleHeartRateApp[1743:1803] -0.017288
An example of the erratic data provided initially is here:
2013-11-04 16:17:28.747 SampleHeartRateApp[1743:3707] 17.271435
2013-11-04 16:17:28.822 SampleHeartRateApp[1743:1803] -0.049067
2013-11-04 16:17:28.922 SampleHeartRateApp[1743:1803] -6.524201
2013-11-04 16:17:29.022 SampleHeartRateApp[1743:1803] -0.766260
2013-11-04 16:17:29.137 SampleHeartRateApp[1743:3707] 9.956407
2013-11-04 16:17:29.221 SampleHeartRateApp[1743:1803] 0.076244
2013-11-04 16:17:29.321 SampleHeartRateApp[1743:1803] -1.049292
2013-11-04 16:17:29.422 SampleHeartRateApp[1743:1803] 0.088634
2013-11-04 16:17:29.522 SampleHeartRateApp[1743:1803] -1.035559
2013-11-04 16:17:29.621 SampleHeartRateApp[1743:1803] 0.019196
2013-11-04 16:17:29.719 SampleHeartRateApp[1743:1803] -1.027754
2013-11-04 16:17:29.821 SampleHeartRateApp[1743:1803] 0.045803
2013-11-04 16:17:29.922 SampleHeartRateApp[1743:1803] -0.857693
2013-11-04 16:17:30.021 SampleHeartRateApp[1743:1803] 0.061945
2013-11-04 16:17:30.143 SampleHeartRateApp[1743:1803] -0.701269
The low pass value goes positive whenever there is a heart beat. So I tried a very simple live detection algorithm which basically looks at the current value, and sees if it is positive, it also looks at the previous value, if negative it detects negative going to positive and plays a beep sound.
The problem with this is the data isn't always as perfect as the above, sometimes there's anomalous positive readings in amongst negative readings and vice versa.
A graph of the low pass value in time looks like this:
Interestingly the above anomaly is quite common, if I record a graph for a while i'll see a very similar shaped anomaly multiple times.
In my very simple beat detection algorithm, if an anomaly as shown above occurs the counted number of beats in the detection period (10 seconds) can shoot up by 4 or 5 beats. This makes the calculated BPM very inaccurate. But as simple as it is it does work around 70% of the time.
To combat this problem I tried the following.
1.Started recording last 3 low pass values in an array
2.Then looked to see whether or not the middle value had two smaller values surrounding it before and after. (Basic peak detection)
3.Counted this scenario as a beat and added it to the running total of beats in a given time.
This method is however just as vulnerable to the anomalies as any other. And actually seemed to be a worse method. (When playing live beeps after detection they seemed far more erratic than the positive to negative algorithm)
My question is can you help me come up with an algorithm that can reliably detect when a heart beat occurs with reasonable accuracy.
Another problem I realise that i'm going to have to address is detecting whether or not a user's finger is on the lens.
I thought about detecting erratic low pass values but the problem there is the low pass filter accounts for erratic values and smooths them out over time. So help there would be appreciated too.
Thanks for your time.
Answer
The answer to this question is a little bit involved, as you need to do several things to process the signal and there is no single "right" way to do this. However, for your filter you want to use a band-pass filter. This type of filter allows you to specify a range of frequencies that are accepted both at the high and low ends. For a human heart beat, we know what these bounds should be (no less than 40 bpm and no higher than 250 bpm) so we can create a filter that removes frequencies outside of this range. The filter also moves the data to be centered at zero, so peak detection becomes much easier. This filter will give you a much more smooth signal, even if your users increases/decreases their finger pressure (to a certain degree). After that, additional smoothing and outlier removal will also need to happen.
A specific type of band-pass filter that I have used is a butterworth filter. This is a little involved to create by hand since the filter changes based on the frequency you are collecting your data at. Fortunately, there is a website that can help with this here. If you are collecting your data at 30 fps, then the frequency will be 30 hz.
I have created a project that wraps all of this together and detects a user's heart rate well enough to include it in my app on the iOS app store. I have made the heart rate detection code available on github.