Create layer mask with custom-shaped hole

Question

I've spent too much time trying to figure this out and simply cannot find a workable solution.

Situation: 1. A picture of 'something' is displayed on the phone. 2. A semi-transparent (e.g. blue) layer is placed on top of the image, completely covering it. 3. A 'hole' in this layer exists where that part of the layer is fully transparent and is movable.

An example could be a zoom effect where you move this 'hole' around the image. Inside the hole you can see the image normally, while outside it's covered by the semi-transparent layer. NOTE: I'm implementing this in a cocos2d layer, where the image is represented by a CCSprite. It shouldn't matter though, if no cocos is used.

Problem: I've tried using CAShapeLayer and bitmaps as masks, but nothing is working (see code snippets below). With the CAShapeLayer, I create a UIBezierPath for the 'hole' and apply it to the colored layer. However, only the hole shows the color, while the rest is transparent. With an image, the mask is simply not working (I have no idea why). I've even tried masking masks to see if that would work. I've also tried swapping colors around...from white to black to clear for fill and background.

A simple solution, if it existed, would be to invert the area of the UIBezierPath. I've tried clipping, as well, using the path...but no luck.

I'm hoping that it's something simple-stupid that I'm simply overlooking. Perhaps one of you will see this. The moving part I'm not, yet, concerned with. I need to get the actual mask working first. The sample code is ignoring the y-axis differences between iPhone SDK and openGL.

CAShapeLayer Example:

CGSize winSize = [[CCDirector sharedDirector] winSize];
UIImage* img = [UIImage imageNamed:@"zebra.png"];
CCSprite* spr = [CCSprite spriteWithCGImage:img.CGImage key:@"img"];
spr.position = ccp( winSize.width / 2, winSize.width / 2 );
[self addSprite:spr];

UIBezierPath* path = [UIBezierPath bezierPathWithRect:rectHole];
CAShapeLayer* maskLayer = [CAShapeLayer layer];
maskLayer.bounds = [spr boundingBox];
maskLayer.position = spr.position;
maskLayer.fillColor = [UIColor whiteColor].CGColor;
maskLayer.backgroundColor = [UIColor clearColor].CGColor;
maskLayer.path = path.CGPath;

CALayer* colorLayer = [CALayer layer];
colorLayer.bounds = [spr boundingBox];
colorLayer.position = maskLayer.position;
[colorLayer setMask:maskLayer];

[[[[CCDirector sharedDirector] openGLView] layer] addSublayer:colorLayer];

Multiple Layer Mask Example:

CGSize winSize = [[CCDirector sharedDirector] winSize];
UIImage* img = [UIImage imageNamed:@"zebra.png"];
CCSprite* spr = [CCSprite spriteWithCGImage:img.CGImage key:@"img"];
spr.position = ccp( winSize.width / 2, winSize.width / 2 );
[self addSprite:spr];

UIBezierPath* path = [UIBezierPath bezierPathWithRect:rectHole];
CAShapeLayer* maskLayer = [CAShapeLayer layer];
maskLayer.bounds = [spr boundingBox];
maskLayer.position = spr.position;
maskLayer.fillColor = [UIColor whiteColor].CGColor;
maskLayer.backgroundColor = [UIColor clearColor].CGColor;
maskLayer.path = path.CGPath;

UIBezierPath* pathOuter = [UIBezierPath bezierPathWithRect:img.frame];
CAShapeLayer* outerLayer = [CAShapeLayer layer];
outerLayer.bounds = [spr boundingBox];
outerLayer.position = spr.position;
outerLayer.fillColor = [UIColor blackColor].CGColor;
outerLayer.backgroundColor = [UIColor whiteColor].CGColor;
outerLayer = pathOuter.CGPath;
[outerLayer setMask:maskLayer];

CALayer* colorLayer = [CALayer layer];
colorLayer.bounds = [spr boundingBox];
colorLayer.position = outerLayer.position;
[colorLayer setMask:outerLayer];

[[[[CCDirector sharedDirector] openGLView] layer] addSublayer:colorLayer];

Image Mask Example:

CGSize winSize = [[CCDirector sharedDirector] winSize];
UIImage* img = [UIImage imageNamed:@"zebra.png"];
CCSprite* spr = [CCSprite spriteWithCGImage:img.CGImage key:@"img"];
spr.position = ccp( winSize.width / 2, winSize.width / 2 );
[self addSprite:spr];

CGRect r = [spr boundingBox];
CGSize sz = CGSizeMake( r.size.width, r.size.height );
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceGray();
CGContextRef context = CGBitmapContextCreate( NULL, w, h, 8, 0, colorSpace, kCGBitmapByteOrderDefault | kCGImageAlphaNone );
CGColorSpaceRelease( colorSpace );
CGContextSetFillColorWithColor( context, [UIColor whiteColor].CGColor );
CGContextFillRect( context, r );
CGContextSetFillColorWithColor( context, [UIColor blackColor].CGColor );
CGContextFillRect( context, rectHole );
CGImageRef ref = CGBitmapContextCreateImage( context );
CGContextRelease( context );

CALayer* maskLayer = [CALayer layer];
maskLayer.bounds = [spr boundingBox];
maskLayer.position = spr.position;
[maskLayer setContents:(id)ref];

CALayer* colorLayer = [CALayer layer];
colorLayer.bounds = [spr boundingBox];
colorLayer.position = maskLayer.position;
[colorLayer setMask:maskLayer];

[[[[CCDirector sharedDirector] openGLView] layer] addSublayer:colorLayer];
CGImageRelease( ref );

Answer 1

I came back to this later after learning other core graphics techniques. The solution is closest to the Multiple Layer Mask Example above. However, instead of creating an inner and outer layer, you need to combine two paths into a single UIBezierPath in opposite directions.

So, e.g., create a path of the inner area to be cropped (CW). NOTE: x,y,w,h are referring to the origin and size of the "hole".

      [path moveToPoint:ccp(x,y)];
      [path addLineToPoint:ccp(x+w,y)];
      [path addLineToPoint:ccp(x+w,y+h)];
      [path addLineToPoint:ccp(x,y+h)];
      [path addLineToPoint:ccp(x,y)];

Then, add to the same path the outer area in the opposite direction (CCW). NOTE: x,y,w,h are referring to the origin and size of the outer rect.

      [path moveToPoint:ccp(x,y)];
      [path addLineToPoint:ccp(x,y+h)];
      [path addLineToPoint:ccp(x+w,y+h)];
      [path addLineToPoint:ccp(x+w,y)];
      [path addLineToPoint:ccp(x,y)];

This path is then applied to a layer (maskLayer), which is used as the mask on the final layer (colorLayer). The "outerLayer" is not needed.