How to generate an UIImage from AVCapturePhoto with correct orientation?

Question

I am calling AVFoundation's delegate method to handle a photo capture, but I am having difficulty converting the AVCapturePhoto it generates into an UIImage with the correct orientation. Although the routine below is successful, I always get a right-oriented UIImage (UIImage.imageOrientation = 3). I have no way of providing an orientation when using the UIImage(data: image) and attempting to first use photo.cgImageRepresentation()?.takeRetainedValue() also doesn't help. Please assist.

Image orientation is critical here as the resulting image is being fed to a Vision Framework workflow.

func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) {
    // capture image finished
    print("Image captured.")
    if let imageData = photo.fileDataRepresentation() {
        if let uiImage = UIImage(data: imageData){
            // do stuff to UIImage
        }
    }
}

UPDATE 1: Reading Apple's Photo Capture Programming Guide (out of date for iOS11), I did manage to find one thing I was doing wrong:

1. On every capture call (self.capturePhotoOutput.capturePhoto) one must setup a connection with the PhotoOutput object and update its orientation to match the device's orientation at the moment the picture is taken. For doing that, I created an extension of UIDeviceOrientation and used it on the snapPhoto() function I created to call the capture routine and wait for the didFinishProcessingPhoto delegate method to be executed. I've added a snapshot of the code because the code sample delimiters here don't seem to be displaying them correctly.

Update 2 Link to full project on GitHub: https://github.com/agu3rra/Out-Loud

Answer 1

Final update: I ran some experiments with the app and came to the following conclusions:

1. kCGImagePropertyOrientation doesn’t seem to influence the orientation of the captured image inside your application and it only varies with the device orientation if you update your photoOutput connection each time you are about to call the capturePhoto method. So:

   func snapPhoto() {
       // prepare and initiate image capture routine
   
       // if I leave the next 4 lines commented, the intented orientation of the image on display will be 6 (right top) - kCGImagePropertyOrientation
       let deviceOrientation = UIDevice.current.orientation // retrieve current orientation from the device
       guard let photoOutputConnection = capturePhotoOutput.connection(with: AVMediaType.video) else {fatalError("Unable to establish input>output connection")}// setup a connection that manages input > output
       guard let videoOrientation = deviceOrientation.getAVCaptureVideoOrientationFromDevice() else {return}
       photoOutputConnection.videoOrientation = videoOrientation // update photo's output connection to match device's orientation
   
       let photoSettings = AVCapturePhotoSettings()
       photoSettings.isAutoStillImageStabilizationEnabled = true
       photoSettings.isHighResolutionPhotoEnabled = true
       photoSettings.flashMode = .auto
       self.capturePhotoOutput.capturePhoto(with: photoSettings, delegate: self) // trigger image capture. It appears to work only if the capture session is running.
   }
   

2. Viewing the generated images on the debugger has shown me how they get generated, so I could infer the required rotation (UIImageOrientation) to get it displayed upright. In other words: updating UIImageOrientation tells how the image should be rotated in order for you to see it in the correct orientation. So I came to the following table:

3. I had to update my UIDeviceOrientation extension to a rather unintuitive form:

   extension UIDeviceOrientation {
       func getUIImageOrientationFromDevice() -> UIImageOrientation {
           // return CGImagePropertyOrientation based on Device Orientation
           // This extented function has been determined based on experimentation with how an UIImage gets displayed.
           switch self {
           case UIDeviceOrientation.portrait, .faceUp: return UIImageOrientation.right
           case UIDeviceOrientation.portraitUpsideDown, .faceDown: return UIImageOrientation.left
           case UIDeviceOrientation.landscapeLeft: return UIImageOrientation.up // this is the base orientation
           case UIDeviceOrientation.landscapeRight: return UIImageOrientation.down
           case UIDeviceOrientation.unknown: return UIImageOrientation.up
           }
       }
   }
   

4. This is how my final delegate method looks now. It displays the image in the expected orientation.

   func photoOutput(_ output: AVCapturePhotoOutput,
                                    didFinishProcessingPhoto photo: AVCapturePhoto,
                                    error: Error?)
   {
       // capture image finished
       print("Image captured.")
   
       let photoMetadata = photo.metadata
       // Returns corresponting NSCFNumber. It seems to specify the origin of the image
       //                print("Metadata orientation: ",photoMetadata["Orientation"])
   
       // Returns corresponting NSCFNumber. It seems to specify the origin of the image
       print("Metadata orientation with key: ",photoMetadata[String(kCGImagePropertyOrientation)] as Any)
   
       guard let imageData = photo.fileDataRepresentation() else {
           print("Error while generating image from photo capture data.");
           self.lastPhoto = nil; self.controller.goToProcessing();
           return
   
       }
   
       guard let uiImage = UIImage(data: imageData) else {
           print("Unable to generate UIImage from image data.");
           self.lastPhoto = nil; self.controller.goToProcessing();
           return
   
       }
   
       // generate a corresponding CGImage
       guard let cgImage = uiImage.cgImage else {
           print("Error generating CGImage");self.lastPhoto=nil;return
   
       }
   
       guard let deviceOrientationOnCapture = self.deviceOrientationOnCapture else {
           print("Error retrieving orientation on capture");self.lastPhoto=nil;
           return
   
       }
   
       self.lastPhoto = UIImage(cgImage: cgImage, scale: 1.0, orientation: deviceOrientationOnCapture.getUIImageOrientationFromDevice())
   
       print(self.lastPhoto)
       print("UIImage generated. Orientation:(self.lastPhoto.imageOrientation.rawValue)")
       self.controller.goToProcessing()
   }
   
   
   func photoOutput(_ output: AVCapturePhotoOutput, 
                      willBeginCaptureFor resolvedSettings: AVCaptureResolvedPhotoSettings) 
                      {
       print("Just about to take a photo.")
       // get device orientation on capture
       self.deviceOrientationOnCapture = UIDevice.current.orientation
       print("Device orientation: \(self.deviceOrientationOnCapture.rawValue)")
   }