Object detection is not working in this scenario

General Discussion
, , ,
#1

Am using OpenCV in my iOS swift application. The frame from OpenCV is in cvmat format. It is converted into CVPixelBuffer using the below method

func pixelBufferFromImage(image: UIImage) -> CVPixelBuffer? {
    let ciimage = CIImage(image: image)
    //let cgimage = convertCIImageToCGImage(inputImage: ciimage!)
    let tmpcontext = CIContext(options: nil)
    let cgimage =  tmpcontext.createCGImage(ciimage!, from: ciimage!.extent)
    
    let cfnumPointer = UnsafeMutablePointer<UnsafeRawPointer>.allocate(capacity: 1)
    let cfnum = CFNumberCreate(kCFAllocatorDefault, .intType, cfnumPointer)
    let keys: [CFString] = [kCVPixelBufferCGImageCompatibilityKey, kCVPixelBufferCGBitmapContextCompatibilityKey, kCVPixelBufferBytesPerRowAlignmentKey]
    let values: [CFTypeRef] = [kCFBooleanTrue, kCFBooleanTrue, cfnum!]
    let keysPointer = UnsafeMutablePointer<UnsafeRawPointer?>.allocate(capacity: 1)
    let valuesPointer =  UnsafeMutablePointer<UnsafeRawPointer?>.allocate(capacity: 1)
    keysPointer.initialize(to: keys)
    valuesPointer.initialize(to: values)
    
    let options = CFDictionaryCreate(kCFAllocatorDefault, keysPointer, valuesPointer, keys.count, nil, nil)
    
    let width = cgimage!.width
    let height = cgimage!.height
    
    var pxbuffer: CVPixelBuffer?
    // if pxbuffer = nil, you will get status = -6661
    var status = CVPixelBufferCreate(kCFAllocatorDefault, width, height,
                                     kCVPixelFormatType_32BGRA, options, &pxbuffer)
    status = CVPixelBufferLockBaseAddress(pxbuffer!, CVPixelBufferLockFlags(rawValue: 0));
    
    let bufferAddress = CVPixelBufferGetBaseAddress(pxbuffer!);
    
    
    let rgbColorSpace = CGColorSpaceCreateDeviceRGB();
    let bytesperrow = CVPixelBufferGetBytesPerRow(pxbuffer!)
    let context = CGContext(data: bufferAddress,
                            width: width,
                            height: height,
                            bitsPerComponent: 8,
                            bytesPerRow: bytesperrow,
                            space: rgbColorSpace,
                            bitmapInfo: CGImageAlphaInfo.premultipliedFirst.rawValue | CGBitmapInfo.byteOrder32Little.rawValue);
    context?.concatenate(CGAffineTransform(rotationAngle: 0))
    context?.concatenate(__CGAffineTransformMake( 1, 0, 0, -1, 0, CGFloat(height) )) //Flip Vertical
    //        context?.concatenate(__CGAffineTransformMake( -1.0, 0.0, 0.0, 1.0, CGFloat(width), 0.0)) //Flip Horizontal
    
    print("Status ...........",status)
    context?.draw(cgimage!, in: CGRect(x:0, y:0, width:CGFloat(width), height:CGFloat(height)));
    status = CVPixelBufferUnlockBaseAddress(pxbuffer!, CVPixelBufferLockFlags(rawValue: 0));
    print("Status1111 ...........",status)
    return pxbuffer;
}

The inference whatever am getting is not correct. The labels are received with empty names. Is there any wrong with the above method?

thanks

#3

@Sudha_Sandhiya Welcome to Tensorflow Forum !

I’m unable to directly analyze the code without viewing it, but I can provide guidance based on common issues and best practices, let us know if any of the below works for you:

Please provide the specific code snippet for the CVMat to CVPixelBuffer conversion. This will allow me to identify potential errors or inconsistencies.Ensure the code adheres to best practices for accurate data transfer between OpenCV and Core Video frameworks.

Use debugging tools to visualize the pixel buffer’s contents and verify that it holds the expected image data. Confirm that the pixel buffer’s format (e.g., BGRA, RGB) matches the model’s input requirements.

Verify that the model is compatible with iOS and Swift environments. Ensure correct preprocessing (e.g., normalization, resizing) is applied to the pixel buffer before feeding it to the model. Double-check that the model’s output labels are correctly mapped to their corresponding names in your code.

You can try below for troubleshooting:-

Test model inference using a pre-existing CVPixelBuffer to rule out problems with the model or preprocessing. Explore alternative approaches to converting CVMat to CVPixelBuffer, such as using CVPixelBufferCreate directly. Ensure proper memory management and avoid potential memory leaks during conversion and inference. Address any potential compatibility issues between OpenCV and Core Video frameworks in your specific iOS version.

I’m ready to provide more specific guidance once you share the relevant code and additional details about your setup.

Let us know!