Playing a video inside each bounding box

Special Interest Groups TF.js
#1

Hello,

My problem set is to add a mp4 video playing inside a bounding box whenever detections are happening, I have tried this with images which works but with video, it’s not working inside a bounding box.

export function renderBoxes(canvasRef: HTMLCanvasElement, boxes_data: Float32Array | Int32Array | Uint8Array, scores_data: Float32Array | Int32Array | Uint8Array, classes_data: Float32Array | Int32Array | Uint8Array, ratios: any[]) {
  const ctx = canvasRef.getContext('2d')
  if (ctx) {
    ctx.clearRect(0, 0, ctx.canvas.width, ctx.canvas.height) // clean canvas

    const colors = new Colors()

    // font configs
    const font = `${Math.max(
      Math.round(Math.max(ctx.canvas.width, ctx.canvas.height) / 40),
      14,
    )}px Arial`
    ctx.font = font
    ctx.textBaseline = 'top'

    for (let i = 0; i < scores_data.length; ++i) {
      // filter based on class threshold
      const klass = labels[classes_data[i]]
      const color = colors.get(classes_data[i])
      const score = (scores_data[i] * 100).toFixed(1)

      let [y1, x1, y2, x2] = boxes_data.slice(i * 4, (i + 1) * 4)
      x1 *= ratios[0]
      x2 *= ratios[0]
      y1 *= ratios[1]
      y2 *= ratios[1]
      const width = x2 - x1
      const height = y2 - y1

      // draw box.
      ctx.fillStyle = Colors.hexToRgba(color, 0.2)!
      ctx.fillRect(x1, y1, width, height)
      console.log(x1,y1)

     //Playing video inside bounding box
      var videoElement = document.createElement('video');
      videoElement.src = 'video.mp4';
      videoElement.autoplay = true;
      videoElement.loop = true;
      videoElement.muted = true;
      videoElement.height = 400;
      videoElement.width = 400;
      videoElement.style.zIndex = "1";

      ctx.drawImage(videoElement, x1,y1,width,height)

      // draw border box.
      ctx.strokeStyle = color
      ctx.lineWidth = Math.max(Math.min(ctx.canvas.width, ctx.canvas.height) / 200, 2.5)
      ctx.strokeRect(x1, y1, width, height)

      // Draw the label background.
      ctx.fillStyle = color
      const textWidth = ctx.measureText(`${klass} - ${score}%`).width
      const textHeight = parseInt(font, 10) // base 10
      const yText = y1 - (textHeight + ctx.lineWidth)
      ctx.fillRect(
        x1 - 1,
        yText < 0 ? 0 : yText, // handle overflow label box
        textWidth + ctx.lineWidth,
        textHeight + ctx.lineWidth,
      )

      // Draw labels
      ctx.fillStyle = '#ffffff'
      ctx.fillText(`${klass} - ${score}%`, x1 - 1, yText < 0 ? 0 : yText)
    }
  }
}

I am not facing errors but warnings are there from the WebGL side

Screenshot 2023-06-07 125435
Screenshot 2023-06-07 125435937×342 20.1 KB

If anyone has experienced this error or has any suggestions on how to fix it, please let me know.

#3

It looks like the warning is out of your function ‘renderBox’, and I would guess it’s come from TFJS’s model inference, because the warning comes from gpgpu_context.ts. You might need to provide more codes to help me understand the error.

Also, I think this warning is complaining you are not using one context and this warning actually is an error which causes you get an empty or all-zero output.

#5

Hi all,

I’ve also noticed the HTML Object (videoElement) get’s added into a Canvas (not possible).

videoElement = document.createElement('video');
canvas.drawImage(html:videoElement);

Instead you can try to create/update a HTML Container as Layer above the canvas.

container = document.createElement('div');
// add or update x, y, w, h and z-index to the DIV HTML/CSS attributes
container.appendChild(html:videoElement);

Hope that helps