Markerless tracking and pose estimation on the web browser with single camera

Special Interest Groups TF.js
,
#1

Hello everyone,

We’re in the process of developing a web application that displays an MJPEG stream. Our goal is to identify specific objects in the video in real-time and perform pose estimation. Ideally, we’d like it to function in a way similar to fiduciary markers, but instead, leverage a machine learning algorithm to enable markerless tracking.

Could anyone share insights on whether this is feasible using TensorFlow.js?

#3

Yes, markerless tracking and pose estimation with a single camera in a web browser using TensorFlow.js is feasible. Use models like PoseNet or MoveNet for pose estimation and Coco SSD for object detection. Implement by:

  1. Including TensorFlow.js in your project.
  2. Loading the appropriate model.
  3. Accessing the camera with getUserMedia.
  4. Processing video frames through the model.
  5. Rendering results on a canvas over the video stream.

These models are efficient and can run in real-time directly in the browser.

1 Like
#5

+1 to what @Niloy_Deb_Barma said. Just following up with a few links to consider:

Custom object detection blogs:

Learn how to make a smart camera in TensorFlow.js using COCO-SSD pretrained model:

Video version of codelab:

MoveNet pose estimation:

BlazePose GHUM 3D pose estimation:

Hope that helps!

Learn more about using TensorFlow.js via my course:

2 Likes