TensorFlow.js announces new model: MoveNet - super fast, and accurate pose estimation

General Discussion
,
#1

Yep, the holiday season came early for TensorFlow.js devs with this new model from a collaboration with Google Research that have created a super fast native TensorFlow.js model (like 120 FPS fast on my modern desktop at peak) for body pose estimation that runs in your web browser. What will you make?

I am sure you want to try it out and learn more so without further ado here is what you need to know:

  1. Read more and learn about MoveNet: Next-Generation Pose Detection with MoveNet and TensorFlow.js — The TensorFlow Blog

  2. Try the demo: https://storage.googleapis.com/tfjs-models/demos/pose-detection/index.html?model=movenet

  3. Try the first community made demo that uses movenet already in a game - just 2 days after launch - amazing! You will have a lot of fun with this one: https://twitter.com/devdevcharlie/status/1395102385387843584

11 Likes
#2

Does this model can be used in nodejs and how ?

1 Like
#3

You can use NPM to import via Node:

import * as poseDetection from '@tensorflow-models/pose-detection';

See the blog post above for more details. The rest of the code should be the same.

1 Like
#4

Yes I have read the post. Look at example and source , but I don’t understand how to give a stream to the method poseEstilamaton if I run my code on server side.

1 Like
#5

This tutorial may be of use to you which shows you how to use imagery with one of our other models in Node but the principle would be similar:

2 Likes
#6

I was quite interested to know or have some similar tutorials, in tensor flow react native ? I am going through some sample general react native tutorials . Do you have some suggestions on that ? It seems like not many tutorials are written on movenet.

#8

This guy is probably making some of the best React Native stuff on YouTube right now that also uses TensorFlow.js:

1 Like
#10

Hi Jason thanks for the info. I think most of the videos , made by him is running in react javacript but perhaps not react native expo . I saw some where , react native tensor flow implementation of some models (like object detection) but nothing specifically, for pose estimation , as far as I have dug into.

#12

@Gant_Laborde May have some experience of React Native maybe? Have you seen anyone use PoseNet or MoveNet in react native Gant? Off the top of my head I can not think of any examples using this right now.

@DLC_Trial That being said if you are going to be the first to try, I would recommend trying our much newer “MoveNet” model which is far more accurate and faster as detailed on this thread if you do decide to try using within React Native.

Also you may find this tutorial useful in how someone ported an arbitrary TFJS model to run in React Native which is what you may need to do for this too: https://heartbeat.fritz.ai/image-classification-on-react-native-with-tensorflow-js-and-mobilenet-48a39185717c

2 Likes
#14

Thanks @Jason , I tried the link for image classification and could compile it . I was thinking an intermediate step could be to try p5js for movenet.

Yes, starting with movenet would be ideal as you suggested . But blazepose /media pipe definetely has more keypoints to track ( but i think the results are about 12fps) from what i read .

Thanks

#16

Here’s a repo that uses TFJS on RN Expo

Using the TFJS platform adapter for React Native works on raw RN and Expo apps as far as I know.

#17

Yep it depends what you want. More keypoints (33 vs 17) or a model that has less “jitter” when predicting etc - movenet was designed for fast movements and more extreme poses I believe and when I last tested it runs at 60 FPS on a current gen iPhone, and around 120 FPS on a desktop with 1070 Nvidia GPU (which is a few generations old now).

We have had companies in physiotherapy use MoveNet for real world production use cases for this reason. So choose what is best for your use case and try them both out! Both run well in browser so you can see what works best for your use case and then once you know you can decide which one you want to port to React Native. Good luck!

1 Like
#19

Hello @Jason
When will movenet support multi-person detection?

#21

Welcome to the community @Takuya. I am unsure of this at present, if I hear of any updates though I shall post something once I hear more details around this area.

#22

@Takuya So I can confirm that this multi-person detection is currently being investigated. Once there are demos and such to share I will be posting them to our usual social channels for all to see so keep an eye out for that when it is launched. If you have not seen any updates in a month or two around this, please feel free to reach out to me again to get a new ETA.

#23

Thanks @Jason
Sorry I asked update schedule here.
I look forward to the update at social channels.

#25

No problems @Takuya Feel free to post if you have anything on your mind related to TensorFlow.js! It is great to hear from you and excited to see what you create :slight_smile: