Hey all! I’m Samperson, you can see some stuff I do on Twitter.
I’m having a problem setting up simple BlazePose detection to mess around!
No errors, it appears to function, it just gives me an empty list for poses every call - even when I can tab over to the live demo using the same camera, on the same model, and it works great!
I can’t identify where I’m going wrong, no matter how much example code + documentation I pull from. Here’s the source- could I get another pair of eyes on it? My entire implementation is <50loc, barebones as hell. I’ve uploaded it all here!.
Thank you so much!
-Sam
- a couple uneducated theories -
Maybe my setup can’t read anything off the video element?
I’m also getting sourcemap errors in the console, I’m wondering if I messed up my HTML script tags?
If I remember correctly BlazePose does not fair well on iOS devices in case you were running on one of those you will need to use the TensorFlow.js version to get support there as shown below.
Looking at your code I see in your HTML you import:
Video not ready is the message it repeats until the video stream is playing, after it asks for webcam permissions - which you’ll never see if it runs within an iFrame, because, security. My bad!
You can run the demo directly right here: https://BarestBonesBlazePose.samnchiet.repl.co
Where the code is still at that old URL.
The first call to estimatePoses takes a WHILE to get going. If you leave it running for 10-30 seconds, it will start printing 0 poses detected.
I have ensured my code matches that documentation exactly now, and it’s still producing the same results.
I gotta just be missing something important. There are some interesting warnings about a bunch of keywords already being defined? Maybe there are still redundancies in the script tags?
I also see a 404 warning about a sourcemap but I don’t think that’s necessarily a problem with anything but debugging.
So I replicated the problem without using your code to ensure it was not something I was missing in your specific implementation, but it seems I also get zero results. Please await (pun intended) while I asynchronously check with our team if there are any known issues here.
Update: So I replicated your code and made a more minimal version of it which you can view here:
The reason it was predicting zero poses is because the video width and height had not been set which BlazePose was using to determine width and heght. Adding these 2 lines once you know the final size of video (after load) will fix issue:
As you can see with my code above this now works as intended.
Also please make sure you use requestAnimationFrame like I have not setTimeout as you will run into issues on older devices that are slower that can not go as fast as you think. By using requestAnimationFrame it lets the browser decide when it has enough resources to process next frame instead which is industry best practice.
Happy hacking and I look forward to seeing your final TensorFlow.js creation. Do tag us on social using the #MadeWithTFJS hashtag so we can find what you create and share with the wider community!
THIS WORKED! Y’all are heroes.
Do the width and height of the video element in the DOM matter? I assume those values have nothing to do with the size, given that the element looked fine before, but it’s worth checking.
e.g. if I stretch the element to the full page, is that going to make TensorFlow act funny?
Video HTML attribute width is the same as video.width here so you do need to set it. You can try setting video.width in JS and see how it changes the size eg:
video.width = 500;
When browser loads the video it will change the element rendered size to be the size of the video automatically, however it doesnt explicitly set the video width / height as an attribute which is what many devs use to find the video width/height. It would appear the MediaPipe implementation samples the passed element’s width/height to decide how to process.
So if your HTML looks like this:
<video id="blah" autoplay> </video>
Then that will cause issues. When you use video.width it is referencing the element property width set in the HTML as attribute but as you can see no width attribute is set. Even if you styled using CSS, then video.width would be undefined (or maybe 0 in some browsers) as no attribute property was explicitly specified which was the case here which is why MediaPipe bailed and returned zero results as it thought it was 0,0 or undefined,undefined which it can not do anything with.
In short, always set your video.width and video.height once you know what it is going to be from getUserMedia’s returned promise (and if it changes for some reason later eg load in new video of different resolution) , just in case other developers are referencing it when drawing to canvas etc as was happening here.
I believe our team is looking into updating to automatically sample videoWidth for video too, but not sure when that change will go through so for now just ensure you set it
By using balzepose we get two sets of x,y,x coordinates, one in pixels and other in 3D world coordinates. I could plot the keypoints3D in a canvas using ThreeJs. Shape is relatively accurate, but centered to the screen. How can I make use of keyPoints (pixel values) to move the plotted shapes. My idea is to make use of blazepose to render and move 3d objects.
