Open Source MoveNet?

Research & Models
,
#1

Hi MoveNet developers,

I’ve been following MoveNet for a while I love how you guys are creating really useful and performant model to be used in mobile applications. I have a use case where I would like to fine-tune/modify the model. (e.g. modify number of key point, provide more targeted training from custom dataset)

Is there any plan to open source/enable fine tuning on MoveNet models? I’m relatively new to ML field (coming from SW) and looking at TFHub suggests that it’s not fine-tunable. (I did try to import/inspect the model on TF and yeah it looked pretty blackbox) I searched around open source implementation and there’s one written in PyTorch,

I’ve no idea how accurate this implementation is, but I guess I have few options.

  1. Use PyTorch implementation as baseline
  2. Port this implementation/build in TF
  3. Ask original devs for generosity

Let me know what you guys have planned, and any suggestions!

Thanks,
Daniel

1 Like
#3

Hello Daniel,

Thanks a lot for your interest in using MoveNet! You are right the MoveNet releases are currently not fine-tunable. But the model implementation is actually based on the Tensorflow Object Detection API so you should be able to access the model code and modify it for your own purpose. As mentioned in the documentation, the MoveNet model is based on CenterNet with some modifications in the postprocessing logics and ops re-write to boost the performance. The closest model architecture can be found here. You can also find some example training configs in the /models/research/object_detection/configs/tf2/ folder.

I’d recommend you to start with the above codebase and modify from it. There are a few internal modifications which have not been released yet. We are still working on it and will let you know once it is there. Thanks.

2 Likes
#4

That was super helpful, thanks!

#6

Hi everyone,
I am running into the same challenge. @Daniel_Kim have you managed to fine-tune movenet? Could we maybe have a short chat about it? The object detection API features close to 5000 lines of code and it would be very helpful to speak to someone who has successfully fine-tuned it already.

My use case is that I want to train on pose detection. I have extracted the features from move-net and have built my own classifier on top to detect custom poses (fitness exercises) of my own dataset. But since I have objects on some frames in the foreground and on all frames in the background MoveNet is often misidentifying the keypoints and sometimes does not find any keypoint at all.

So I need to use some of the final layers to fine-tune on my custom data, which I have labelled (inkluding keypoints based on coco).

Any tips would be highly appreciated.

Best
Julian