Hi all, I posted this in the Tensorflow.js Discord but am reposting here for additional visibility.
I’ve been trying to port a number of recent research papers’ repositories into Tensorflow.js with limited success.
It seems like quite a lot of the recent research code implementations I’ve seen are written in Pytorch (this article ostensibly provides some hard numbers, though I’m not clear where the data comes from PyTorch vs TensorFlow in 2023).
I’m curious whether the Tensorflow team has any plans to better support ONNX models, either through conversion to Tensorflow or natively? Or if the team has a recommendation for ingesting the latest research into Tensorflow short of reimplementing the Pytorch models natively in Tensorflow?
I think there’s a number of recent models whose inference speeds are fast enough to run on the web and Tensorflow.js would be a great vehicle to share them.
I would love to hear from others too who are having issues here. Specifically:
What types of models you are trying to convert
The main blocker you are enounting
The use case / impact it would have if we found a way to make Pytorch to TensorFlow / JS more seamless.
If others see this thread who are trying similar (either PyTorch to TF Python or PyTorch to TF JS) then do let us know! Our command line converter from TF to TFJS is actively maintained so my guess the main bottleneck is the ONNX stage?
ValueError: '/conv_in/Conv_pad/' is not a valid root scope name. A root scope name has to match the following pattern: ^[A-Za-z0-9.][A-Za-z0-9_.\\/>-]*$
And none of the other name command line options working either.
my guess the main bottleneck is the ONNX stage?
While the second seems like an ONNX → TF issue, the first issue seems like a legitimate issue in the TFJS converter (since the Python TF version works). Still, to your point I agree with you that the ONNX → TF conversion step is the one I’m most worried about, since there seems to be no actively maintained library I’m aware of for doing the conversion.
Thanks for this extra context. For the TFJS converter issues you found do you know why it failed? Missing OP or such? Or any extra error message you can share for the TFJS Converter? I can then pass that back to our team to check if it is something we are aware of or not. If it is a missing op then that is usually the most common issue, and can only be solved with gaining more op parity but if there are ops that keep coming up when and if we do have capacity to look into that we can then at least prioritize based on the valuable feedback in this thread to see what could help most folk.
I don’t know if this will be useful to you guys, but I am building my own tool to generate saved_model, h5 and tflite from onnx.
saved_model is a specification that does not allow OPs to have leading slashes in their names. Thus, my conversion tool automatically replaces leading slashes with harmless strings. I believe this is a problem caused by the hdf5 specification.
@PINTO0309 Thank you for sharing this work with us! I was hoping to find gems like this from folk reading so this is good to see what others have been up to do solve so far too as I am sure many folk are effected by this more generally - as you say you also target TFLite too as an output which is good to know that this is useful for folk converting to other TF ecosystems too.
I’ve had these same issues. The hardest issue is when there’s a custom operation that needs to be implemented, but just getting all the Python dependencies correct, and getting the flow to work has been super challenging. Even converting from the Python version of TF to TFJS can be tough. I managed to get super slo-mo and deinterlace converted and using the pretrained models. However, like Kevin, despite many hours of attempts, I have yet to get a PyTorch model converted. My next target is the Maxim models, which are being refactored to work with arbitrary resolutions.
Currently I am just using onnx runtime, which works in node js and the browser, but it is far more convenient to have it in tfjs, as it works pretty well for the pre/post-processing steps, which onnxruntimes can not do.