Cannot run model.fit locally (help)

I am running Google Colab locally on my Macbook Air running the M1 chip. I have installed the pre-released TensorFlow and am trying to run a ConvNet model, but get an error saying ‘ImportError: Image transformations require SciPy. Install SciPy’. I have read the readme on the GitHub for the pre-released TF and it says SciPy is currently not available. I tried installing SciPy on the virtual environment that TF was installed in but still no luck, does not import when running an interactive python session on my terminal and also when running the model again. Has anyone gotten around this? I cannot stand waiting 15+ minutes per epoch on 30 epochs when training my model when using Colabs hosted server.

1 Like

It Is a temp known limit but there are workarounds:

E.g. See Installing TensorFlow 2.4 and JupyterLab on Mac with M1 (outdated)

1 Like

Thank you. I managed to install SciPy into my virtual env using some commands that were written by a user in of the websites in the link you gave me, but I still ran into the same issue: ‘ImportError: Image transformations require SciPy. Install SciPy’ after doing that. Seems like I will probably just have to buy a new pc in order to train my model much faster than what I am getting now. Had to scale down my steps_per_epoch by 10, and the speed at which its training is still appalling!

1 Like

You can also try to read/interact with this ticket:

And subscribe/upvote this:

2 Likes

Followed the instructions on the first GitHub ticket, ended up working, model finally runs locally, averaging around 2s / step, as opposed to to the ~40s / step I was getting before when not running it locally. Thank you for your help, very much appreciated!

2 Likes

It looks like you’re hitting the error in this file:

From the code I linked, that doesn’t sound possible. The error is only triggered is import scipy; import scipy.ndimage fails.

Anyway, one good workaround for this is to not use keras.preprocessing since these run in python/numpy/scipy not in tensorflow. keras.layers.experimental.preprocessing contains pure-tensorflow layers that implemnent common image transformations.

https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/preprocessing/RandomRotation?version=nightly

We have several examples that use these.

2 Likes

Interesting. I was trying to do data augmentation and was using ImageDataGenerator, the keras.layers.experimental.preprocessing I think has everything I need apart from shear_range which is in ImageDataGenerator, how would I go about applying a shear_range in that ? I tried searching for it and was not able to find it.

1 Like

I think we have TF native shear in the model garden package

2 Likes

And it looks like, similar to the keras.preprocessing implementation, all those layers.experimental.preprocessing layers use this transform function which just takes a projection matrix:

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/keras/layers/preprocessing/image_preprocessing.py#L621

So I think you’d just need to work out the right range of matrices there. Maybe there’s something you can copy from the keras.preprocessing version.

Or maybe someone’s already done all that work.

2 Likes

More than one :grin: I hope we could unify the image processing API one of this days:

2 Likes

It is a “private” API. We have tried to expose this but the evaluation was to keep this private. See

1 Like