I am running Google Colab locally on my Macbook Air running the M1 chip. I have installed the pre-released TensorFlow and am trying to run a ConvNet model, but get an error saying ‘ImportError: Image transformations require SciPy. Install SciPy’. I have read the readme on the GitHub for the pre-released TF and it says SciPy is currently not available. I tried installing SciPy on the virtual environment that TF was installed in but still no luck, does not import when running an interactive python session on my terminal and also when running the model again. Has anyone gotten around this? I cannot stand waiting 15+ minutes per epoch on 30 epochs when training my model when using Colabs hosted server.
1 Like
Bhack
June 2, 2021, 12:10pm
#2
1 Like
Thank you. I managed to install SciPy into my virtual env using some commands that were written by a user in of the websites in the link you gave me, but I still ran into the same issue: ‘ImportError: Image transformations require SciPy. Install SciPy’ after doing that. Seems like I will probably just have to buy a new pc in order to train my model much faster than what I am getting now. Had to scale down my steps_per_epoch by 10, and the speed at which its training is still appalling!
1 Like
Bhack
June 2, 2021, 2:46pm
#4
You can also try to read/interact with this ticket:
opened 03:15AM - 03 Feb 21 UTC
This is not so much an issue as opposed to a 'How To' if you'd like to install t… his version of Tensorflow in Conda.
## Prerequisites: You must be on macOS Big Sur
If you have an Apple Silicon Mac, this is a freebie, you're already on Big Sur. If you're on an Intel Mac, the Intel versions of TensorFlow are Big Sur only.
**Sanity Check before Proceeding**: To ensure you're on the right version of macOS, run `sw_vers -productVersion` in your terminal. If it's not version 11.##, you're not on Big Sur and must upgrade to it from the macOS App Store.
## Prerequisites: Install XCode Command Line Tools
Install Xcode Command Line tools if you haven't. To do so, run this in your terminal: `xcode-select --install`
**Sanity Check before Proceeding**: To ensure installation worked, run `which xcrun` in your terminal and you should get a path like `/usr/bin/xcrun`. If you haven't, you did not install it correctly.
## Prerequisites: Install Miniforge
**Where to download Miniforge from**
Miniforge, is a 'lightweight' Python interpreter that has full access to the Conda ecosystem. You can download Miniforge from https://github.com/conda-forge/miniforge#miniforge3. You can use Anaconda if you're on Intel, but note that this guide will be written from the perspective of using miniforge.
**Sanity Check before Proceeding**:
- Run `file $(which python)` in your terminal (thanks to @lebigot for this shortcut!). Please make sure that you got:
- This path implies you're running your `miniforge` version of Python. It'll probably be `<your home dir>/miniforge3/bin/python`.
- If you have an Apple Silicon Mac, it should also say `Mach-O 64-bit executable arm64`. If you have an Intel Mac, it should also say `Mach-O 64-bit executable x86_64`.
- Run `which pip` in your terminal and it too should resolve to some path that implies you're using miniforge3.
If any of those sanity checks failed, you must redo this section. Please ensure that you downloaded the correct Miniforge for your system architecture and installed it. If you did all that, set your environment paths to Miniforge's Python Installation. To do that, you need to figure out where conda was installed to (it's probably `~/miniforge3/condabin/conda`) and then run `~/miniforge3/condabin/conda init` in your terminal.
## Apple Silicon Only Warning: You CANNOT use Anaconda
This warning only applies to Apple Silicon Macs. Anaconda comes with many Python packages included, some of which are not Apple Silicon (i.e. ARM) compatible and thus Anaconda is not ARM compatible. You can use Anaconda if you're using an Intel Mac though.
If you were planning to use Anaconda on ARM, please scroll back up and install Miniforge. Miniforge has Conda, which means you can install many of the packages you want such as Pandas, Scipy, and Numpy -- unlike Anaconda, you just have to do the install manually by running `conda install mypackagenamehere`.
## Intel Only Warning: Python Bugs in Big Sur
This warning only apply to Intel Macs. For Intel, both Anaconda and MiniForge have a [Python Bug](https://www.python.org/downloads/release/python-387/) which prevents you from running Python correctly in some instances on macOS Big Sur. Until the Python community fixes this, each time prior to loading Python, you must run `export SYSTEM_VERSION_COMPAT=0`. You could also add this to your `.bash_profile` or other shell environment file if you have one, to do this automatically for you.
## Installing TensorFlow
Attached to this Issue is a YAML file which will help you create a Conda Environment with TensorFlow, along with all the prerequisites you need from the ARM conda-forge channel.
1. Download [environment.yml](https://raw.githubusercontent.com/mwidjaja1/DSOnMacARM/main/environment.yml), which contains the instructions to create a Python environment with the dependencies you need -- we'll install TensorFlow afterwards. Some browsers insist on adding `.txt` to the end of the file -- do not let your browser do that. [thanks to @isuruf for streamlining this file to be all Conda]
2. In your terminal run this command, replacing the uppercase variables with the path to your environment.yml file and your desired name for this environment: `conda env create --file=PATH_TO_ENVIRONMENT.YML --name=YOUR_ENV_NAME_HERE`.
3. Activate that environment by running this command, replacing the uppercase variable with your environment's name: `conda activate YOUR_ENV_NAME_HERE`
4. Pip install the TensorFlow wheels by running the commands below. By the way, the URLs for the TensorFlow wheel files came from the [Releases](https://github.com/apple/tensorflow_macos/releases) page, so you can swap these wheel files out with a prior version of TensorFlow as needed.
**For X86 as of 03/11/2021:**
Thanks to @edwin-yan for the updated commands
```
pip install --upgrade --force --no-dependencies https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_x86_64.whl https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_addons_macos-0.1a3-cp38-cp38-macosx_11_0_x86_64.whl
```
**For Apple Silicon as of 03/11/2021:**
```
pip install --upgrade --force --no-dependencies https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_addons_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl
```
5. Finally, give it a spin. Run `python` and try importing `tensorflow`.
### Example Commands
In this below example, I'm installing & running the ARM version of tensorflow from an environment I've named `test`. The yml file is placed in the same directory I'm running this command from, which is my home directory (i.e. `~`)
```
conda env create --file=environment.yml --name=test
conda activate test
pip install --upgrade --force --no-dependencies https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha2/tensorflow_addons_macos-0.1a2-cp38-cp38-macosx_11_0_arm64.whl https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha2/tensorflow_macos-0.1a2-cp38-cp38-macosx_11_0_arm64.whl
python
import tensorflow
```
<img width="873" alt="Screen Shot 2021-02-02 at 10 19 26 PM" src="https://user-images.githubusercontent.com/7283317/106693560-bc1d2280-65a4-11eb-8555-ff2d1d9b8caa.png">
### Troubleshooting for importing TensorFlow
- Type in `which python` and then `which pip` in your terminal. Both paths should point to a Python that is **inside the environment** you created in Step 2. If it doesn't, you may not have installed Miniforge correctly, ran Step 2 correctly, and/or may not have ran Step 3.
- Run `python --version` and it should be version 3.8. If it isn't, you most likely did not create or activate your environment correctly, as per Steps 2 & 3. Do those again.
- If python is correctly pointed to the right environment but you cannot import tensorflow, consider running step 5 again just to make sure you installed Tensorflow in the appropriate environment.
- If you are using Intel and got a `not a supported wheel on this platform` error, run `export SYSTEM_VERSION_COMPAT=0` in your terminal and try again. If this works, you'll need to do this everytime you use Python until a [Python Bug](https://www.python.org/downloads/release/python-387/) is resolved.
- **Please verify that you did ALL of the Sanity Checks from the previous section and that they resolve appropriately before posting your issue here.** If you do post your issue, please provide the terminal outputs from those steps and bonus points if you share the results of your Sanity Check and run `pip` with a `-v` flag for additional logging. Remember I'm just a volunteer -- I'll try to help but there's only so much I can help with.
### Troubleshooting for setting up TensorFlow
- For those having issues with tf.keras.models.load_model about a `failed to decode` error: Try downgrading to h5py to the 2.10.0 wheel file that was [packaged with this alpha release](https://github.com/apple/tensorflow_macos/releases/download/) (`pip install ~/path to h5py.whl`). Thanks to @ramicaza.
And subscribe/upvote this:
opened 05:17PM - 21 Dec 20 UTC
I'm trying to train a model which works on kaggle, but won't work on my M1 macbo… ok pro. It says:
`ImportError: Image transformations require SciPy. Install SciPy`
I just built a model to identify plants and diseases, but the problem is that it isn't working when it's fitting. Compiling is fine, but it stops and gives me errors when the model tries to fit.
2 Likes
Followed the instructions on the first GitHub ticket, ended up working, model finally runs locally, averaging around 2s / step, as opposed to to the ~40s / step I was getting before when not running it locally. Thank you for your help, very much appreciated!
2 Likes
It looks like you’re hitting the error in this file:
"""Utilities for performing affine transformations on image data.
"""
import numpy as np
from .utils import array_to_img, img_to_array
try:
import scipy
# scipy.ndimage cannot be accessed until explicitly imported
from scipy import ndimage
except ImportError:
scipy = None
try:
from PIL import Image as pil_image
from PIL import ImageEnhance
except ImportError:
pil_image = None
ImageEnhance = None
This file has been truncated. show original
From the code I linked, that doesn’t sound possible. The error is only triggered is import scipy; import scipy.ndimage
fails.
Anyway, one good workaround for this is to not use keras.preprocessing
since these run in python/numpy/scipy not in tensorflow. keras.layers.experimental.preprocessing
contains pure-tensorflow layers that implemnent common image transformations.
https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/preprocessing/RandomRotation?version=nightly
We have several examples that use these.
2 Likes
Interesting. I was trying to do data augmentation and was using ImageDataGenerator, the keras.layers.experimental.preprocessing I think has everything I need apart from shear_range which is in ImageDataGenerator, how would I go about applying a shear_range in that ? I tried searching for it and was not able to find it.
1 Like
Bhack
June 2, 2021, 5:57pm
#8
I think we have TF native shear in the model garden package
2 Likes
And it looks like, similar to the keras.preprocessing implementation, all those layers.experimental.preprocessing
layers use this transform
function which just takes a projection matrix:
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/keras/layers/preprocessing/image_preprocessing.py#L621
So I think you’d just need to work out the right range of matrices there. Maybe there’s something you can copy from the keras.preprocessing
version.
Or maybe someone’s already done all that work.
2 Likes
Bhack
June 2, 2021, 6:05pm
#10
More than one I hope we could unify the image processing API one of this days:
2 Likes
Bhack
June 2, 2021, 7:27pm
#11
It is a “private” API. We have tried to expose this but the evaluation was to keep this private. See
tensorflow:master
← bhack:patch-11
opened 11:57AM - 16 Dec 20 UTC
See https://github.com/tensorflow/addons/pull/2293
1 Like