I’ve been running a script that trains a neural net to perform image restoration (CSBDeep – a toolbox for CARE — CSBDeep 0.7.3 documentation)
It works fully when I run it from an admin account, but on a non-admin user account (“User” from now on) it runs on the CPU. The User account doesn’t even recognise that there is a GPU there (according to ```
from tensorflow.python.client import device_lib), but the admin account does recognise the GPU.
I have gone back to NVIDIA’s site and re-downloaded the CUDA drivers and made sure that I haven’t missed something like “install for all users on this machine”. I have made a new install of Anaconda on the User account and directly copied the environment . I have enabled permissions for the desired user in /Program Data/Anaconda3/, and it’s still not working.
I don’t understand why Tensorflow cannot see the GPU on a non-admin account. If anyone can help, I would be extremely grateful because I’m at the stage of tearing my hair out!
@cmci_concordia,
Could you please tell us which account (administrator or user) you used to configure the GPU? Does user account has necessary access to the CUDA directory?
What version of Tensorflow are you using? also give more information about your environment to debug further.
Thank you!
It sounds like you’re experiencing a common issue where TensorFlow is unable to access the GPU from a non-admin user account, despite working fine under an admin account. This can be due to several reasons related to permissions, environment variables, or driver installations. Here are a few steps you can take to troubleshoot and potentially resolve the issue:
1. Check NVIDIA Driver and CUDA Toolkit Installation:
Ensure that the NVIDIA driver and CUDA Toolkit are correctly installed and accessible to all users on the system. Sometimes, installations or certain components might be restricted to admin users only. Reinstalling or updating them with the option (if available) to install for all users might help.
2. Verify CUDA and cuDNN Paths:
Ensure that the CUDA and cuDNN paths are correctly set in the environment variables for the non-admin user account. TensorFlow requires these paths to be set so it can recognize and utilize the GPU. The required paths are typically something like C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\vX.Y\bin, C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\vX.Y\libnvvp, and the corresponding paths for cuDNN.
3. Permissions on CUDA Folders:
Check the permissions on the CUDA installation directories and ensure that the non-admin user account has read and execute permissions. Sometimes, the lack of appropriate permissions can prevent TensorFlow from accessing the necessary GPU resources.
4. TensorFlow and Dependency Versions:
Ensure that the TensorFlow version you are using, along with its dependencies, supports the installed versions of CUDA and cuDNN. Compatibility issues might cause TensorFlow to default to CPU usage if it cannot work with the available GPU resources.
5. Virtual Environment:
Since you mentioned using Anaconda, ensure that the virtual environment is correctly set up and activated in the non-admin account. Copying the environment might sometimes miss environment variables or other configurations essential for GPU recognition.
6. Check for Administrative Restrictions:
Some systems, especially those managed by organizations, might have policies that restrict hardware access to non-admin users. Verify if there are any such restrictions in place that might be preventing GPU access.
7. Use TensorFlow GPU Test Script:
Run a simple TensorFlow script that lists available devices to see if the GPU is recognized under the non-admin account. The script you mentioned (from tensorflow.python.client import device_lib; print(device_lib.list_local_devices())) is a good start.
8. Consult System Logs:
Check system logs for any errors related to GPU access or CUDA when trying to run TensorFlow from the non-admin account. This can provide clues if there’s a permission issue or a missing dependency.