GPU utilization doesn't go above 20%

General Discussion
#1

Hey,
I am training a CNN model to classify images of the DERMNET dataset. My GPU utilization doesn’t go above 20%. Can someone help, please?

I have tried using:
tf.config.experimental.set_virtual_device_configuration(
physical_devices[0],
[tf.config.experimental.VirtualDeviceConfiguration(memory_limit=1024*10)])

And this is the code for my model:
input_shape = (batch_size, image_size, image_size, channels)
n_classes = 23
model = keras.models.Sequential([

resize_rescale,
data_augmentation,

keras.layers.Conv2D(32, (3,3), activation='relu', input_shape = input_shape),
keras.layers.MaxPool2D((2,2)),
keras.layers.Conv2D(64,  kernel_size = (3,3), activation='relu'),
keras.layers.MaxPooling2D((2, 2)),
keras.layers.Conv2D(64,  kernel_size = (3,3), activation='relu'),
keras.layers.MaxPooling2D((2, 2)),
keras.layers.Conv2D(64, (3, 3), activation='relu'),
keras.layers.MaxPooling2D((2, 2)),
keras.layers.Conv2D(64, (3, 3), activation='relu'),
keras.layers.MaxPooling2D((2, 2)),
keras.layers.Conv2D(64, (3, 3), activation='relu'),
keras.layers.MaxPooling2D((2, 2)),
keras.layers.Flatten(),
keras.layers.Dense(64, activation='relu'),
keras.layers.Dense(n_classes, activation='softmax'),

])

model.build(input_shape=input_shape)

#2

i think i have a answer.

#4

Hey, Can you tell me the answer?

#6

I think it lies with the size of your network. When the network size is quite small the GPU dosn’t utilize that much. But in my experience this should not be a big issue.

#8

Hi @Abdul_Rahmaan, May i know the batch_size you are using while training. Thank You.

#10

I was using 32 as the batch size now I am using it as 5 still the utilization has not increased

#11

Did you try logical_devices in place of physical_devices?..In its docs its written that VirtualDeviceConfiguration class is configuration class for logical_devices…maybe it’s not assigning the memory limit because of the physical device instance given?

#12

It shows no logical device found

#13

can you show the outputs for either tf.config.list_logical_devices() or tf.config.list_physical_devices() ? i suspect your gpu is not compatible with the tensorflow installed or its not properly installed

1 Like
#14

image
image946×528 15.3 KB

#15

Screenshot 2023-09-18 213129
Screenshot 2023-09-18 2131291212×109 5.6 KB

#16

can you try this … tf.config.experimental.set_virtual_device_configuration(physical_devices[0],[tf.config.LogicalDeviceConfiguration(memory_limit=1024)])

btw my bad actually we need to give physical device instance only in set_virtual_device_configuration even i was using it in my pc…for some reason i read the documentation incorrectly

1 Like
#17

i tried it, the utilization didn’t improve, is there any other way to increase it?

#18

This is the exact code given in the section Limiting GPU memory growth
I am currently using the same, I see that its not working with tf.config.experimental.set_virtual_device_configuration can you try this exact block instead? Thanks

gpus = tf.config.list_physical_devices('GPU')
if gpus:
  # Restrict TensorFlow to only allocate 1GB of memory on the first GPU
  try:
    tf.config.set_logical_device_configuration(
        gpus[0],
        [tf.config.LogicalDeviceConfiguration(memory_limit=1024)])
    logical_gpus = tf.config.list_logical_devices('GPU')
    print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
  except RuntimeError as e:
    # Virtual devices must be set before GPUs have been initialized
    print(e)
1 Like
#19

Hey, they utilzation hasn’t improved. Is there any changes that can be made in the system, like control panel or device manager as even when I play games, the utilization doesn’t go above 50%

#21

hit me up here bro we can discuss more there Slack

1 Like