Tensorflow 2.15 installation problem

Lucas_She · February 7, 2024, 5:12am

Recently I am trying to install tf 2.15 on my laptop runs Ubuntu22.04, with RTX4090 and driver version 535.146.02. I just followed the instructions for pip installing.
I created a virtual environment with anaconda and runs ```
pip install tensorflow[and-cuda]

It successfully installed, however when I tried to verify this installation with tf.test.is_gpu_available(), 
It shows me 

WARNING:tensorflow:From <stdin>:1: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
2024-02-07 12:44:09.953717: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-02-07 12:44:10.247841: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-02-07 12:44:10.247949: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-02-07 12:44:10.335250: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-02-07 12:44:10.335335: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-02-07 12:44:10.335405: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-02-07 12:44:10.335472: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /device:GPU:0 with 13183 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 4090 Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.9
True

Though it shows True, when I tried to create a model, it still runs on CPU

The tensorflow and CUDA and CUDNN versions are listed:
nvidia-cublas-cu12           12.2.5.6
nvidia-cuda-cupti-cu12       12.2.142
nvidia-cuda-nvcc-cu12        12.2.140
nvidia-cuda-nvrtc-cu12       12.2.140
nvidia-cuda-runtime-cu12     12.2.140
nvidia-cudnn-cu12            8.9.4.25
nvidia-cufft-cu12            11.0.8.103
nvidia-curand-cu12           10.3.3.141
nvidia-cusolver-cu12         11.5.2.141
nvidia-cusparse-cu12         12.1.2.141
nvidia-nccl-cu12             2.16.5
nvidia-nvjitlink-cu12        12.2.140
tensorflow                   2.15.0.post1
tensorflow-estimator         2.15.0
tensorflow-io-gcs-filesystem 0.36.0

Kiran_Sai_Ramineni · February 7, 2024, 6:31am

Hi @Lucas_She, Could you please let us know how you are able to mention that code is utilizing the CPU instead of GPU. If possible could you please provide the reproducible code. Thank You.

Lucas_She · February 7, 2024, 7:17am

I just tried to run my codes again, however when I am using model.compile(optimizer=optimizer, loss=“sparse_categorical_crossentropy”), there raises a error
2024-02-07 15:13:57.163290: W external/local_tsl/tsl/framework/bfc_allocator.cc:485] Allocator (GPU_0_bfc) ran out of memory trying to allocate 384.0KiB (rounded to 393216)requested by op AddV2
If the cause is memory fragmentation maybe the environment variable ‘TF_GPU_ALLOCATOR=cuda_malloc_async’ will improve the situation.
Current allocation summary follows.
Current allocation summary follows.
2024-02-07 15:13:57.163329: I external/local_tsl/tsl/framework/bfc_allocator.cc:1039] BFCAllocator dump for GPU_0_bfc
2024-02-07 15:13:57.163337: I external/local_tsl/tsl/framework/bfc_allocator.cc:1046] Bin (256): Total Chunks: 22, Chunks in use: 22. 5.5KiB allocated for chunks. 5.5KiB in use in bin. 1.1KiB client-requested in use in bin.
2024-02-07 15:13:57.163342: I external/local_tsl/tsl/framework/bfc_allocator.cc:1046] Bin (512): Total Chunks: 2, Chunks in use: 2. 1.0KiB allocated for chunks. 1.0KiB in use in bin. 1.0KiB client-requested in use in bin.
2024-02-07 15:13:57.163347: I external/local_tsl/tsl/framework/bfc_allocator.cc:1046] Bin (1024): Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2024-02-07 15:13:57.163352: I external/local_tsl/tsl/framework/bfc_allocator.cc:1046] Bin (2048): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-02-07 15:13:57.163356: I external/local_tsl/tsl/framework/bfc_allocator.cc:1046] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-02-07 15:13:57.163362: I external/local_tsl/tsl/framework/bfc_allocator.cc:1046] Bin (8192): Total Chunks: 4, Chunks in use: 2. 40.5KiB allocated for chunks. 18.0KiB in use in bin. 18.0KiB client-requested in use in bin.
2024-02-07 15:13:57.163366: I external/local_tsl/tsl/framework/bfc_allocator.cc:1046] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-02-07 15:13:57.163370: I external/local_tsl/tsl/framework/bfc_allocator.cc:1046] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-02-07 15:13:57.163374: I external/local_tsl/tsl/framework/bfc_allocator.cc:1046] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-02-07 15:13:57.163379: I external/local_tsl/tsl/framework/bfc_allocator.cc:1046] Bin (131072): Total Chunks: 3, Chunks in use: 2. 576.0KiB allocated for chunks. 384.0KiB in use in bin. 384.0KiB client-requested in use in bin.
2024-02-07 15:13:57.163387: I external/local_tsl/tsl/framework/bfc_allocator.cc:1046] Bin (262144): Total Chunks: 3, Chunks in use: 2. 1.11MiB allocated for chunks. 768.0KiB in use in bin. 768.0KiB client-requested in use in bin.
2024-02-07 15:13:57.163393: I external/local_tsl/tsl/framework/bfc_allocator.cc:1046] Bin (524288): Total Chunks: 2, Chunks in use: 2. 1.41MiB allocated for chunks. 1.41MiB in use in bin. 1.12MiB client-requested in use in bin.
2024-02-07 15:13:57.163398: I external/local_tsl/tsl/framework/bfc_allocator.cc:1046] Bin (1048576): Total Chunks: 1, Chunks in use: 1. 1.50MiB allocated for chunks. 1.50MiB in use in bin. 1.50MiB client-requested in use in bin.
2024-02-07 15:13:57.163402: I external/local_tsl/tsl/framework/bfc_allocator.cc:1046] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-02-07 15:13:57.163406: I external/local_tsl/tsl/framework/bfc_allocator.cc:1046] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-02-07 15:13:57.163411: I external/local_tsl/tsl/framework/bfc_allocator.cc:1046] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-02-07 15:13:57.163415: I external/local_tsl/tsl/framework/bfc_allocator.cc:1046] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-02-07 15:13:57.163422: I external/local_tsl/tsl/framework/bfc_allocator.cc:1046] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-02-07 15:13:57.163426: I external/local_tsl/tsl/framework/bfc_allocator.cc:1046] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-02-07 15:13:57.163431: I external/local_tsl/tsl/framework/bfc_allocator.cc:1046] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-02-07 15:13:57.163435: I external/local_tsl/tsl/framework/bfc_allocator.cc:1046] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-02-07 15:13:57.163440: I external/local_tsl/tsl/framework/bfc_allocator.cc:1062] Bin for 384.0KiB was 256.0KiB, Chunk State:
2024-02-07 15:13:57.163448: I external/local_tsl/tsl/framework/bfc_allocator.cc:1068] Size: 365.8KiB | Requested Size: 192.0KiB | in_use: 0 | bin_num: 10, prev: Size: 9.0KiB | Requested Size: 9.0KiB | in_use: 1 | bin_num: -1, next: Size: 192.0KiB | Requested Size: 192.0KiB | in_use: 1 | bin_num: -1
2024-02-07 15:13:57.163452: I external/local_tsl/tsl/framework/bfc_allocator.cc:1075] Next region of size 4849664
2024-02-07 15:13:57.163457: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1400000 of size 256 next 1
2024-02-07 15:13:57.163462: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1400100 of size 1280 next 2
2024-02-07 15:13:57.163465: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1400600 of size 256 next 3
2024-02-07 15:13:57.163469: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1400700 of size 256 next 4
2024-02-07 15:13:57.163473: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1400800 of size 256 next 5
2024-02-07 15:13:57.163476: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1400900 of size 256 next 7
2024-02-07 15:13:57.163480: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1400a00 of size 256 next 8
2024-02-07 15:13:57.163484: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1400b00 of size 256 next 6
2024-02-07 15:13:57.163488: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1400c00 of size 256 next 9
2024-02-07 15:13:57.163491: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1400d00 of size 256 next 14
2024-02-07 15:13:57.163495: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1400e00 of size 256 next 12
2024-02-07 15:13:57.163499: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1400f00 of size 256 next 13
2024-02-07 15:13:57.163503: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1401000 of size 512 next 17
2024-02-07 15:13:57.163507: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1401200 of size 256 next 18
2024-02-07 15:13:57.163511: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1401300 of size 256 next 21
2024-02-07 15:13:57.163514: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1401400 of size 512 next 24
2024-02-07 15:13:57.163518: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1401600 of size 256 next 22
2024-02-07 15:13:57.163522: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1401700 of size 256 next 23
2024-02-07 15:13:57.163526: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1401800 of size 256 next 27
2024-02-07 15:13:57.163532: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1401900 of size 256 next 28
2024-02-07 15:13:57.163536: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1401a00 of size 256 next 29
2024-02-07 15:13:57.163540: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] Free at 7f53d1401b00 of size 14592 next 10
2024-02-07 15:13:57.163544: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1405400 of size 9216 next 11
2024-02-07 15:13:57.163550: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1407800 of size 256 next 31
2024-02-07 15:13:57.163554: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1407900 of size 256 next 30
2024-02-07 15:13:57.163557: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1407a00 of size 256 next 32
2024-02-07 15:13:57.163561: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1407b00 of size 256 next 35
2024-02-07 15:13:57.163565: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] Free at 7f53d1407c00 of size 8448 next 33
2024-02-07 15:13:57.163568: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1409d00 of size 9216 next 34
2024-02-07 15:13:57.163572: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] Free at 7f53d140c100 of size 374528 next 16
2024-02-07 15:13:57.163576: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1467800 of size 196608 next 15
2024-02-07 15:13:57.163580: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] Free at 7f53d1497800 of size 196608 next 36
2024-02-07 15:13:57.163583: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d14c7800 of size 196608 next 20
2024-02-07 15:13:57.163587: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d14f7800 of size 393216 next 19
2024-02-07 15:13:57.163591: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1557800 of size 1572864 next 26
2024-02-07 15:13:57.163595: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d16d7800 of size 786432 next 25
2024-02-07 15:13:57.163599: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d1797800 of size 393216 next 37
2024-02-07 15:13:57.163603: I external/local_tsl/tsl/framework/bfc_allocator.cc:1095] InUse at 7f53d17f7800 of size 690176 next 18446744073709551615
2024-02-07 15:13:57.163607: I external/local_tsl/tsl/framework/bfc_allocator.cc:1100] Summary of in-use Chunks by size:
2024-02-07 15:13:57.163612: I external/local_tsl/tsl/framework/bfc_allocator.cc:1103] 22 Chunks of size 256 totalling 5.5KiB
2024-02-07 15:13:57.163617: I external/local_tsl/tsl/framework/bfc_allocator.cc:1103] 2 Chunks of size 512 totalling 1.0KiB
2024-02-07 15:13:57.163621: I external/local_tsl/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 1280 totalling 1.2KiB
2024-02-07 15:13:57.163625: I external/local_tsl/tsl/framework/bfc_allocator.cc:1103] 2 Chunks of size 9216 totalling 18.0KiB
2024-02-07 15:13:57.163630: I external/local_tsl/tsl/framework/bfc_allocator.cc:1103] 2 Chunks of size 196608 totalling 384.0KiB
2024-02-07 15:13:57.163634: I external/local_tsl/tsl/framework/bfc_allocator.cc:1103] 2 Chunks of size 393216 totalling 768.0KiB
2024-02-07 15:13:57.163639: I external/local_tsl/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 690176 totalling 674.0KiB
2024-02-07 15:13:57.163643: I external/local_tsl/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 786432 totalling 768.0KiB
2024-02-07 15:13:57.163647: I external/local_tsl/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 1572864 totalling 1.50MiB
2024-02-07 15:13:57.163652: I external/local_tsl/tsl/framework/bfc_allocator.cc:1107] Sum Total of in-use chunks: 4.06MiB
2024-02-07 15:13:57.163656: I external/local_tsl/tsl/framework/bfc_allocator.cc:1109] Total bytes in pool: 4849664 memory_limit_: 4849664 available bytes: 0 curr_region_allocation_bytes_: 9699328
2024-02-07 15:13:57.163662: I external/local_tsl/tsl/framework/bfc_allocator.cc:1114] Stats:
Limit: 4849664
InUse: 4255488
MaxInUse: 4255744
NumAllocs: 65
MaxAllocSize: 1572864
Reserved: 0
PeakReserved: 0
LargestFreeBlock: 0

2024-02-07 15:13:57.163669: W external/local_tsl/tsl/framework/bfc_allocator.cc:497] ______***___******************************************************************************xxxxxx
2024-02-07 15:13:57.163691: W tensorflow/core/framework/op_kernel.cc:1827] RESOURCE_EXHAUSTED: failed to allocate memory

ResourceExhaustedError Traceback (most recent call last)
Cell In[21], line 2
1 from tensorflow.keras.metrics import IoU
----> 2 model = unet()
3 model.compile(optimizer=optimizer, loss=“sparse_categorical_crossentropy”)

Cell In[18], line 7, in unet()
5 p0 = inputs
6 c1, p1 = down_block(p0, 64) #128 → 64
----> 7 c2, p2 = down_block(p1, 642) #64 → 32
8 c3, p3 = down_block(p2, 644) #32 → 16
9 c4, p4 = down_block(p3, 64*8) #16 ->8

Cell In[17], line 2, in down_block(x, filters, kernel_size, padding, strides)
1 def down_block(x, filters, kernel_size=(3, 4), padding=“same”, strides=1):
----> 2 c = layers.Conv2D(filters, kernel_size, padding=padding, strides=strides, activation=“relu”)(x)
3 c = layers.Conv2D(filters, kernel_size, padding=padding, strides=strides, activation=“relu”)(c)
4 #c = tf.nn.dropout(c, 0.5) #add one dropout

File ~/.conda/envs/tf/lib/python3.9/site-packages/keras/src/utils/traceback_utils.py:70, in filter_traceback..error_handler(*args, **kwargs)
67 filtered_tb = _process_traceback_frames(e.traceback)
68 # To get the full stack trace, call:
69 # tf.debugging.disable_traceback_filtering()
—> 70 raise e.with_traceback(filtered_tb) from None
71 finally:
72 del filtered_tb

File ~/.conda/envs/tf/lib/python3.9/site-packages/keras/src/backend.py:2100, in RandomGenerator.random_uniform(self, shape, minval, maxval, dtype, nonce)
2098 if nonce:
2099 seed = tf.random.experimental.stateless_fold_in(seed, nonce)
→ 2100 return tf.random.stateless_uniform(
2101 shape=shape,
2102 minval=minval,
2103 maxval=maxval,
2104 dtype=dtype,
2105 seed=seed,
2106 )
2107 return tf.random.uniform(
2108 shape=shape,
2109 minval=minval,
(…)
2112 seed=self.make_legacy_seed(),
2113 )

ResourceExhaustedError: {{function_node _wrapped__AddV2_device/job:localhost/replica:0/task:0/device:GPU:0}} failed to allocate memory [Op:AddV2] name:

I am sure that this code works well on tensorflow2.9 with GPU on windows

Kiran_Sai_Ramineni · February 8, 2024, 6:16am

Hi @Lucas_She, If possible could you please provide the stand alone code to reproduce the issue. Thank You.

Lucas_She · February 8, 2024, 7:49am

My code is here https://github.com/Lucas-SJY/segmentation/blob/master/sun.ipynb
This code works fine on Windows and it got a little issues on Ubuntu 22.04. The error messages should in the ipynb notebook. However it sometimes runs on GPU, I have no idea how to make my environment stable.