How to handle TensorFlow warning "W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 1234567890 exceeds 10% of free system memory."

andreabazerla · January 11, 2023, 3:25am

Hi. How can I handle the following TensorFlow warning?

I know, it’s just a warning. But it crash my training phase after calculating validation loss at the end of epochs.

Otherwise, how can I catch that warning? Any solutions?

I read suggestions to reduce batch size, model size, dataset, etc. But it doesn’t help me because I’m hypertuning with Keras Tuner and there are some combinations of hyperparameters to exclude: too complex to do it first! I’ve been clear?

Thank you for your help.

2023-01-10 22:20:54.151974: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 1535203200 exceeds 10% of free system memory.
2023-01-10 22:21:05.259244: W tensorflow/core/common_runtime/bfc_allocator.cc:462] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.43GiB (rounded to 1535203328)requested by op _EagerConst
If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation. 
Current allocation summary follows.
Current allocation summary follows.
2023-01-10 22:21:05.259353: I tensorflow/core/common_runtime/bfc_allocator.cc:1010] BFCAllocator dump for GPU_0_bfc
2023-01-10 22:21:05.259395: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (256):  Total Chunks: 69, Chunks in use: 67. 17.2KiB allocated for chunks. 16.8KiB in use in bin. 2.0KiB client-requested in use in bin.
2023-01-10 22:21:05.259425: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (512):  Total Chunks: 7, Chunks in use: 6. 4.5KiB allocated for chunks. 4.0KiB in use in bin. 3.3KiB client-requested in use in bin.
2023-01-10 22:21:05.259454: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (1024):         Total Chunks: 7, Chunks in use: 7. 10.2KiB allocated for chunks. 10.2KiB in use in bin. 8.6KiB client-requested in use in bin.
2023-01-10 22:21:05.259482: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (2048):         Total Chunks: 4, Chunks in use: 3. 13.5KiB allocated for chunks. 11.2KiB in use in bin. 11.1KiB client-requested in use in bin.
2023-01-10 22:21:05.259510: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (4096):         Total Chunks: 3, Chunks in use: 3. 18.5KiB allocated for chunks. 18.5KiB in use in bin. 18.2KiB client-requested in use in bin.
2023-01-10 22:21:05.259541: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (8192):         Total Chunks: 10, Chunks in use: 10. 134.0KiB allocated for chunks. 134.0KiB in use in bin. 118.5KiB client-requested in use in bin.
2023-01-10 22:21:05.259568: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (16384):        Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-01-10 22:21:05.259598: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (32768):        Total Chunks: 3, Chunks in use: 3. 126.2KiB allocated for chunks. 126.2KiB in use in bin. 99.7KiB client-requested in use in bin.
2023-01-10 22:21:05.259626: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (65536):        Total Chunks: 1, Chunks in use: 0. 121.5KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-01-10 22:21:05.259653: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (131072):       Total Chunks: 2, Chunks in use: 0. 378.5KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-01-10 22:21:05.259678: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (262144):       Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-01-10 22:21:05.259703: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (524288):       Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-01-10 22:21:05.259728: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (1048576):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-01-10 22:21:05.259755: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (2097152):      Total Chunks: 1, Chunks in use: 1. 2.27MiB allocated for chunks. 2.27MiB in use in bin. 2.27MiB client-requested in use in bin.
2023-01-10 22:21:05.259781: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (4194304):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-01-10 22:21:05.259809: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (8388608):      Total Chunks: 2, Chunks in use: 2. 16.08MiB allocated for chunks. 16.08MiB in use in bin. 16.08MiB client-requested in use in bin.
2023-01-10 22:21:05.259839: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (16777216):     Total Chunks: 1, Chunks in use: 1. 16.82MiB allocated for chunks. 16.82MiB in use in bin. 16.82MiB client-requested in use in bin.
2023-01-10 22:21:05.259869: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (33554432):     Total Chunks: 3, Chunks in use: 3. 164.61MiB allocated for chunks. 164.61MiB in use in bin. 164.61MiB client-requested in use in bin.
2023-01-10 22:21:05.259894: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (67108864):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-01-10 22:21:05.259924: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (134217728):    Total Chunks: 1, Chunks in use: 1. 173.36MiB allocated for chunks. 173.36MiB in use in bin. 160.88MiB client-requested in use in bin.
2023-01-10 22:21:05.259952: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (268435456):    Total Chunks: 3, Chunks in use: 2. 2.14GiB allocated for chunks. 1.83GiB in use in bin. 1.83GiB client-requested in use in bin.
2023-01-10 22:21:05.260390: I tensorflow/core/common_runtime/bfc_allocator.cc:1033] Bin for 1.43GiB was 256.00MiB, Chunk State: 
2023-01-10 22:21:05.260439: I tensorflow/core/common_runtime/bfc_allocator.cc:1039]   Size: 314.40MiB | Requested Size: 8.12MiB | in_use: 0 | bin_num: 20, prev:   Size: 45.52MiB | Requested Size: 45.52MiB | in_use: 1 | bin_num: -1
2023-01-10 22:21:05.260461: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Next region of size 2691366912
2023-01-10 22:21:05.261721: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a000000 of size 256 next 1
2023-01-10 22:21:05.261763: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a000100 of size 1280 next 2
2023-01-10 22:21:05.261783: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a000600 of size 256 next 3
2023-01-10 22:21:05.261802: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a000700 of size 256 next 4
2023-01-10 22:21:05.261821: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a000800 of size 256 next 5
2023-01-10 22:21:05.261841: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a000900 of size 256 next 6
2023-01-10 22:21:05.261861: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a000a00 of size 256 next 9
2023-01-10 22:21:05.261881: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a000b00 of size 256 next 12
2023-01-10 22:21:05.261899: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a000c00 of size 256 next 10
2023-01-10 22:21:05.261918: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a000d00 of size 256 next 13
2023-01-10 22:21:05.261938: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a000e00 of size 256 next 15
2023-01-10 22:21:05.261958: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a000f00 of size 256 next 16
2023-01-10 22:21:05.261978: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a001000 of size 256 next 17
2023-01-10 22:21:05.261997: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a001100 of size 256 next 25
2023-01-10 22:21:05.262017: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a001200 of size 256 next 26
2023-01-10 22:21:05.262041: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a001300 of size 256 next 27
2023-01-10 22:21:05.262061: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a001400 of size 256 next 28
2023-01-10 22:21:05.262081: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a001500 of size 256 next 20
2023-01-10 22:21:05.262101: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a001600 of size 256 next 103
2023-01-10 22:21:05.262121: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a001700 of size 256 next 52
2023-01-10 22:21:05.262140: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a001800 of size 256 next 93
2023-01-10 22:21:05.262160: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a001900 of size 256 next 89
2023-01-10 22:21:05.262179: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a001a00 of size 256 next 22
2023-01-10 22:21:05.262199: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a001b00 of size 256 next 11
2023-01-10 22:21:05.262218: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a001c00 of size 256 next 97
2023-01-10 22:21:05.262238: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a001d00 of size 256 next 7
2023-01-10 22:21:05.262258: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a001e00 of size 256 next 74
2023-01-10 22:21:05.262279: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a001f00 of size 768 next 14
2023-01-10 22:21:05.262300: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a002200 of size 256 next 29
2023-01-10 22:21:05.262319: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a002300 of size 256 next 30
2023-01-10 22:21:05.262339: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a002400 of size 3840 next 24
2023-01-10 22:21:05.262384: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a003300 of size 3840 next 82
2023-01-10 22:21:05.262411: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a004200 of size 256 next 106
2023-01-10 22:21:05.262464: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a004300 of size 256 next 42
2023-01-10 22:21:05.262507: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a004400 of size 256 next 99
2023-01-10 22:21:05.262549: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a004500 of size 256 next 46
2023-01-10 22:21:05.262576: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a004600 of size 256 next 115
2023-01-10 22:21:05.262596: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 7f755a004700 of size 256 next 112
2023-01-10 22:21:05.262617: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a004800 of size 256 next 110
2023-01-10 22:21:05.262636: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a004900 of size 256 next 116
2023-01-10 22:21:05.262656: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a004a00 of size 256 next 109
2023-01-10 22:21:05.262676: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 7f755a004b00 of size 256 next 132
2023-01-10 22:21:05.262695: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a004c00 of size 256 next 131
2023-01-10 22:21:05.262715: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 7f755a004d00 of size 2304 next 33
2023-01-10 22:21:05.262736: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a005600 of size 5120 next 37
2023-01-10 22:21:05.262755: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a006a00 of size 256 next 43
2023-01-10 22:21:05.262775: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a006b00 of size 256 next 87
2023-01-10 22:21:05.262794: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a006c00 of size 256 next 62
2023-01-10 22:21:05.262829: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 7f755a006d00 of size 512 next 98
2023-01-10 22:21:05.262869: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a006f00 of size 256 next 44
2023-01-10 22:21:05.262911: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a007000 of size 1536 next 31
2023-01-10 22:21:05.262961: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a007600 of size 3840 next 41
2023-01-10 22:21:05.262987: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a008500 of size 1536 next 69
2023-01-10 22:21:05.263007: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a008b00 of size 1536 next 21
2023-01-10 22:21:05.263027: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a009100 of size 256 next 104
2023-01-10 22:21:05.263047: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a009200 of size 256 next 96
2023-01-10 22:21:05.263066: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a009300 of size 256 next 105
2023-01-10 22:21:05.263086: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a009400 of size 256 next 100
2023-01-10 22:21:05.263106: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a009500 of size 256 next 8
2023-01-10 22:21:05.263125: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a009600 of size 256 next 70
2023-01-10 22:21:05.263145: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a009700 of size 1536 next 55
2023-01-10 22:21:05.263164: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a009d00 of size 1536 next 80
2023-01-10 22:21:05.263199: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a00a300 of size 768 next 59
2023-01-10 22:21:05.263239: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a00a600 of size 1536 next 81
2023-01-10 22:21:05.263282: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a00ac00 of size 256 next 66
2023-01-10 22:21:05.263315: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a00ad00 of size 768 next 79
2023-01-10 22:21:05.263337: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a00b000 of size 14592 next 101
2023-01-10 22:21:05.263358: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a00e900 of size 14592 next 107
2023-01-10 22:21:05.263398: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a012200 of size 14592 next 94
2023-01-10 22:21:05.263440: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a015b00 of size 15104 next 18
2023-01-10 22:21:05.263485: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a019600 of size 256 next 56
2023-01-10 22:21:05.263515: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a019700 of size 61184 next 34
2023-01-10 22:21:05.263537: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a028600 of size 512 next 108
2023-01-10 22:21:05.263558: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a028800 of size 34048 next 84
2023-01-10 22:21:05.263578: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a030d00 of size 34048 next 102
2023-01-10 22:21:05.263598: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 7f755a039200 of size 136448 next 39
2023-01-10 22:21:05.263618: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a05a700 of size 256 next 40
2023-01-10 22:21:05.263638: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a05a800 of size 256 next 36
2023-01-10 22:21:05.263658: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a05a900 of size 512 next 35
2023-01-10 22:21:05.263677: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a05ab00 of size 256 next 71
2023-01-10 22:21:05.263698: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a05ac00 of size 256 next 47
2023-01-10 22:21:05.263741: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a05ad00 of size 256 next 38
2023-01-10 22:21:05.263779: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a05ae00 of size 768 next 48
2023-01-10 22:21:05.263815: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a05b100 of size 256 next 49
2023-01-10 22:21:05.263858: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a05b200 of size 256 next 50
2023-01-10 22:21:05.263901: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a05b300 of size 256 next 51
2023-01-10 22:21:05.263925: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a05b400 of size 256 next 23
2023-01-10 22:21:05.263945: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a05b500 of size 256 next 54
2023-01-10 22:21:05.263965: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a05b600 of size 256 next 77
2023-01-10 22:21:05.263984: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a05b700 of size 256 next 61
2023-01-10 22:21:05.264004: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a05b800 of size 256 next 90
2023-01-10 22:21:05.264038: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a05b900 of size 256 next 60
2023-01-10 22:21:05.264082: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a05ba00 of size 256 next 53
2023-01-10 22:21:05.264128: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a05bb00 of size 256 next 64
2023-01-10 22:21:05.264161: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a05bc00 of size 256 next 88
2023-01-10 22:21:05.264182: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a05bd00 of size 256 next 85
2023-01-10 22:21:05.264204: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a05be00 of size 14848 next 78
2023-01-10 22:21:05.264248: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a05f800 of size 6912 next 92
2023-01-10 22:21:05.264290: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a061300 of size 6912 next 58
2023-01-10 22:21:05.264328: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a062e00 of size 9728 next 76
2023-01-10 22:21:05.264369: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 7f755a065400 of size 124416 next 121
2023-01-10 22:21:05.264412: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a083a00 of size 9984 next 123
2023-01-10 22:21:05.264452: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 7f755a086100 of size 251136 next 68
2023-01-10 22:21:05.264497: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a0c3600 of size 14592 next 57
2023-01-10 22:21:05.264531: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a0c6f00 of size 14592 next 72
2023-01-10 22:21:05.264567: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755a0ca800 of size 62440192 next 65
2023-01-10 22:21:05.264610: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755dc56b00 of size 8429568 next 86
2023-01-10 22:21:05.264658: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755e460b00 of size 14592 next 91
2023-01-10 22:21:05.264698: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755e464400 of size 17640192 next 75
2023-01-10 22:21:05.264731: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755f536f00 of size 2381568 next 63
2023-01-10 22:21:05.264764: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f755f77c600 of size 181783040 next 32
2023-01-10 22:21:05.264796: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f756a4d9000 of size 62440192 next 67
2023-01-10 22:21:05.264828: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f756e065300 of size 8429568 next 83
2023-01-10 22:21:05.264861: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f756e86f300 of size 1535203328 next 73
2023-01-10 22:21:05.264904: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f75ca084b00 of size 434369024 next 113
2023-01-10 22:21:05.264953: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f75e3ec3d00 of size 47730176 next 122
2023-01-10 22:21:05.264985: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 7f75e6c48b00 of size 329676032 next 18446744073709551615
2023-01-10 22:21:05.265006: I tensorflow/core/common_runtime/bfc_allocator.cc:1071]      Summary of in-use Chunks by size: 
2023-01-10 22:21:05.265037: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 67 Chunks of size 256 totalling 16.8KiB
2023-01-10 22:21:05.265088: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 512 totalling 1.0KiB
2023-01-10 22:21:05.265135: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 4 Chunks of size 768 totalling 3.0KiB
2023-01-10 22:21:05.265183: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 1280 totalling 1.2KiB
2023-01-10 22:21:05.265231: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 6 Chunks of size 1536 totalling 9.0KiB
2023-01-10 22:21:05.265281: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 3 Chunks of size 3840 totalling 11.2KiB
2023-01-10 22:21:05.265323: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 5120 totalling 5.0KiB
2023-01-10 22:21:05.265359: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 6912 totalling 13.5KiB
2023-01-10 22:21:05.265395: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 9728 totalling 9.5KiB
2023-01-10 22:21:05.265431: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 9984 totalling 9.8KiB
2023-01-10 22:21:05.265466: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 6 Chunks of size 14592 totalling 85.5KiB
2023-01-10 22:21:05.265502: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 14848 totalling 14.5KiB
2023-01-10 22:21:05.265543: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 15104 totalling 14.8KiB
2023-01-10 22:21:05.265594: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 34048 totalling 66.5KiB
2023-01-10 22:21:05.265648: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 61184 totalling 59.8KiB
2023-01-10 22:21:05.265688: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 2381568 totalling 2.27MiB
2023-01-10 22:21:05.265725: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 8429568 totalling 16.08MiB
2023-01-10 22:21:05.265761: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 17640192 totalling 16.82MiB
2023-01-10 22:21:05.265798: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 47730176 totalling 45.52MiB
2023-01-10 22:21:05.265839: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 62440192 totalling 119.09MiB
2023-01-10 22:21:05.265886: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 181783040 totalling 173.36MiB
2023-01-10 22:21:05.265926: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 434369024 totalling 414.25MiB
2023-01-10 22:21:05.265950: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 1535203328 totalling 1.43GiB
2023-01-10 22:21:05.265972: I tensorflow/core/common_runtime/bfc_allocator.cc:1078] Sum Total of in-use chunks: 2.20GiB
2023-01-10 22:21:05.266017: I tensorflow/core/common_runtime/bfc_allocator.cc:1080] total_region_allocated_bytes_: 2691366912 memory_limit_: 2691366912 available bytes: 0 curr_region_allocation_bytes_: 5382733824
2023-01-10 22:21:05.266453: I tensorflow/core/common_runtime/bfc_allocator.cc:1086] Stats: 
Limit:                      2691366912
InUse:                      2361175552
MaxInUse:                   2370018048
NumAllocs:                     2465021
MaxAllocSize:               1535203328
Reserved:                            0
PeakReserved:                        0
LargestFreeBlock:                    0

2023-01-10 22:21:05.266529: W tensorflow/core/common_runtime/bfc_allocator.cc:474] ****************************************************************************************___________

Antony_Smith · February 23, 2023, 1:35pm

Did you ever find a solution to this issue?
I am having the same problem with ResNet-50, Inception-V3 and VGG-16 models on both local Linux systems and Google Colab.
The dataset is small (3600) with image dims[224, 224, 3] and I’ve already reduced batch size from 128 to 32 to 10.
It finishes fitting but then crashes when calculating validation.
Anyone have a better fix for this? Something along the lines of more dynamic memory allocation.

Mah_Neh · February 26, 2023, 5:51pm

When I had to run VGG16, I had those problems until I wrapped the the main function (in my case it was the function that was processing images) before passing them to .fit I think, into a .tidy(myFn)

I have not used it but I think the memory() function may tell you the same, i.e that some tensors arent disposed and so you run out of memory.

@Antony_Smith i dont mind to test your colab if you need help.

Antony_Smith · February 27, 2023, 6:01am

Thanks, I will have a look at this suggestion.
I did find a solution that helped with my EfficientNet-B0 issue that was also giving this “memory over-resourced” error.
Running this line in the script before the other library imports seemed to manage the memory dynamically and dispose of memory leaks:

os.environ["tf_gpu_allocator"]="cuda_malloc_async"

However, it did not solve the problem with VGG-16 (keras import), nor with a manually constructed copy of the VGG-16 model. Both still have memory over-resourced errors.

Thank you for the offer to test Colab, I am open to sending you the project if you would like to look at it? Although, I will have to also share the dataset with you, as it is a slightly modified version of the Kaggle dataset.

Mah_Neh · February 27, 2023, 9:54am

Yes, just send the link through a message, I will take a look and see. I am not an expert but maybe my prev error was similar enough.