Hi every one
I have two gpu cards(3090) to execute two different tensorflow(2.9.1) programs. Each of the programs may consume over half of cpu memory (512GB) on my computer.
When I executed the programs written in the functional API style(model.fit was used), the two programs carried out the training with cpu memory consumption around 90 %. However after I rewrote the programs in the subclassing style, the programs or the computer were stopped on the way after the cpu memory consumption of around 100%.
I tested to limit the buffersize in prefetch() and size in batch() in the pipline of the input data, however there is no effect.
How can I limit the cpu memory consumption in the subclassing style?
Best regards
Yu
Hi @Yuichiro_Morishita, As per my knowledge setting the memory limit on the CPU is not supported, currently it is only available on the GPU.As this is a feature request type of issue could you please create a issue on the github. Thank You
Dear Kiran
Thank you for your reply.
I will deal with the problem by increasing the swap memory.
Dear
this is my self-reply to my question.
Increasing the swap memory helps to avoid the hung up the computer, but the speed is too slow to put up with. I changed the pipeline from reading from CPU memory to from HDD( in the former all the data are loaded in memory, while in the latter the filenames are loaded). In this case, two different programs are run in parallel.
You can try adjusting batch sizes, prefetch buffer sizes, and consider using techniques like data sharding or data augmentation to reduce memory usage. Additionally, monitoring memory usage with tools like TensorFlow Profiler or system monitoring tools can help identify memory bottlenecks more precisely.
Thank you for your reply
Although not yet tested, I think that cache() in the data pipeline may fill out the cpu memory.