GPU memory is fully occupied despite setting `set_memory_growth`

Hey,
I want to limit my GPU memory and I tried setting set_memory_growth and tf.config.LogicalDeviceConfiguration and seems like the GPU still consumes all the memory, even when I set batch size = 1. Is there any solution on this? Thanks