Increasing training accuracy of U-Net segmentation model

I was working with segmentation using u-net and MobileNet.

While I training with input 256*256 it had an output with Val loss: 0.044 (In this time dense layer was 256, 128, 64, 32 with a learning rate of 0.0001).

But I needed to modify input for deployment to 128*128. But then the val_loss increased drastically around 0.052 which is not acceptable.

What should I take care of?