Fully Convolutional Network with 2 different-sized inputs

Hi, everyone.

First time long time.

Build a fully convolutional neural network that mimics pan-sharpening (take a low-res image and upsample and refine based on a high-res image).

The model errors out that the image shapes are incompatible.


def Sentinel2Model(scaling_factor = 4,
                   filter_size  = 128,
                   num_blocks   = 6,
                   interpolation = 'bilinear',
                   filter_first = None,
                   filter_res   = None,
                   filter_last  = None):

    if filter_first == None: filter_first = filter_size
    if filter_res   == None: filter_res   = filter_size
    if filter_last  == None: filter_last  = filter_size

    # Hi_Res input
    in_hi = Input(shape = [None, None, 1], name = 'input_hi')

    #Low-res input
    in_low = Input(shape = [None, None, 1], name = 'input_low')

    # Up-rez the input
    up_low = UpSampling2D(size = (scaling_factor, scaling_factor), interpolation = interpolation, name = 'Upsampling_low-resolution_input')(in_low)

    concat = Concatenate(name='Concatenate_two-channels')([in_hi,up_low])
    # First treat the concatenation 
    x = Conv2D(filter_first, (3,3), kernel_initializer = 'glorot_uniform', activation='relu', padding='same', name= 'conv2d_initial')(concat)

    # Residual Blocks
    for block in range(num_blocks): 
        x = ResidualBlock(filters = filter_size)(x)

    x = Conv2D(1, (3,3), kernel_initializer = 'glorot_uniform', padding='same',name='conv2d_final')(x)

    output = Add(name='addition_final')([x, in_low])

    return K.models.Model(inputs = [in_hi, in_low], outputs = output)

…preprocessing code snippets…

def normalize(image):
    return image / 255.0

def decode_file(image_file):
    image = tf.io.read_file(image_file)
    image = tf.image.decode_png(image, channels=1)
    if len(image.shape) == 2:
        image = tf.expand_dims(image, axis = -1)
    return normalize(tf.cast(image, tf.float32))

def create_full_dataset_2(X_hi, X_low, Y_hi, X_hi_name='input_hi', X_low_name='input_low', scale=4):
    X_hi = X_hi.map(decode_file)
    X_low = X_low.map(decode_file)
    Y_hi = Y_hi.map(decode_file)

    ds_X = tf.data.Dataset.zip( (X_hi, X_low) ).map(lambda X_hi, X_low: {X_hi_name : X_hi, X_low_name : X_low } )
    return tf.data.Dataset.zip( (ds_X, Y_hi) )

    ds_train = create_full_dataset_2(ds_train_hi, ds_train_lo, ds_target_tr)
    ds_valid = create_full_dataset_2(ds_valid_hi, ds_valid_lo, ds_target_va)
    ds_train = ds_train.batch(int(config['training']['dataset']['batch_size']))
    ds_valid = ds_valid.batch(int(config['training']['dataset']['batch_size']))
    model = Sentinel2Model(
            scaling_factor = config['training']['model']['scaling_factor'],
            filter_size = config['training']['model']['filter_size'],
            num_blocks = config['training']['model']['num_blocks'],
            interpolation = config['training']['model']['interpolation']
    history = model.fit(
            epochs = epochs,
            verbose = 1,
            callbacks = callbacks,
            steps_per_epoch = steps_per_epoch,
            validation_data =  ds_valid,
            validation_steps = config['training']['fit']['validation_steps'] )

That eventually gets passed to a model.fit after the model is built, compiled, and given an optimizer. All that is fine. Here is the error that I get:

Incompatible shapes: [2,256,256,1] vs. [2,64,64,1]

…Note that the concatenate command comes AFTER the Upsampling. I am very confused. I have tried issuing a resize command inside another preprocessing function to map the low-input to the high-input size, but that fails as well.

What do I do? Why is it telling me that the sizes are incompatible? Any help would be appreciated.

Here is the rest of the error:

File "home/dryglicki/code/DSen2/training/quick_training.py", line 192, in <module>
File "home/dryglicki/code/DSen2/training/quick_training.py", line 180, in main
  history = model.fit(
File "data3/anaconda3-2022_05/envs/forecast-AI/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
  return fn(*args, **kwargs)
File "data3/anaconda3-2022_05/envs/forecast-AI/lib/python3.10/site-packages/keras/engine/training.py", line 1409, in fit
  tmp_logs = self.train_function(iterator)
File "data3/anaconda3-2022_05/envs/forecast-AI/lib/python3.10/site-packages/keras/engine/training.py", line 1051, in train_function
  return step_function(self, iterator)
File "data3/anaconda3-2022_05/envs/forecast-AI/lib/python3.10/site-packages/keras/engine/training.py", line 1040, in step_function
  outputs = model.distribute_strategy.run(run_step, args=(data,))
File "data3/anaconda3-2022_05/envs/forecast-AI/lib/python3.10/site-packages/keras/engine/training.py", line 1030, in run_step
  outputs = model.train_step(data)
File "data3/anaconda3-2022_05/envs/forecast-AI/lib/python3.10/site-packages/keras/engine/training.py", line 893, in train_step
  self.optimizer.minimize(loss, self.trainable_variables, tape=tape)
File "data3/anaconda3-2022_05/envs/forecast-AI/lib/python3.10/site-packages/keras/optimizers/optimizer_v2/optimizer_v2.py", line 537, in minimize
  grads_and_vars = self._compute_gradients(
File "data3/anaconda3-2022_05/envs/forecast-AI/lib/python3.10/site-packages/keras/optimizers/optimizer_v2/optimizer_v2.py", line 590, in _compute_gradients
  grads_and_vars = self._get_gradients(tape, loss, var_list, grad_loss)
File "data3/anaconda3-2022_05/envs/forecast-AI/lib/python3.10/site-packages/keras/optimizers/optimizer_v2/optimizer_v2.py", line 471, in _get_gradients
  grads = tape.gradient(loss, var_list, grad_loss)

FCN is a network that does not contain any “Dense” layers (as in traditional CNNs) instead it contains 1x1 convolutions that perform the task of fully connected layers (Dense layers).

I eventually figured it out. Dumb user error. I was trying to add the low-res second input instead of the up-rezzed version of it. That solved the issue. Apologies for the confusion.

        x = Add(name='addition_final')([x, up_low])