How to solve custom loss error in tensoflow?

Keras
,
#1

Hello, I just signed up for the first time today to ask about my problem. I posted this question elsewhere yesterday, but haven’t gotten an answer yet https://stackoverflow.com/questions/77814314/how-to-solve-custom-loss-error-in-tensoflow

I am currently designing simple object detector using tensorflow,but I am facing to custom loss problems. And current dev environment is windows 10 native, and tensorflow-gpu=2.10.1.

def calc_iou_2D(set1: np.array, set2: np.array):

    assert len(set1.shape) == len(set2.shape) == 2

    SMOOTH = tf.cast(tf.constant(1e-6), dtype=tf.float32)
    iou_2D = []
    tf.print('11111111111111111111111')
    for item in set1:
        
        tf.print('item', item)
        tf.print('2222222222222222222222222222')
        x0_inter = tf.math.maximum(item[0], set2[:,0])
        y0_inter = tf.math.maximum(item[1], set2[:,1])
        x1_inter = tf.math.minimum(item[2], set2[:,2])
        y1_inter = tf.math.minimum(item[3], set2[:,3])

        intersection = tf.clip_by_value((x1_inter - x0_inter), 0, 1) * tf.clip_by_value((y1_inter - y0_inter), 0, 1)

        area_A = (item[2] - item[0]) * (item[3] - item[1])
        area_B = (set2[:,2] - set2[:,0]) * (set2[:,3] - set2[:,1])
        union = area_A + area_B - intersection
        tf.print('333333333333333333333')

        iou = (intersection + SMOOTH) / (union + SMOOTH)
        tf.print('44444444444444444444444')
        iou_2D.append((iou))
        tf.print('5555555555555555555555')

    return tf.transpose(tf.convert_to_tensor(iou_2D))


def test(dataset, anchor_boxes_xy):
    gt_boxes = np.array([[0.264     , 0.30960854, 0.392     , 0.4341637 ],
                   [0.206     , 0.27402135, 0.748     , 0.64768683],
                   [0.05      , 0.66903915, 0.086     , 0.84341637],
                   [0.388     , 0.63701068, 0.424     , 0.8113879 ]])
    gt_boxes = tf.convert_to_tensor(gt_boxes.astype('float32'))

    iou = calc_iou_2D(gt_boxes, anchor_boxes_xy)
    value, index = argmax(iou, axis=0)

class ssd_loss(tf.keras.losses.Loss):

    def __init__(self, anchorboxes_xy, name="ssd_loss"):
        super(ssd_loss, self).__init__(name=name)
        self.anchorboxes_xy = tf.convert_to_tensor(np.float32(anchorboxes_xy))
    
    def call(self, y_true, y_pred):
        y_true_cls_batch, y_true_boxes_batch = y_true[:, :, :1], y_true[:, :, 1:]
        y_pred_cls_batch, y_pred_boxes_batch = y_pred[:, :, :param['n_classes']], y_pred[:, :, param['n_classes']:]

        for i in range(param['batch_size']):
            iou = calc_iou_2D(y_true_boxes_batch[i], self.anchorboxes_xy)
            value, index = argmax(iou, axis=0)
            tf.print('value', value)
        
        loss_cls = tf.keras.losses.SparseCategoricalCrossentropy()(y_true_cls_batch, y_pred_cls_batch)
        loss_box = tf.keras.losses.MeanSquaredError()(y_true_boxes_batch, y_pred_boxes_batch)

        return loss_cls + loss_box

calc_iou_2D is working well inside test function but it does not work in ssd_loss function as follows.

H = model.fit(train_ds, batch_size = param['batch_size'], epochs = param['epochs'], verbose=1)
(Pdb) n
11111111111111111111111
item [0.206 0.274021357 0.748 0.647686839]
2222222222222222222222222222
333333333333333333333  
44444444444444444444444
5555555555555555555555 
item [0.264 0.309608549 0.392 0.43416369]
2222222222222222222222222222
333333333333333333333
44444444444444444444444
5555555555555555555555
item [0.388 0.637010694 0.424 0.811387897]
2222222222222222222222222222
333333333333333333333
44444444444444444444444
5555555555555555555555
item [0.05 0.66903913 0.086 0.843416393]
2222222222222222222222222222
333333333333333333333
44444444444444444444444
5555555555555555555555
11111111111111111111111
item [0.355967075 0.2 0.716049373 0.7]
2222222222222222222222222222
333333333333333333333
44444444444444444444444
5555555555555555555555
11111111111111111111111
item [0.016 0.289617479 0.996 0.715847]
2222222222222222222222222222
333333333333333333333
44444444444444444444444
5555555555555555555555
item [0.84 0.543715835 0.962 0.614754081]
2222222222222222222222222222
333333333333333333333
44444444444444444444444
5555555555555555555555
item [0.648 0.510929 0.82 0.606557369]
2222222222222222222222222222
333333333333333333333
44444444444444444444444
5555555555555555555555
tensorflow.python.framework.errors_impl.InaccessibleTensorError: in user code:

    File "C:\Users\S-WONJU\AppData\Local\Programs\Python\Python310\lib\site-packages\keras\engine\training.py", line 1160, in train_function  *
        return step_function(self, iterator)
    File "H:\object_detection\SSD\my_ssd\basic_OD_1.py", line 532, in call  *
        iou = calc_iou_2D(y_true_boxes_batch[i], self.anchorboxes_xy)
    File "H:\object_detection\SSD\my_ssd\basic_OD_1.py", line 142, in calc_iou_2D  *
        return tf.transpose(tf.convert_to_tensor(iou_2D))

    InaccessibleTensorError: <tf.Tensor 'ssd_loss/while/truediv:0' shape=(49,) dtype=float32> is out of scope and cannot be used here. Use return values, explicit Python locals or TensorFlow collections to access it.
    Please see https://www.tensorflow.org/guide/function#all_outputs_of_a_tffunction_must_be_return_values for more information.  

    <tf.Tensor 'ssd_loss/while/truediv:0' shape=(49,) dtype=float32> was defined here:
        File "H:\object_detection\SSD\my_ssd\basic_OD_1.py", line 817, in <module>
          train()
        File "H:\object_detection\SSD\my_ssd\basic_OD_1.py", line 622, in train
          H = model.fit(train_ds, batch_size = param['batch_size'], epochs = param['epochs'], verbose=1)
        File "C:\Users\S-WONJU\AppData\Local\Programs\Python\Python310\lib\site-packages\keras\utils\traceback_utils.py", line 65, in error_handler
          return fn(*args, **kwargs)
        File "C:\Users\S-WONJU\AppData\Local\Programs\Python\Python310\lib\site-packages\keras\engine\training.py", line 1564, in 
fit
          tmp_logs = self.train_function(iterator)
        File "C:\Users\S-WONJU\AppData\Local\Programs\Python\Python310\lib\site-packages\keras\engine\training.py", line 1160, in 
train_function
          return step_function(self, iterator)
        File "C:\Users\S-WONJU\AppData\Local\Programs\Python\Python310\lib\site-packages\keras\engine\training.py", line 1146, in 
step_function
          outputs = model.distribute_strategy.run(run_step, args=(data,))
        File "C:\Users\S-WONJU\AppData\Local\Programs\Python\Python310\lib\site-packages\keras\engine\training.py", line 1135, in 
run_step
          outputs = model.train_step(data)
        File "C:\Users\S-WONJU\AppData\Local\Programs\Python\Python310\lib\site-packages\keras\engine\training.py", line 994, in train_step
          loss = self.compute_loss(x, y, y_pred, sample_weight)
        File "C:\Users\S-WONJU\AppData\Local\Programs\Python\Python310\lib\site-packages\keras\engine\training.py", line 1052, in 
compute_loss
          return self.compiled_loss(
        File "C:\Users\S-WONJU\AppData\Local\Programs\Python\Python310\lib\site-packages\keras\engine\compile_utils.py", line 265, in __call__
          loss_value = loss_obj(y_t, y_p, sample_weight=sw)
        File "C:\Users\S-WONJU\AppData\Local\Programs\Python\Python310\lib\site-packages\keras\losses.py", line 152, in __call__  
          losses = call_fn(y_true, y_pred)
        File "H:\object_detection\SSD\my_ssd\basic_OD_1.py", line 531, in call
          for i in range(param['batch_size']):
        File "H:\object_detection\SSD\my_ssd\basic_OD_1.py", line 532, in call
          iou = calc_iou_2D(y_true_boxes_batch[i], self.anchorboxes_xy)
        File "H:\object_detection\SSD\my_ssd\basic_OD_1.py", line 119, in calc_iou_2D
          for item in set1:
        File "H:\object_detection\SSD\my_ssd\basic_OD_1.py", line 137, in calc_iou_2D
          iou = (intersection + SMOOTH) / (union + SMOOTH)

    The tensor <tf.Tensor 'ssd_loss/while/truediv:0' shape=(49,) dtype=float32> cannot be accessed from FuncGraph(name=train_function, id=2585060633536), because it was defined in FuncGraph(name=ssd_loss_while_body_3411, id=2585079461824), which is out of scope.
> h:\object_detection\ssd\my_ssd\basic_od_1.py(622)train()
-> H = model.fit(train_ds, batch_size = param['batch_size'], epochs = param['epochs'], verbose=1)

Based on print results, first batch data is well used, but i guess second batch is not accessible.

I just want to run calc_iou_2D inside the loss function. I don’t know how to solve this problem, please help me.

#3

Hi @Sangmin_Suh ,

The error arises because the calc_iou_2D function creates a tensor (tf.Tensor 'ssd_loss/while/truediv:0' ) within a loop inside the ssd_loss function. This tensor is not accessible outside the loop’s scope, leading to the error.

Return Tensors from the Loop:

  • Modify calc_iou_2D to return a list of IoU tensors, one for each iteration and Collect the results in a list outside the loop in ssd_loss, This could solve the problem.

or You can modify your code to make use of TensorFlow operations directly instead of using NumPy operations inside your custom loss function. Additionally, using tf.TensorArray might help to collect the intermediate results within a loop.

Thanks.

2 Likes
#5

As you suggested, I removed the loop from the function and implemented it in a different way (vectorized code) and the error disappeared. Thanks.