Masking with Attention layer

Hello everyone,

I’ve been running an implementation of Transformers with keras functional API and I got an issue on this line:

attention = Attention()([query, value, key], mask=[query._keras_mask, value._keras_mask])

It triggers the following error:
TypeError: Cannot convert a symbolic Keras input/output to a numpy array.
After a look at the documentation, it seems that the passed tensors are of correct format. What’s the problem with this?

Thanks in advance,
Kevin

@Kevin_Guimard

Welcome to the Tensorflow Forum!

Can you try to pass the inputs to the Attention layer as shown below and let us know?

attention = Attention()(query, value, key, mask=[query._keras_mask, value._keras_mask])

Thank you!

For an unknown reason, I now get this error:

AttributeError: ‘Tensor’ object has no attribute ‘_keras_mask’

Though, the query and the value objects are Dense layers applied to inputs, they should be KerasTensors that hold the _keras_mask field.

Could you please share the standalone code to debug your issue?