Fitting transformers model for text 2 text generation

General Discussion
,
#1

Hi,

I am looking to fit a transformers model on a large amount of text and then to fine tune it for a specific task using transfer learning. I have looked at this tutorial Modelo transformador para la comprensión del lenguaje.  |  Text  |  TensorFlow

But there seems to be only one “fitting” procedure. Do you know if there is any example on fitting a transformers model first on raw text so that it learns the language and then to fit it on a specific task through transfer learning?

Many thanks,

Antoine

#3

You could check KerasNLP tutorial:

1 Like