I am working about MLM model. So i am training tokenizer. and model. But my data is big.(80 gb corpus) so i am training to model part by part . and tokenizer too . I am trained a tokenizer with part 1 of data. But ı cant train update with part 2 of data. what should i do ?
I have been tried a lot of codes. But I cant.
What should i do ?
Thank you for now