Assume max seq length $=10,$ there are multiple samples, such as

$[1,1,1,1]$ length of $4$

$[2,2,2,2,2]$length of $5$

During actual pretraining, the two will be spliced together as one sample for training, for example:

$[1,1,1,1,$EOS$\_$token,$2,2,2,2,2,$EOS$\_$token$]$ In this way, for sample $2,$ it can actually see the token of sample $1.$ Is there any reason for this?