Assertion failure when there are more than 255 tokenized data files (assert num_datasets < 255 in blendable_dataset.py)

[Jeronymous]

I am hitting this assertion:

  File ".../Megatron-DeepSpeed/megatron/data/blendable_dataset.py", line 36, in _build_indices
    assert num_datasets < 255

I am then wondering why something like what is done in this PR was not merged?

-            assert num_datasets < 255
-            dataset_index = np.zeros(self.size, dtype=np.uint8)
+            # assert num_datasets < 255
+            dataset_index = np.zeros(self.size, dtype=np.int64)

Is there a performance issues if the user have thousands of tokenized files (instead of a few dozens)?

Also, to workaround this limitation, is there a script somewhere to concatenate several ".bin" and ".idx" files (with tokenized text inside)?