Wikidepia/IndoConvBERT-small

The current version of the model is trained on Indo4B and small Twitter dump . We pre-train the model with 512 sequence length for 1M steps on a v3-8 TPU . We follow a different training procedure: instead of using a two-phase approach, that pre-trains the model for 90% with 128 sequence length and 10% with 256 sequence length, we use a pre-training procedure on a TPU with 512 sequences . The model is currently being used to train on various corpus, such as small Twitter dumps, Indo4b and Indo5B . The training procedure is now being used on a free TPU using TensorFlow Research Cloud . The,