Wikidepia/IndoConvBERT-medium-small

The current version of the model is trained on Indo4B and small Twitter dump . We pre-train the model with 512 sequence length for 1M steps on a v3-8 TPU . We follow a different training procedure: instead of using a two-phase approach, that pre-trains the model for 90% with 128 sequence length and 10% with 256 sequence length, we use a 1M step-step training procedure instead of a two phase approach . We use a free TPU for training on a free version of TFRC (TensorFlow Research Cloud) Big thanks to TFRC for providing free tPU <3.0 for training the model on the model .,