dmitry-vorobiev/rubert_ria_headlines

The model was trained on the first 99% of "Rossiya Segodnya" news dataset for 2 epochs . It was trained for 3 epochs with effective batch size 192 and soft restarts (warmup steps 1500 / 500 / 500 with new optimizer state on each epoch start). The model has been trained for three epochs using free TPUv3 on kaggle. The model is trained for the first 1% of the ria dataset. It has been validated using gazeta_ru test split. It was used to train the model on the last 1/2% of ria news dataset. The data set is based on the DeepPavlov,