pradhyra/AWSBlogBert

The input text contains around 3000 blog articles on AWS Blogs . The model acheived a training loss of 3.6 on the MLM task over 10 epochs . I then followed HuggingFace's Transformers blog post to train the model . I chose to follow the following training set-up: 28k training steps with batches of 64 sequences of length 512 with an initial learning rate 5e-5.6 . The training process was followed by a series of batches of 512 sequences of text length of 512 with a learning rate of 5e.6 and a learning loss rate of 3e-4.6 over the time period of training . It is pre-trained on blog articles,