amazon/bort

Amazon's BORT is a highly compressed version of bert-large that is up to 10 times faster at inference . It is also 7.9x faster on a CPU, as well as being better performing than other compressed variants of the architecture . Bort is able to be pretrained in 288 GPU hours, which is 1.2% of the time required to pretrain the highest-performing BERT parametric architectural variant, RoBERTa-large (Liu et al., 2019), and about 33% of that of the world-record required to train BERT-large on the same hardware . It obtains performance improvements of between 0.3% and 31%, absolute,,