huawei-noah/DynaBERT_MNLI

DynaBERT: Dynamic BERT with Adaptive Width and Depth can flexibly adjust the size and latency by selecting adaptive width and depth . The subnetworks of it have competitive performances as other similar-sized compressed models . The code is modified based on the repository developed by Hugging Face: Transformers v2.1.1, and is released in GitHub . The training process includes first training a width-adaptive BERT and then allowing both adaptive width & depth using knowledge distillation. The training is based on knowledge distillations from a knowledge-distillated version of the DynaBerT code. The DynaberT code has been released on GitHub. The code,