squeezebert/squeezebert-uncased

SqueezeBERT is a pretrained model for the English language using a masked language modeling (MLM) and Sentence Order Prediction (SOP) objective . The model architecture is similar to BERT-base-uncased, but with the pointwise fully-connected layers replaced with grouped convolutions . SqueezBERT has been pretrained but not finetuned . For most text classification tasks, we recommend using squeezebert-mnli-headless as a starting point . Finetuning SqueezzBERT with distillation is not yet implemented in this repo . If the author (Forrest Iandola - forrest.dnn@gmail.com),