sebastian-hofstaetter/distilbert-dot-margin_mse-T2-msmarco

We provide a retrieval trained DistilBert-based model (we call the architecture BERT_Dot) Our model is trained with Margin-MSE using a 3 teacher BERT scoring ensemble on MSMARCO-Passage . The architecture is a 6-layer DistilBERT, without architecture additions or modifications (we only change the weights during training) To receive a query/passage representation we pool the CLS vector . We use the same BERT layers for both query and passage encoding (yields better results, and lowers memory requirements). This instance can be used to re-rank a candidate set or directly for a vector index based dense retrieval . For re-,