simonlevine/bioclinical-roberta-long

You'll need to instantiate a special RoBERTa class . Though technically a "Longformer", the elongated model will still need to be pulled in as such . To do so, use the following classes: RobertaLongSelfAttention(RobertaForMaskedLM) or RobertaForSequenceClassification . Note you may get untrained weights warnings from the model as usual . You can replace it with a different task-specific class from Huggingface, such as Roberta forSequence Classification . For example, replace the 'modeling_bert.Bert' object with a 'LongformerSelf' object instead of a 'longformer' object . For,