google/t5-v1_1-large

Google's T5 Version 1.1 includes the following improvements compared to the original T5 model- GEGLU activation in feed-forward activation, rather than ReLU - see here . The model shapes are a bit different - larger d_model and smaller num_heads and d_ff . Dropout was turned off in pre-training (turning off) Dropout should be re-enabled during fine-tuning . To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code . By combining the insights from our exploration with scale and our new “Colossal Clean Corpus”, we achieve state-of-the,