SEBIS/legal_t5_small_trans_de_sv

The legal_t5_small_trans_de_sv model is based on the t5-small model and was trained on a large corpus of parallel text . It scales the baseline model of t5 down by using dmodel = 512, dff = 2,048, 8-headed attention, and only 6 layers each in the encoder and decoder. This variant has about 60 million parameters. The model could be used for translation of legal texts from Deustch to Swedish. It has a total of approximately 220M parameters . The optimizer used is AdaFactor with inverse square root learning rate schedule for pre-training. An unigram model trained with 88M lines,