mrm8488/electricidad-small-discriminator

Electra is a new method for self-supervised language representation learning . It can be used to pre-train transformer networks using relatively little compute . Electra like model (discriminator) trained on a + 20 GB of the OSCAR Spanish corpus . At large scale, ELECTRA achieves state-of-the-art results on the SQuAD 2.0 dataset . The Electra model is trained to distinguish "real" input tokens vs "fake" input token generated by another neural network . It is similar to the discriminator of a GAN . At small scale, Electra achieves strong results even when trained on the single GPU . I thank Google for helping me with,