krevas/finance-koelectra-small-generator

ELECTRA is a new method for self-supervised language representation learning . It can be used to pre-train transformers networks using relatively little compute . The current version of the model is trained on a financial news data of Naver news . The final training corpus has a size of 25GB and 2.3B tokens . The ELECTRA model was trained a cased model on a TITAN RTX for 500k steps . More details about ELECTRA can be found in the ICLR paper or in the official ELECTRA repository on GitHub . All models are available on the Huggingface model hub, and the model hub is open to all users of the ElectRA repository . For more,