LongNet: Scaling Transformers to 1,000,000,000 Tokens

• Code release: https://github.com/microsoft/torchscale
• July 2023: release preprint LongNet: Scaling Transformers to 1,000,000,000 Tokens

<p> <a href="https://github.com/microsoft/torchscale/blob/main/LICENSE"></a> <a href="https://pypi.org/project/torchscale"></a> </p>