56 lines
2.5 KiB
Plaintext
56 lines
2.5 KiB
Plaintext
Skip to main content
|
||
Computer Science > Computation and Language
|
||
[Submitted on 18 Dec 2024 (v1), last revised 19 Dec 2024 (this version, v2)]
|
||
Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference
|
||
Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, Iacopo Poli
|
||
Encoder-only transformer models such as BERT offer a great performance-size tradeoff for retrieval and classification tasks with respect to larger decoder-only models. Despite being the workhorse of numerous production pipelines, there have been limited Pareto improvements to BERT since its release. In this paper, we introduce ModernBERT, bringing modern model optimizations to encoder-only models and representing a major Pareto improvement over older encoders. Trained on 2 trillion tokens with a native 8192 sequence length, ModernBERT models exhibit state-of-the-art results on a large pool of evaluations encompassing diverse classification tasks and both single and multi-vector retrieval on different domains (including code). In addition to strong downstream performance, ModernBERT is also the most speed and memory efficient encoder and is designed for inference on common GPUs.
|
||
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
|
||
Cite as: arXiv:2412.13663 [cs.CL]
|
||
(or arXiv:2412.13663v2 [cs.CL] for this version)
|
||
|
||
https://doi.org/10.48550/arXiv.2412.13663
|
||
Focus to learn more
|
||
Submission history
|
||
From: Benjamin Clavié [view email]
|
||
[v1] Wed, 18 Dec 2024 09:39:44 UTC (81 KB)
|
||
[v2] Thu, 19 Dec 2024 06:32:26 UTC (81 KB)
|
||
|
||
Access Paper:
|
||
View PDFHTML (experimental)TeX Source
|
||
view license
|
||
Current browse context: cs.CL
|
||
< prev next >
|
||
|
||
newrecent2024-12
|
||
Change to browse by: cs cs.AI
|
||
References & Citations
|
||
NASA ADS
|
||
Google Scholar
|
||
Semantic Scholar
|
||
Export BibTeX Citation
|
||
Bookmark
|
||
Bibliographic Tools
|
||
Bibliographic and Citation Tools
|
||
Bibliographic Explorer Toggle
|
||
Bibliographic Explorer (What is the Explorer?)
|
||
Connected Papers Toggle
|
||
Connected Papers (What is Connected Papers?)
|
||
Litmaps Toggle
|
||
Litmaps (What is Litmaps?)
|
||
scite.ai Toggle
|
||
scite Smart Citations (What are Smart Citations?)
|
||
Code, Data, Media
|
||
Demos
|
||
Related Papers
|
||
About arXivLabs
|
||
Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
|
||
About
|
||
Help
|
||
Contact
|
||
Subscribe
|
||
Copyright
|
||
Privacy Policy
|
||
Web Accessibility Assistance
|
||
|
||
arXiv Operational Status
|