66 lines
3.3 KiB
Plaintext
66 lines
3.3 KiB
Plaintext
Skip to main content
|
||
Computer Science > Machine Learning
|
||
[Submitted on 7 Jul 2021 (v1), last revised 14 Jul 2021 (this version, v2)]
|
||
Evaluating Large Language Models Trained on Code
|
||
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, Wojciech Zaremba
|
||
We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.
|
||
Comments: corrected typos, added references, added authors, added acknowledgements
|
||
Subjects: Machine Learning (cs.LG)
|
||
Cite as: arXiv:2107.03374 [cs.LG]
|
||
(or arXiv:2107.03374v2 [cs.LG] for this version)
|
||
|
||
https://doi.org/10.48550/arXiv.2107.03374
|
||
Focus to learn more
|
||
Submission history
|
||
From: Mark Chen [view email]
|
||
[v1] Wed, 7 Jul 2021 17:41:24 UTC (1,466 KB)
|
||
[v2] Wed, 14 Jul 2021 17:16:02 UTC (1,467 KB)
|
||
|
||
Access Paper:
|
||
View PDFTeX Source
|
||
view license
|
||
Current browse context: cs.LG
|
||
< prev next >
|
||
|
||
newrecent2021-07
|
||
Change to browse by: cs
|
||
References & Citations
|
||
NASA ADS
|
||
Google Scholar
|
||
Semantic Scholar
|
||
4 blog links (what is this?)
|
||
DBLP - CS Bibliography
|
||
listing | bibtex
|
||
Heewoo Jun
|
||
Jared Kaplan
|
||
Harrison Edwards
|
||
Yuri Burda
|
||
Greg Brockman
|
||
…
|
||
Export BibTeX Citation
|
||
Bookmark
|
||
Bibliographic Tools
|
||
Bibliographic and Citation Tools
|
||
Bibliographic Explorer Toggle
|
||
Bibliographic Explorer (What is the Explorer?)
|
||
Connected Papers Toggle
|
||
Connected Papers (What is Connected Papers?)
|
||
Litmaps Toggle
|
||
Litmaps (What is Litmaps?)
|
||
scite.ai Toggle
|
||
scite Smart Citations (What are Smart Citations?)
|
||
Code, Data, Media
|
||
Demos
|
||
Related Papers
|
||
About arXivLabs
|
||
Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
|
||
About
|
||
Help
|
||
Contact
|
||
Subscribe
|
||
Copyright
|
||
Privacy Policy
|
||
Web Accessibility Assistance
|
||
|
||
arXiv Operational Status
|