58 lines
3.0 KiB
Plaintext
58 lines
3.0 KiB
Plaintext
Skip to main content
|
||
Computer Science > Artificial Intelligence
|
||
[Submitted on 12 Jul 2025 (v1), last revised 25 Jul 2025 (this version, v2)]
|
||
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
|
||
Joel Becker, Nate Rush, Elizabeth Barnes, David Rein
|
||
Despite widespread adoption, the impact of AI tools on software development in the wild remains understudied. We conduct a randomized controlled trial (RCT) to understand how AI tools at the February-June 2025 frontier affect the productivity of experienced open-source developers. 16 developers with moderate AI experience complete 246 tasks in mature projects on which they have an average of 5 years of prior experience. Each task is randomly assigned to allow or disallow usage of early 2025 AI tools. When AI tools are allowed, developers primarily use Cursor Pro, a popular code editor, and Claude 3.5/3.7 Sonnet. Before starting tasks, developers forecast that allowing AI will reduce completion time by 24%. After completing the study, developers estimate that allowing AI reduced completion time by 20%. Surprisingly, we find that allowing AI actually increases completion time by 19%--AI tooling slowed developers down. This slowdown also contradicts predictions from experts in economics (39% shorter) and ML (38% shorter). To understand this result, we collect and evaluate evidence for 20 properties of our setting that a priori could contribute to the observed slowdown effect--for example, the size and quality standards of projects, or prior developer experience with AI tooling. Although the influence of experimental artifacts cannot be entirely ruled out, the robustness of the slowdown effect across our analyses suggests it is unlikely to primarily be a function of our experimental design.
|
||
Comments: 51 pages, 8 tables, 22 figures
|
||
Subjects: Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC); Software Engineering (cs.SE)
|
||
ACM classes: I.2
|
||
Cite as: arXiv:2507.09089 [cs.AI]
|
||
(or arXiv:2507.09089v2 [cs.AI] for this version)
|
||
|
||
https://doi.org/10.48550/arXiv.2507.09089
|
||
Focus to learn more
|
||
Submission history
|
||
From: Nate Rush [view email]
|
||
[v1] Sat, 12 Jul 2025 00:16:33 UTC (15,206 KB)
|
||
[v2] Fri, 25 Jul 2025 00:43:07 UTC (5,596 KB)
|
||
|
||
Access Paper:
|
||
View PDFTeX Source
|
||
view license
|
||
Current browse context: cs.AI
|
||
< prev next >
|
||
|
||
newrecent2025-07
|
||
Change to browse by: cs cs.HC cs.SE
|
||
References & Citations
|
||
NASA ADS
|
||
Google Scholar
|
||
Semantic Scholar
|
||
Export BibTeX Citation
|
||
Bookmark
|
||
Bibliographic Tools
|
||
Bibliographic and Citation Tools
|
||
Bibliographic Explorer Toggle
|
||
Bibliographic Explorer (What is the Explorer?)
|
||
Connected Papers Toggle
|
||
Connected Papers (What is Connected Papers?)
|
||
Litmaps Toggle
|
||
Litmaps (What is Litmaps?)
|
||
scite.ai Toggle
|
||
scite Smart Citations (What are Smart Citations?)
|
||
Code, Data, Media
|
||
Demos
|
||
Related Papers
|
||
About arXivLabs
|
||
Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
|
||
About
|
||
Help
|
||
Contact
|
||
Subscribe
|
||
Copyright
|
||
Privacy Policy
|
||
Web Accessibility Assistance
|
||
|
||
arXiv Operational Status
|