Augmented Large Language Models for Software Engineering
(2024) INTM01 20241Innovation Engineering
- Abstract
- Since the release of the recent generation of Large Language Models (LLMs) such as GPT-4 (GPT standing for generative pre-trained transformer), companies and individuals are exploring opportunities to improve their effectiveness by automating or receiving assistance with knowledge-intensive tasks by using artificial intelligence (AI). Within software engineering, a plethora of commercial services promising impressive results has reached the market in the past year. This thesis explores the practical implications of applying LLMs in this software engineering context. Through a case study at Craft Docs, a software development company, we investigated the current use of LLMs and identified specific opportunities to leverage these models as a... (More)
- Since the release of the recent generation of Large Language Models (LLMs) such as GPT-4 (GPT standing for generative pre-trained transformer), companies and individuals are exploring opportunities to improve their effectiveness by automating or receiving assistance with knowledge-intensive tasks by using artificial intelligence (AI). Within software engineering, a plethora of commercial services promising impressive results has reached the market in the past year. This thesis explores the practical implications of applying LLMs in this software engineering context. Through a case study at Craft Docs, a software development company, we investigated the current use of LLMs and identified specific opportunities to leverage these models as a way of improving the efficacy of software engineers.
Action research (AR) was chosen as the research methodology, carried out by devel- oping a prototype that was deployed and evaluated at the company. A literature study and interviews surveying the use of and sentiment towards LLMs at the case company guided the development, and the prototype was evaluated with practical tests and retrospective interviews.
Findings from the project suggest that while LLMs can enhance certain aspects of software engineering, their effectiveness is highly dependent on the context of use and the specific tasks they are designed to support. The prototype developed in this study demonstrates a practical implementation of retrieval-augmented generation (RAG) to provide additional context to LLMs, showing both the strengths and limitations of these technologies in real-world settings. Most apparent was that even with access to relevant code context, tasks given to an LLM need to be carefully scoped to yield effective results. (Less) - Popular Abstract
- The rise of new advanced generative AI models like ChatGPT has been a game changer for how companies work, presenting a large range of opportunities. Programming and software development is one of the areas where high hopes have emerged, with prominent CEOs stating that English will be the new programming language. However, beyond the hype, a bunch of challenges have to be addressed to actually make these models useful – some technical, others regarding how we interact with this technology.
Please use this url to cite or link to this publication:
http://lup.lub.lu.se/student-papers/record/9162888
- author
- Rosén, Fabian LU and Rübner, Moritz LU
- supervisor
- organization
- alternative title
- Anpassade stora språkmodeller för mjukvaruutveckling
- course
- INTM01 20241
- year
- 2024
- type
- H2 - Master's Degree (Two Years)
- subject
- keywords
- ai, artificial intelligence, large language models, llm, retrieval-augmented generation, rag, software engineering, code, programming, artificiell intelligens, stora språkmodeller, programvaruutveckling, kod, programmering
- language
- English
- id
- 9162888
- date added to LUP
- 2024-06-20 09:24:45
- date last changed
- 2024-06-20 09:24:45
@misc{9162888,
abstract = {{Since the release of the recent generation of Large Language Models (LLMs) such as GPT-4 (GPT standing for generative pre-trained transformer), companies and individuals are exploring opportunities to improve their effectiveness by automating or receiving assistance with knowledge-intensive tasks by using artificial intelligence (AI). Within software engineering, a plethora of commercial services promising impressive results has reached the market in the past year. This thesis explores the practical implications of applying LLMs in this software engineering context. Through a case study at Craft Docs, a software development company, we investigated the current use of LLMs and identified specific opportunities to leverage these models as a way of improving the efficacy of software engineers.
Action research (AR) was chosen as the research methodology, carried out by devel- oping a prototype that was deployed and evaluated at the company. A literature study and interviews surveying the use of and sentiment towards LLMs at the case company guided the development, and the prototype was evaluated with practical tests and retrospective interviews.
Findings from the project suggest that while LLMs can enhance certain aspects of software engineering, their effectiveness is highly dependent on the context of use and the specific tasks they are designed to support. The prototype developed in this study demonstrates a practical implementation of retrieval-augmented generation (RAG) to provide additional context to LLMs, showing both the strengths and limitations of these technologies in real-world settings. Most apparent was that even with access to relevant code context, tasks given to an LLM need to be carefully scoped to yield effective results.}},
author = {{Rosén, Fabian and Rübner, Moritz}},
language = {{eng}},
note = {{Student Paper}},
title = {{Augmented Large Language Models for Software Engineering}},
year = {{2024}},
}