Benchmarking Large Language Models for Vulnerability Detection: Comparing Local and Cloud LLMs

Abstract

The number and complexity of modern software systems have increased substantially since their inception. This unrelenting growth is making manual code review more difficult year by year, and the gap between the amount of code and security verification creates a critical need for automated tools that can assist developers in detecting vulnerabilities. A possible solution to this is large language model driven vulnerability detection.

This thesis investigates the possibility of utilizing locally fine-tuned LLMs in order to discover and flag memory related security flaws in C/C++ code. Five locally fine-tuned models have been examined and compared to each other, their non-fine-tuned versions, as well as proprietary cloud models. The models were fed functions taken from C/C++ projects, and were asked to determine whether the function in question was vulnerable. Two different prompting methods were used during the evaluation, zero-shot prompting and few-shot prompting. After each evaluation, performance metrics such as accuracy and F1-score were calculated.

While fine-tuning enhanced the performances of the local models with respect to F1-score, their ability to detect vulnerabilities remained unsatisfactory. The highest performing model, CodeLlama 7B, achieved a F1-score of only 0.12. Cloud models are orders of magnitude larger in parameter size and have had more extensive pre-training. However, as the cloud models did not outperform this, it indicates that the methods utilized in the thesis were sub-optimal. The two prompting methods did not significantly impact the results of any model. Further research to improve model performance may include Chain-of-Thought prompting, Retrieval-Augmented Generation, or fine-tuning of the cloud models.

Popular Abstract

With the rising usage of software in the world, the attack surface has increased in tandem. Therefore, the importance of secure software has grown. This thesis investigates how locally fine-tuned large language models (LLMs) compare to cloud-based models for vulnerability detection in C/C++ code. Can a local, fine-tuned LLM specialized in vulnerability detection outperform a cloud-based LLM?

In an era where software governs everything from your social media to national security, a single coding error can have catastrophic consequences. From the infamous Heartbleed Bug to recent global IT disruptions, the message is clear: we need better ways to find and fix vulnerabilities before they are exploited. Traditionally, this required intensive manual code review, but a new generation of artificial intelligence is emerging.

This study investigated an important topic for the future of cybersecurity. It compared larger cloud LLMs such as ChatGPT against smaller, open-source models such as Mistral. While cloud models have immense processing power, local models offer a significant advantage: data privacy. For a cybersecurity firm or a government agency, sending sensitive, proprietary code to a third-party cloud provider is often a deal-breaker. But in order to match the capabilities of the enormous cloud models, the local LLMs need to be specialized into advanced vulnerability detection models. This was achieved through a powerful tool: Fine-tuning.


Large language models can have different sizes, which is usually measured in the parameter count of the model. If one wants to give the relatively small, local, billion parameter-LLMs a fighting chance against the trillion parameter-cloud LLMs, the local models need to be specialized. This is achieved by way of presenting the local models with example tasks, and gradually updating the parameters of said models to enhance their performance. This process is known as fine-tuning.

The outcome of the battle was unexpected: one of the locally fine-tuned models actually outperformed the cloud giants. The local LLM CodeLlama 7B emerged as the top performer, achieving the highest F1-score of all models. The cloud models, despite their size, struggled with the tasks presented before them. Furthermore, fine-tuning provided the local LLMs with essential knowledge of the formatting of the desired output, as the local models were unable to structure their answers correctly without it.

While locally fine-tuned models won, the study found that they are not quite ready for real world application within cybersecurity. Their overall score remained low, and they showed a bias toward assuming code was safe.

• author: Pykälistö, Alexandra and Müller-Uri, Karl
• supervisor: Christian Gehrmann; Marcus Kindberg; Victor Arvidsson
• organization: Department of Electrical and Information Technology
• alternative title: Utvärdering av stora språkmodeller för detektering av sårbarheter: En jämförelse mellan lokala och molnbaserade LLM:er
• course: EITM01 20261
• year: 2026
• type: H2 - Master's Degree (Two Years)
• subject: Technology and Engineering
• report number: LU/LTH-EIT 2026-1122
• language: English
• id: 9229367
• date added to LUP: 2026-06-03 08:50:53
• date last changed: 2026-06-03 08:50:53