From Base Model to Maintenance Agent: Designing and Evaluating LLM Assistants for Industrial Settings
(2026) In Master's Theses in Mathematical Sciences FMSM01 20261Mathematical Statistics
- Abstract
- This thesis investigates the design and evaluation of Large Language Model (LLM) agent assistants for industrial maintenance, developed in cooperation with SSAB. The study focuses on identifying what combinations of tools and architectures best serve the needs of maintenance technicians, with an emphasis on practical usability and factual correctness.
Multiple agents were developed and compared, incorporating combinations of Retrieval-Augmented Generation (RAG), internet search, direct document upload, and numerical database retrieval from a simulated Computerized Maintenance Management System (CMMS). All agents were built on the ChatGPT-4.1 base model. Evaluation was carried out through LLM-as-a-judge tests and a user survey involving... (More) - This thesis investigates the design and evaluation of Large Language Model (LLM) agent assistants for industrial maintenance, developed in cooperation with SSAB. The study focuses on identifying what combinations of tools and architectures best serve the needs of maintenance technicians, with an emphasis on practical usability and factual correctness.
Multiple agents were developed and compared, incorporating combinations of Retrieval-Augmented Generation (RAG), internet search, direct document upload, and numerical database retrieval from a simulated Computerized Maintenance Management System (CMMS). All agents were built on the ChatGPT-4.1 base model. Evaluation was carried out through LLM-as-a-judge tests and a user survey involving subject matter experts and non-experts, analysed using nonparametric statistical methods and the Total Survey Error framework.
Results indicate that the combination of RAG with preprocessed data, limited internet search, and direct UI document upload yields the most effective performance for maintenance-related queries. Agents equipped with CMMS database retrieval via a Knowledge Base architecture performed poorly, suggesting that LLM-based retrieval logic is insufficient for reliable numerical reasoning in this context. The survey provided no significant evidence that increased functionality degrades the structural quality of agent responses.
In conclusion, this thesis demonstrates that a multi-tool LLM agent can provide meaningful support in an industrial maintenance context, and establishes a foundation for future work, particularly the adoption of a Model Context Protocol-based architecture for more robust numerical data access. (Less)
Please use this url to cite or link to this publication:
https://lup.lub.lu.se/student-papers/record/9231989
- author
- Bengtsson, Gabriel LU and Nilsson, Martin LU
- supervisor
- organization
- course
- FMSM01 20261
- year
- 2026
- type
- H2 - Master's Degree (Two Years)
- subject
- keywords
- Large Language Model, LLM, Artifical Intelligence, AI, AI Agents, LLM Agents, Retrieval Augmented Generation, RAG, Industrial Maintenance, Maintenance
- publication/series
- Master's Theses in Mathematical Sciences
- report number
- LUTFMS-3555-2026
- ISSN
- 1404-6342
- other publication id
- 2026:E48
- language
- English
- id
- 9231989
- date added to LUP
- 2026-06-08 14:21:36
- date last changed
- 2026-06-08 14:21:36
@misc{9231989,
abstract = {{This thesis investigates the design and evaluation of Large Language Model (LLM) agent assistants for industrial maintenance, developed in cooperation with SSAB. The study focuses on identifying what combinations of tools and architectures best serve the needs of maintenance technicians, with an emphasis on practical usability and factual correctness.
Multiple agents were developed and compared, incorporating combinations of Retrieval-Augmented Generation (RAG), internet search, direct document upload, and numerical database retrieval from a simulated Computerized Maintenance Management System (CMMS). All agents were built on the ChatGPT-4.1 base model. Evaluation was carried out through LLM-as-a-judge tests and a user survey involving subject matter experts and non-experts, analysed using nonparametric statistical methods and the Total Survey Error framework.
Results indicate that the combination of RAG with preprocessed data, limited internet search, and direct UI document upload yields the most effective performance for maintenance-related queries. Agents equipped with CMMS database retrieval via a Knowledge Base architecture performed poorly, suggesting that LLM-based retrieval logic is insufficient for reliable numerical reasoning in this context. The survey provided no significant evidence that increased functionality degrades the structural quality of agent responses.
In conclusion, this thesis demonstrates that a multi-tool LLM agent can provide meaningful support in an industrial maintenance context, and establishes a foundation for future work, particularly the adoption of a Model Context Protocol-based architecture for more robust numerical data access.}},
author = {{Bengtsson, Gabriel and Nilsson, Martin}},
issn = {{1404-6342}},
language = {{eng}},
note = {{Student Paper}},
series = {{Master's Theses in Mathematical Sciences}},
title = {{From Base Model to Maintenance Agent: Designing and Evaluating LLM Assistants for Industrial Settings}},
year = {{2026}},
}