Effective Troubleshooting in the Wild
(2026) FLMU16 20262Division of Risk Management and Societal Safety
- Abstract
- Context
Society has become increasingly dependent on digital services. When software services are unavailable or produce incorrect results, the consequences range from economic loss and inconvenience to life or death in the case of certain critical systems such as emergency telephone systems or hospital information systems. When systems fail, practitioners must understand the causal mechanism of the issue. This sensemaking process is a key underpinning of reliability in digital services, but it is not well understood.
Purpose
The objective of this research is to understand how practitioners operating digital systems perform sensemaking during troubleshooting: is it primarily a process based on deductive reasoning, or is it a complex... (More) - Context
Society has become increasingly dependent on digital services. When software services are unavailable or produce incorrect results, the consequences range from economic loss and inconvenience to life or death in the case of certain critical systems such as emergency telephone systems or hospital information systems. When systems fail, practitioners must understand the causal mechanism of the issue. This sensemaking process is a key underpinning of reliability in digital services, but it is not well understood.
Purpose
The objective of this research is to understand how practitioners operating digital systems perform sensemaking during troubleshooting: is it primarily a process based on deductive reasoning, or is it a complex search? What presents difficulties, and how are these difficulties resolved?
Methodology
A survey of 18 expert troubleshooters was performed and accounts of 14 particularly difficult troubleshooting cases were collected via interviewing, using the Critical Decision Method. The case data was analysed using a process tracing approach, and coded according to a number of dimensions. It is analysed alongside the quantitative and qualitative survey data.
Results
Sensemaking during troubleshooting involves building a mental model of the causal mechanism of a problem manifesting in a digital system. Practitioners generally approach it as a search, guided by cues that they have interpreted, combined with their experience. This means that process normally involves iteratively searching for answers to questions about the system, starting with broad questions and becoming more specific, while elaborating a mental model of the system and its behaviour. Effective troubleshooting does not usually involve proposing fully-developed hypotheses and testing them. The search can become stalled, due to fixation, thematic vagabonding, or an inability to generate a direction of inquiry or find answers to questions about the system’s state and behaviour. Becoming unstuck in most cases involves external assistance from another engineer, or a change in approach. Latent problems are the most challenging to make sense of.
Discussion
Expertise plays a significant role in effective troubleshooting. Experts are better able to overcome myriad sources of friction which arise during troubleshooting. All troubleshooting involves building mental models of the system and its dynamic behaviour: experts can draw on a larger set of previously-encountered architectural and behavioural patterns which help them to build mental models. However, successful investigations of issues in digital systems are normally deeply grounded in observation and searches for data, with simulation playing a more minor role, to help practitioners ‘join the dots’. This is because digital systems are complex, varied, and ever-changing, limiting practitioners' ability to rely on stable models from which specific hypotheses can simply be deduced.
Modern tooling used by software practitioners does a poor job of supporting the cognitive processes of sensemaking. Explicit modelling of the complex search that constitutes troubleshooting could make a very difficult job somewhat easier by helping to track the state of that search.
Digital systems operations is currently experiencing some ‘ironies of automation’ in that the advent of managed cloud services and the decline of self-hosting have resulted in a workforce that is less likely to have a lot of hands-on troubleshooting experience than in the past. Present development of tools which can ‘root-cause’ some kinds of issues in digital systems - but most likely not the most challenging and novel issues - risk provoking further ironies. (Less) - Popular Abstract
- Context
Society has become increasingly dependent on digital services. When software services are unavailable or produce incorrect results, the consequences range from economic loss and inconvenience to life or death in the case of certain critical systems such as emergency telephone systems or hospital information systems. When systems fail, practitioners must understand the causal mechanism of the issue. This sensemaking process is a key underpinning of reliability in digital services, but it is not well understood.
Purpose
The objective of this research is to understand how practitioners operating digital systems perform sensemaking during troubleshooting: is it primarily a process based on deductive reasoning, or is it a complex... (More) - Context
Society has become increasingly dependent on digital services. When software services are unavailable or produce incorrect results, the consequences range from economic loss and inconvenience to life or death in the case of certain critical systems such as emergency telephone systems or hospital information systems. When systems fail, practitioners must understand the causal mechanism of the issue. This sensemaking process is a key underpinning of reliability in digital services, but it is not well understood.
Purpose
The objective of this research is to understand how practitioners operating digital systems perform sensemaking during troubleshooting: is it primarily a process based on deductive reasoning, or is it a complex search? What presents difficulties, and how are these difficulties resolved?
Methodology
A survey of 18 expert troubleshooters was performed and accounts of 14 particularly difficult troubleshooting cases were collected via interviewing, using the Critical Decision Method. The case data was analysed using a process tracing approach, and coded according to a number of dimensions. It is analysed alongside the quantitative and qualitative survey data.
Results
Sensemaking during troubleshooting involves building a mental model of the causal mechanism of a problem manifesting in a digital system. Practitioners generally approach it as a search, guided by cues that they have interpreted, combined with their experience. This means that process normally involves iteratively searching for answers to questions about the system, starting with broad questions and becoming more specific, while elaborating a mental model of the system and its behaviour. Effective troubleshooting does not usually involve proposing fully-developed hypotheses and testing them. The search can become stalled, due to fixation, thematic vagabonding, or an inability to generate a direction of inquiry or find answers to questions about the system’s state and behaviour. Becoming unstuck in most cases involves external assistance from another engineer, or a change in approach. Latent problems are the most challenging to make sense of.
Discussion
Expertise plays a significant role in effective troubleshooting. Experts are better able to overcome myriad sources of friction which arise during troubleshooting. All troubleshooting involves building mental models of the system and its dynamic behaviour: experts can draw on a larger set of previously-encountered architectural and behavioural patterns which help them to build mental models. However, successful investigations of issues in digital systems are normally deeply grounded in observation and searches for data, with simulation playing a more minor role, to help practitioners ‘join the dots’. This is because digital systems are complex, varied, and ever-changing, limiting practitioners' ability to rely on stable models from which specific hypotheses can simply be deduced.
Modern tooling used by software practitioners does a poor job of supporting the cognitive processes of sensemaking. Explicit modelling of the complex search that constitutes troubleshooting could make a very difficult job somewhat easier by helping to track the state of that search.
Digital systems operations is currently experiencing some ‘ironies of automation’ in that the advent of managed cloud services and the decline of self-hosting have resulted in a workforce that is less likely to have a lot of hands-on troubleshooting experience than in the past. Present development of tools which can ‘root-cause’ some kinds of issues in digital systems - but most likely not the most challenging and novel issues - risk provoking further ironies. (Less)
Please use this url to cite or link to this publication:
https://lup.lub.lu.se/student-papers/record/9250837
- author
- Nolan, Laura LU
- supervisor
- organization
- course
- FLMU16 20262
- year
- 2026
- type
- H1 - Master's Degree (One Year)
- subject
- keywords
- FLMU06
- language
- English
- id
- 9250837
- date added to LUP
- 2026-09-15 11:03:31
- date last changed
- 2026-09-15 11:03:31
@misc{9250837,
abstract = {{Context
Society has become increasingly dependent on digital services. When software services are unavailable or produce incorrect results, the consequences range from economic loss and inconvenience to life or death in the case of certain critical systems such as emergency telephone systems or hospital information systems. When systems fail, practitioners must understand the causal mechanism of the issue. This sensemaking process is a key underpinning of reliability in digital services, but it is not well understood.
Purpose
The objective of this research is to understand how practitioners operating digital systems perform sensemaking during troubleshooting: is it primarily a process based on deductive reasoning, or is it a complex search? What presents difficulties, and how are these difficulties resolved?
Methodology
A survey of 18 expert troubleshooters was performed and accounts of 14 particularly difficult troubleshooting cases were collected via interviewing, using the Critical Decision Method. The case data was analysed using a process tracing approach, and coded according to a number of dimensions. It is analysed alongside the quantitative and qualitative survey data.
Results
Sensemaking during troubleshooting involves building a mental model of the causal mechanism of a problem manifesting in a digital system. Practitioners generally approach it as a search, guided by cues that they have interpreted, combined with their experience. This means that process normally involves iteratively searching for answers to questions about the system, starting with broad questions and becoming more specific, while elaborating a mental model of the system and its behaviour. Effective troubleshooting does not usually involve proposing fully-developed hypotheses and testing them. The search can become stalled, due to fixation, thematic vagabonding, or an inability to generate a direction of inquiry or find answers to questions about the system’s state and behaviour. Becoming unstuck in most cases involves external assistance from another engineer, or a change in approach. Latent problems are the most challenging to make sense of.
Discussion
Expertise plays a significant role in effective troubleshooting. Experts are better able to overcome myriad sources of friction which arise during troubleshooting. All troubleshooting involves building mental models of the system and its dynamic behaviour: experts can draw on a larger set of previously-encountered architectural and behavioural patterns which help them to build mental models. However, successful investigations of issues in digital systems are normally deeply grounded in observation and searches for data, with simulation playing a more minor role, to help practitioners ‘join the dots’. This is because digital systems are complex, varied, and ever-changing, limiting practitioners' ability to rely on stable models from which specific hypotheses can simply be deduced.
Modern tooling used by software practitioners does a poor job of supporting the cognitive processes of sensemaking. Explicit modelling of the complex search that constitutes troubleshooting could make a very difficult job somewhat easier by helping to track the state of that search.
Digital systems operations is currently experiencing some ‘ironies of automation’ in that the advent of managed cloud services and the decline of self-hosting have resulted in a workforce that is less likely to have a lot of hands-on troubleshooting experience than in the past. Present development of tools which can ‘root-cause’ some kinds of issues in digital systems - but most likely not the most challenging and novel issues - risk provoking further ironies.}},
author = {{Nolan, Laura}},
language = {{eng}},
note = {{Student Paper}},
title = {{Effective Troubleshooting in the Wild}},
year = {{2026}},
}