Skip to main content

Lund University Publications

LUND UNIVERSITY LIBRARIES

Superficially Plausible Outputs from a Black Box : Problematising GenAI Tools for Analysing Qualitative SoTL Data

Glessmer, Mirjam Sophia LU orcid and Forsyth, Rachel LU orcid (2025) In Teaching and Learning Inquiry 13.
Abstract

Generative AI tools (GenAI) are increasingly used for academic tasks, including qualitative data analysis for the Scholarship of Teaching and Learning (SoTL). In our practice as academic developers, we are frequently asked for advice on whether this use for GenAI is reliable, valid, and ethical. Since this is a new field, we have not been able to answer this confidently based on published literature, which depicts both very positive as well as highly cautionary accounts. To fill this gap, we experiment with the use of chatbot style GenAI (namely ChatGPT 4, ChatGPT 4o, and Microsoft Copilot) to support or conduct qualitative analysis of survey and interview data from a SoTL project, which had previously been analysed by experienced... (More)

Generative AI tools (GenAI) are increasingly used for academic tasks, including qualitative data analysis for the Scholarship of Teaching and Learning (SoTL). In our practice as academic developers, we are frequently asked for advice on whether this use for GenAI is reliable, valid, and ethical. Since this is a new field, we have not been able to answer this confidently based on published literature, which depicts both very positive as well as highly cautionary accounts. To fill this gap, we experiment with the use of chatbot style GenAI (namely ChatGPT 4, ChatGPT 4o, and Microsoft Copilot) to support or conduct qualitative analysis of survey and interview data from a SoTL project, which had previously been analysed by experienced researchers using thematic analysis. At first sight, the output looked plausible, but the results were incomplete and not reproducible. In some instances, interpretations and extrapolations of data happened when it was clearly stated in the prompt that the tool should only analyse a specified dataset based on explicit instructions. Since both algorithm and training data of the GenAI tools are undisclosed, it is impossible to know how the outputs had been arrived at. We conclude that while results may look plausible initially, digging deeper soon reveals serious problems; the lack of transparency about how analyses are conducted and results are generated means that no reproducible method can be described. We therefore warn against an uncritical use of GenAI in qualitative analysis of SoTL data.

(Less)
Please use this url to cite or link to this publication:
author
and
organization
publishing date
type
Contribution to journal
publication status
published
subject
keywords
academic development, GenAI, generative AI, qualitative analysis, research methods
in
Teaching and Learning Inquiry
volume
13
publisher
University of Calgary Press
external identifiers
  • scopus:85217710517
ISSN
2167-4779
DOI
10.20343/teachlearninqu.13.4
language
English
LU publication?
yes
additional info
Publisher Copyright: © 2025 University of Calgary. All rights reserved.
id
fa6dd2dd-0190-4975-b02b-151ac7afc77a
date added to LUP
2025-02-21 10:55:43
date last changed
2025-04-04 14:55:56
@article{fa6dd2dd-0190-4975-b02b-151ac7afc77a,
  abstract     = {{<p>Generative AI tools (GenAI) are increasingly used for academic tasks, including qualitative data analysis for the Scholarship of Teaching and Learning (SoTL). In our practice as academic developers, we are frequently asked for advice on whether this use for GenAI is reliable, valid, and ethical. Since this is a new field, we have not been able to answer this confidently based on published literature, which depicts both very positive as well as highly cautionary accounts. To fill this gap, we experiment with the use of chatbot style GenAI (namely ChatGPT 4, ChatGPT 4o, and Microsoft Copilot) to support or conduct qualitative analysis of survey and interview data from a SoTL project, which had previously been analysed by experienced researchers using thematic analysis. At first sight, the output looked plausible, but the results were incomplete and not reproducible. In some instances, interpretations and extrapolations of data happened when it was clearly stated in the prompt that the tool should only analyse a specified dataset based on explicit instructions. Since both algorithm and training data of the GenAI tools are undisclosed, it is impossible to know how the outputs had been arrived at. We conclude that while results may look plausible initially, digging deeper soon reveals serious problems; the lack of transparency about how analyses are conducted and results are generated means that no reproducible method can be described. We therefore warn against an uncritical use of GenAI in qualitative analysis of SoTL data.</p>}},
  author       = {{Glessmer, Mirjam Sophia and Forsyth, Rachel}},
  issn         = {{2167-4779}},
  keywords     = {{academic development; GenAI; generative AI; qualitative analysis; research methods}},
  language     = {{eng}},
  publisher    = {{University of Calgary Press}},
  series       = {{Teaching and Learning Inquiry}},
  title        = {{Superficially Plausible Outputs from a Black Box : Problematising GenAI Tools for Analysing Qualitative SoTL Data}},
  url          = {{http://dx.doi.org/10.20343/teachlearninqu.13.4}},
  doi          = {{10.20343/teachlearninqu.13.4}},
  volume       = {{13}},
  year         = {{2025}},
}