Use of 4 Open-Ended Text Responses to Help Identify People at Risk of Gaming Disorder : Preregistered Development and Usability Study Using Natural Language Processing
Strojny, Paweł ; Kapela, Ksawery ; Lipp, Natalia and Sikström, Sverker LU
(2024)
In JMIR Serious Games
12.
- Abstract
Background: Words are a natural way to describe mental states in humans, while numerical values are a convenient and effective way to carry out quantitative psychological research. With the growing interest of researchers in gaming disorder, the number of screening tools is growing. However, they all require self-quantification of mental states. The rapid development of natural language processing creates an opportunity to supplement traditional rating scales with a question-based computational language assessment approach that gives a deeper understanding of the studied phenomenon without losing the rigor of quantitative data analysis. Objective: The aim of the study was to investigate whether transformer-based language model analysis... (More)
Background: Words are a natural way to describe mental states in humans, while numerical values are a convenient and effective way to carry out quantitative psychological research. With the growing interest of researchers in gaming disorder, the number of screening tools is growing. However, they all require self-quantification of mental states. The rapid development of natural language processing creates an opportunity to supplement traditional rating scales with a question-based computational language assessment approach that gives a deeper understanding of the studied phenomenon without losing the rigor of quantitative data analysis. Objective: The aim of the study was to investigate whether transformer-based language model analysis of text responses from active gamers is a potential supplement to traditional rating scales. We compared a tool consisting of 4 open-ended questions formulated based on the clinician's intuition (not directly related to any existing rating scales for measuring gaming disorders) with the results of one of the commonly used rating scales. Methods: Participants recruited using an online panel were asked to answer the Word-Based Gaming Disorder Test, consisting of 4 open-ended questions about gaming. Subsequently, they completed a closed-ended Gaming Disorders Test based on a numerical scale. Of the initial 522 responses collected, we removed a total of 105 due to 1 of 3 criteria (suspiciously low survey completion time, providing nonrelevant or incomplete responses). Final analyses were conducted on the responses of 417 participants. The responses to the open-ended questions were vectorized using HerBERT, a large language model based on Google's Bidirectional Encoder Representations from Transformers (BERT). Last, a machine learning model, specifically ridge regression, was used to predict the scores of the Gaming Disorder Test based on the features of the vectorized open-ended responses. Results: The Pearson correlation between the observable scores from the Gaming Disorder test and the predictions made by the model was 0.476 when using the answers of the 4 respondents as features. When using only 1 of the 4 text responses, the correlation ranged from 0.274 to 0.406. Conclusions: Short open responses analyzed using natural language processing can contribute to a deeper understanding of gaming disorder at no additional cost in time. The obtained results confirmed 2 of 3 preregistered hypotheses. The written statements analyzed using the results of the model correlated with the rating scale. Furthermore, the inclusion in the model of data from more responses that take into account different perspectives on gaming improved the performance of the model. However, there is room for improvement, especially in terms of supplementing the questions with content that corresponds more directly to the definition of gaming disorder.
(Less)
- author
- Strojny, Paweł
; Kapela, Ksawery
; Lipp, Natalia
and Sikström, Sverker
LU
- organization
- publishing date
- 2024
- type
- Contribution to journal
- publication status
- published
- subject
- keywords
- correlation, gaming disorder, language model analysis, machine learning, mental health, natural language processing, NLP, open-ended, Pearson, Polish, psychological, Python, QCLA, Question-based Computational Language Assessment, response, risk, text, transformers-based
- in
- JMIR Serious Games
- volume
- 12
- article number
- e56663
- publisher
- JMIR Publications Inc.
- external identifiers
-
- pmid:39739378
- scopus:85214314280
- ISSN
- 2291-9279
- DOI
- 10.2196/56663
- language
- English
- LU publication?
- yes
- id
- f645eb84-dec3-4b96-a9ed-f6bcbba70cc6
- date added to LUP
- 2025-02-24 14:00:51
- date last changed
- 2026-01-13 17:24:02
@article{f645eb84-dec3-4b96-a9ed-f6bcbba70cc6,
abstract = {{<p>Background: Words are a natural way to describe mental states in humans, while numerical values are a convenient and effective way to carry out quantitative psychological research. With the growing interest of researchers in gaming disorder, the number of screening tools is growing. However, they all require self-quantification of mental states. The rapid development of natural language processing creates an opportunity to supplement traditional rating scales with a question-based computational language assessment approach that gives a deeper understanding of the studied phenomenon without losing the rigor of quantitative data analysis. Objective: The aim of the study was to investigate whether transformer-based language model analysis of text responses from active gamers is a potential supplement to traditional rating scales. We compared a tool consisting of 4 open-ended questions formulated based on the clinician's intuition (not directly related to any existing rating scales for measuring gaming disorders) with the results of one of the commonly used rating scales. Methods: Participants recruited using an online panel were asked to answer the Word-Based Gaming Disorder Test, consisting of 4 open-ended questions about gaming. Subsequently, they completed a closed-ended Gaming Disorders Test based on a numerical scale. Of the initial 522 responses collected, we removed a total of 105 due to 1 of 3 criteria (suspiciously low survey completion time, providing nonrelevant or incomplete responses). Final analyses were conducted on the responses of 417 participants. The responses to the open-ended questions were vectorized using HerBERT, a large language model based on Google's Bidirectional Encoder Representations from Transformers (BERT). Last, a machine learning model, specifically ridge regression, was used to predict the scores of the Gaming Disorder Test based on the features of the vectorized open-ended responses. Results: The Pearson correlation between the observable scores from the Gaming Disorder test and the predictions made by the model was 0.476 when using the answers of the 4 respondents as features. When using only 1 of the 4 text responses, the correlation ranged from 0.274 to 0.406. Conclusions: Short open responses analyzed using natural language processing can contribute to a deeper understanding of gaming disorder at no additional cost in time. The obtained results confirmed 2 of 3 preregistered hypotheses. The written statements analyzed using the results of the model correlated with the rating scale. Furthermore, the inclusion in the model of data from more responses that take into account different perspectives on gaming improved the performance of the model. However, there is room for improvement, especially in terms of supplementing the questions with content that corresponds more directly to the definition of gaming disorder.</p>}},
author = {{Strojny, Paweł and Kapela, Ksawery and Lipp, Natalia and Sikström, Sverker}},
issn = {{2291-9279}},
keywords = {{correlation; gaming disorder; language model analysis; machine learning; mental health; natural language processing; NLP; open-ended; Pearson; Polish; psychological; Python; QCLA; Question-based Computational Language Assessment; response; risk; text; transformers-based}},
language = {{eng}},
publisher = {{JMIR Publications Inc.}},
series = {{JMIR Serious Games}},
title = {{Use of 4 Open-Ended Text Responses to Help Identify People at Risk of Gaming Disorder : Preregistered Development and Usability Study Using Natural Language Processing}},
url = {{http://dx.doi.org/10.2196/56663}},
doi = {{10.2196/56663}},
volume = {{12}},
year = {{2024}},
}