Advanced

Exploring inter-rater reliability and measurement properties of environmental ratings using kappa and colocation quotients

Björk, Jonas LU ; Rittner, Ralf LU and Cromley, Ellen LU (2014) In Environmental Health 13.
Abstract
Background: Available evidence suggest that perceptions or ratings of the neighborhood, e.g. as being green, walkable or noisy, are important for effects on health and wellbeing, also after controlling for objective measures of identical or similar features. When evaluating effects of the perceived environment, it is important that measurement properties and the reliability of the environmental ratings are evaluated before decisions about how these ratings should be handled in the statistical analyses are made. In this paper we broaden the usage of two association measures, the well-known kappa statistic and the novel colocation quotient (CLQ), to studies of inter-rater reliability and of associations between different categorical ratings... (More)
Background: Available evidence suggest that perceptions or ratings of the neighborhood, e.g. as being green, walkable or noisy, are important for effects on health and wellbeing, also after controlling for objective measures of identical or similar features. When evaluating effects of the perceived environment, it is important that measurement properties and the reliability of the environmental ratings are evaluated before decisions about how these ratings should be handled in the statistical analyses are made. In this paper we broaden the usage of two association measures, the well-known kappa statistic and the novel colocation quotient (CLQ), to studies of inter-rater reliability and of associations between different categorical ratings in spatial contexts. Methods: We conducted reliability analysis of a survey instrument for assessing perceived greenness at geographical point locations, here the close outdoor environment within 5-10 minutes walking distance from home. Data were obtained from a public health survey conducted in 2008 in Scania, southern Sweden (n = 27 967 participants). Results: The results demonstrate the usefulness of kappa and CLQ as tools for assessing reliability and measurement properties of environmental rating scales when used at geographical point locations. We further show that the two measures are interchangeable, i.e. kappa can be accurately approximated from CLQ and vice versa, but can be used for somewhat different purposes in reliability analyses. Inter-rater reliability between the nearest neighbors was demonstrated for all five items of the evaluated instrument for assessing perceived greenness, albeit with clear differences across the items. Conclusion: Reliability analysis employing kappa and CLQ can be used as a basis for informed decisions about, for instance, how dichotomizations of the ratings should be defined and how missing or indefinite ratings should be handled. Such reliability analyses can thus serve as guidance for subsequent epidemiological studies of associations between environmental ratings, health and wellbeing. (Less)
Please use this url to cite or link to this publication:
author
organization
publishing date
type
Contribution to journal
publication status
published
subject
keywords
Epidemiologic methods, Spatial analysis, Reproducibility of results, Statistics as topic, Perceived greenness
in
Environmental Health
volume
13
publisher
BioMed Central
external identifiers
  • wos:000345792400001
  • scopus:84924075141
ISSN
1476-069X
DOI
10.1186/1476-069X-13-86
language
English
LU publication?
yes
id
787289f7-e721-4570-ac88-d4fc5b0a282f (old id 4965699)
date added to LUP
2015-02-03 07:02:49
date last changed
2017-05-28 04:00:46
@article{787289f7-e721-4570-ac88-d4fc5b0a282f,
  abstract     = {Background: Available evidence suggest that perceptions or ratings of the neighborhood, e.g. as being green, walkable or noisy, are important for effects on health and wellbeing, also after controlling for objective measures of identical or similar features. When evaluating effects of the perceived environment, it is important that measurement properties and the reliability of the environmental ratings are evaluated before decisions about how these ratings should be handled in the statistical analyses are made. In this paper we broaden the usage of two association measures, the well-known kappa statistic and the novel colocation quotient (CLQ), to studies of inter-rater reliability and of associations between different categorical ratings in spatial contexts. Methods: We conducted reliability analysis of a survey instrument for assessing perceived greenness at geographical point locations, here the close outdoor environment within 5-10 minutes walking distance from home. Data were obtained from a public health survey conducted in 2008 in Scania, southern Sweden (n = 27 967 participants). Results: The results demonstrate the usefulness of kappa and CLQ as tools for assessing reliability and measurement properties of environmental rating scales when used at geographical point locations. We further show that the two measures are interchangeable, i.e. kappa can be accurately approximated from CLQ and vice versa, but can be used for somewhat different purposes in reliability analyses. Inter-rater reliability between the nearest neighbors was demonstrated for all five items of the evaluated instrument for assessing perceived greenness, albeit with clear differences across the items. Conclusion: Reliability analysis employing kappa and CLQ can be used as a basis for informed decisions about, for instance, how dichotomizations of the ratings should be defined and how missing or indefinite ratings should be handled. Such reliability analyses can thus serve as guidance for subsequent epidemiological studies of associations between environmental ratings, health and wellbeing.},
  articleno    = {86},
  author       = {Björk, Jonas and Rittner, Ralf and Cromley, Ellen},
  issn         = {1476-069X},
  keyword      = {Epidemiologic methods,Spatial analysis,Reproducibility of results,Statistics as topic,Perceived greenness},
  language     = {eng},
  publisher    = {BioMed Central},
  series       = {Environmental Health},
  title        = {Exploring inter-rater reliability and measurement properties of environmental ratings using kappa and colocation quotients},
  url          = {http://dx.doi.org/10.1186/1476-069X-13-86},
  volume       = {13},
  year         = {2014},
}