Time-dependent evaluation of recommender systems
(2021) 2021 Perspectives on the Evaluation of Recommender Systems Workshop, Perspectives 2021 In CEUR Workshop Proceedings 2955.- Abstract
Evaluation of recommender systems is an actively discussed topic in the recommender system community. However, some aspects of evaluation have received little to no attention, one of them being whether evaluating recommender system algorithms with single-number metrics is sufficient. When presenting results as a single number, the only possible assumption is a stable performance over time regardless of changes in the datasets, while it intuitively seems more likely that the performance changes over time. We suggest presenting results over time, making it possible to identify trends and changes in performance as the dataset grows and changes. In this paper, we conduct an analysis of 6 algorithms on 10 datasets over time to identify the... (More)
Evaluation of recommender systems is an actively discussed topic in the recommender system community. However, some aspects of evaluation have received little to no attention, one of them being whether evaluating recommender system algorithms with single-number metrics is sufficient. When presenting results as a single number, the only possible assumption is a stable performance over time regardless of changes in the datasets, while it intuitively seems more likely that the performance changes over time. We suggest presenting results over time, making it possible to identify trends and changes in performance as the dataset grows and changes. In this paper, we conduct an analysis of 6 algorithms on 10 datasets over time to identify the need for a time-dependent evaluation. To enable this evaluation over time, we split the datasets based on the provided timesteps into smaller subsets. At every tested timepoint we use all available data up to this timepoint, simulating a growing dataset as encountered in the realworld. Our results show that for 90% of the datasets the performance changes over time and in 60% even the ranking of algorithms changes over time.
(Less)
- author
- Scheidt, Teresa and Beel, Joeran
- publishing date
- 2021
- type
- Chapter in Book/Report/Conference proceeding
- publication status
- published
- subject
- keywords
- Evaluation, Recommender systems, Time-dependent evaluation
- host publication
- Perspectives 2021 : Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop 2021 - Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop 2021
- series title
- CEUR Workshop Proceedings
- volume
- 2955
- conference name
- 2021 Perspectives on the Evaluation of Recommender Systems Workshop, Perspectives 2021
- conference location
- Amsterdam, Netherlands
- conference dates
- 2021-09-25
- external identifiers
-
- scopus:85116263192
- ISSN
- 1613-0073
- language
- English
- LU publication?
- no
- additional info
- Publisher Copyright: © 2021 Copyright for this paper by its authors.
- id
- 44265d57-d70f-4725-bf73-9c67800a5f77
- alternative location
- http://ceur-ws.org/Vol-2955/paper10.pdf
- date added to LUP
- 2021-11-01 13:38:42
- date last changed
- 2022-04-27 05:21:40
@inproceedings{44265d57-d70f-4725-bf73-9c67800a5f77,
abstract = {{<p>Evaluation of recommender systems is an actively discussed topic in the recommender system community. However, some aspects of evaluation have received little to no attention, one of them being whether evaluating recommender system algorithms with single-number metrics is sufficient. When presenting results as a single number, the only possible assumption is a stable performance over time regardless of changes in the datasets, while it intuitively seems more likely that the performance changes over time. We suggest presenting results over time, making it possible to identify trends and changes in performance as the dataset grows and changes. In this paper, we conduct an analysis of 6 algorithms on 10 datasets over time to identify the need for a time-dependent evaluation. To enable this evaluation over time, we split the datasets based on the provided timesteps into smaller subsets. At every tested timepoint we use all available data up to this timepoint, simulating a growing dataset as encountered in the realworld. Our results show that for 90% of the datasets the performance changes over time and in 60% even the ranking of algorithms changes over time.</p>}},
author = {{Scheidt, Teresa and Beel, Joeran}},
booktitle = {{Perspectives 2021 : Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop 2021}},
issn = {{1613-0073}},
keywords = {{Evaluation; Recommender systems; Time-dependent evaluation}},
language = {{eng}},
series = {{CEUR Workshop Proceedings}},
title = {{Time-dependent evaluation of recommender systems}},
url = {{http://ceur-ws.org/Vol-2955/paper10.pdf}},
volume = {{2955}},
year = {{2021}},
}