Benchopt : Reproducible, efficient and collaborative optimization benchmarks
(2022) 36th Conference on Neural Information Processing Systems, NeurIPS 2022 In Advances in Neural Information Processing Systems 35. p.25404-25421- Abstract
Numerical validation is at the core of machine learning research as it allows to assess the actual impact of new methods, and to confirm the agreement between theory and practice. Yet, the rapid development of the field poses several challenges: researchers are confronted with a profusion of methods to compare, limited transparency and consensus on best practices, as well as tedious re-implementation work. As a result, validation is often very partial, which can lead to wrong conclusions that slow down the progress of research. We propose Benchopt, a collaborative framework to automate, reproduce and publish optimization benchmarks in machine learning across programming languages and hardware architectures. Benchopt simplifies... (More)
Numerical validation is at the core of machine learning research as it allows to assess the actual impact of new methods, and to confirm the agreement between theory and practice. Yet, the rapid development of the field poses several challenges: researchers are confronted with a profusion of methods to compare, limited transparency and consensus on best practices, as well as tedious re-implementation work. As a result, validation is often very partial, which can lead to wrong conclusions that slow down the progress of research. We propose Benchopt, a collaborative framework to automate, reproduce and publish optimization benchmarks in machine learning across programming languages and hardware architectures. Benchopt simplifies benchmarking for the community by providing an off-the-shelf tool for running, sharing and extending experiments. To demonstrate its broad usability, we showcase benchmarks on three standard learning tasks: ℓ2-regularized logistic regression, Lasso, and ResNet18 training for image classification. These benchmarks highlight key practical findings that give a more nuanced view of the state-of-the-art for these problems, showing that for practical evaluation, the devil is in the details. We hope that Benchopt will foster collaborative work in the community hence improving the reproducibility of research findings.
(Less)
- author
- Moreau, Thomas
; Massias, Mathurin
; Gramfort, Alexandre
; Ablin, Pierre
; Bannier, Pierre Antoine
; Charlier, Benjamin
; Dagréou, Mathieu
; la Tour, Tom Dupré
; Durif, Ghislain
; Dantas, Cassio F.
; Klopfenstein, Quentin
; Larsson, Johan
LU
; Lai, En
; Lefort, Tanguy
; Malézieux, Benoit
; Moufad, Badr
; Nguyen, Binh T.
; Rakotomamonjy, Alain
; Ramzi, Zaccharie
; Salmon, Joseph
and Vaiter, Samuel
(Less) - organization
- publishing date
- 2022-12-06
- type
- Chapter in Book/Report/Conference proceeding
- publication status
- published
- subject
- keywords
- Logistic regression, Machine learning
- host publication
- Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022
- series title
- Advances in Neural Information Processing Systems
- editor
- Koyejo, S. ; Mohamed, S. ; Agarwal, A. ; Belgrave, D. ; Cho, K. and Oh, A.
- volume
- 35
- pages
- 25404 - 25421
- publisher
- Curran Associates, Inc
- conference name
- 36th Conference on Neural Information Processing Systems, NeurIPS 2022
- conference location
- New Orleans, United States
- conference dates
- 2022-11-28 - 2022-12-09
- external identifiers
-
- scopus:85141381738
- ISSN
- 1049-5258
- ISBN
- 9781713871088
- project
- Optimization and Algorithms for Sparse Regression
- language
- English
- LU publication?
- yes
- additional info
- Publisher Copyright: © 2022 Neural information processing systems foundation. All rights reserved.
- id
- 876fdff9-0b5a-4717-9c7b-25f73e93d612
- date added to LUP
- 2023-09-14 11:57:55
- date last changed
- 2023-12-19 16:07:49
@inproceedings{876fdff9-0b5a-4717-9c7b-25f73e93d612,
abstract = {{<p>Numerical validation is at the core of machine learning research as it allows to assess the actual impact of new methods, and to confirm the agreement between theory and practice. Yet, the rapid development of the field poses several challenges: researchers are confronted with a profusion of methods to compare, limited transparency and consensus on best practices, as well as tedious re-implementation work. As a result, validation is often very partial, which can lead to wrong conclusions that slow down the progress of research. We propose Benchopt, a collaborative framework to automate, reproduce and publish optimization benchmarks in machine learning across programming languages and hardware architectures. Benchopt simplifies benchmarking for the community by providing an off-the-shelf tool for running, sharing and extending experiments. To demonstrate its broad usability, we showcase benchmarks on three standard learning tasks: ℓ<sub>2</sub>-regularized logistic regression, Lasso, and ResNet18 training for image classification. These benchmarks highlight key practical findings that give a more nuanced view of the state-of-the-art for these problems, showing that for practical evaluation, the devil is in the details. We hope that Benchopt will foster collaborative work in the community hence improving the reproducibility of research findings.</p>}},
author = {{Moreau, Thomas and Massias, Mathurin and Gramfort, Alexandre and Ablin, Pierre and Bannier, Pierre Antoine and Charlier, Benjamin and Dagréou, Mathieu and la Tour, Tom Dupré and Durif, Ghislain and Dantas, Cassio F. and Klopfenstein, Quentin and Larsson, Johan and Lai, En and Lefort, Tanguy and Malézieux, Benoit and Moufad, Badr and Nguyen, Binh T. and Rakotomamonjy, Alain and Ramzi, Zaccharie and Salmon, Joseph and Vaiter, Samuel}},
booktitle = {{Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022}},
editor = {{Koyejo, S. and Mohamed, S. and Agarwal, A. and Belgrave, D. and Cho, K. and Oh, A.}},
isbn = {{9781713871088}},
issn = {{1049-5258}},
keywords = {{Logistic regression; Machine learning}},
language = {{eng}},
month = {{12}},
pages = {{25404--25421}},
publisher = {{Curran Associates, Inc}},
series = {{Advances in Neural Information Processing Systems}},
title = {{Benchopt : Reproducible, efficient and collaborative optimization benchmarks}},
volume = {{35}},
year = {{2022}},
}