Nonasymptotic Regret Analysis of Adaptive Linear Quadratic Control with Model Misspecification
Lee, Bruce D. LU ; Rantzer, Anders LU
and Matni, Nikolai
(2024)
6th Annual Learning for Dynamics and Control Conference, L4DC 2024
In Proceedings of Machine Learning Research
242.
p.980-992
- Abstract
The strategy of pre-training a large model on a diverse dataset, then fine-tuning for a particular application has yielded impressive results in computer vision, natural language processing, and robotic control. This strategy has vast potential in adaptive control, where it is necessary to rapidly adapt to changing conditions with limited data. Toward concretely understanding the benefit of pre-training for adaptive control, we study the adaptive linear quadratic control problem in the setting where the learner has prior knowledge of a collection of basis matrices for the dynamics. This basis is misspecified in the sense that it cannot perfectly represent the dynamics of the underlying data generating process. We propose an algorithm... (More)
The strategy of pre-training a large model on a diverse dataset, then fine-tuning for a particular application has yielded impressive results in computer vision, natural language processing, and robotic control. This strategy has vast potential in adaptive control, where it is necessary to rapidly adapt to changing conditions with limited data. Toward concretely understanding the benefit of pre-training for adaptive control, we study the adaptive linear quadratic control problem in the setting where the learner has prior knowledge of a collection of basis matrices for the dynamics. This basis is misspecified in the sense that it cannot perfectly represent the dynamics of the underlying data generating process. We propose an algorithm that uses this prior knowledge, and prove upper bounds on the expected regret after T interactions with the system. In the regime where T is small, the upper bounds are dominated by a term that scales with either poly(log T) or √T, depending on the prior knowledge available to the learner. When T is large, the regret is dominated by a term that grows with δT, where δ quantifies the level of misspecification. This linear term arises due to the inability to perfectly estimate the underlying dynamics using the misspecified basis, and is therefore unavoidable unless the basis matrices are also adapted online. However, it only dominates for large T, after the sublinear terms arising due to the error in estimating the weights for the basis matrices become negligible. We provide simulations that validate our analysis. Our simulations also show that offline data from a collection of related systems can be used as part of a pre-training stage to estimate a misspecified dynamics basis, which is in turn used by our adaptive controller.
(Less)
- author
- Lee, Bruce D.
LU
; Rantzer, Anders
LU
and Matni, Nikolai
- organization
- publishing date
- 2024
- type
- Chapter in Book/Report/Conference proceeding
- publication status
- published
- subject
- host publication
- Proceedings of Machine Learning Research
- series title
- Proceedings of Machine Learning Research
- volume
- 242
- pages
- 13 pages
- publisher
- ML Research Press
- conference name
- 6th Annual Learning for Dynamics and Control Conference, L4DC 2024
- conference location
- Oxford, United Kingdom
- conference dates
- 2024-07-15 - 2024-07-17
- external identifiers
-
- scopus:85203672403
- ISSN
- 2640-3498
- language
- English
- LU publication?
- yes
- additional info
- Publisher Copyright: © 2024 B.D. Lee1, A. Rantzer2 & N. Matni1.
- id
- 8872082d-821d-48cf-a157-3a7ed508cdda
- alternative location
- https://proceedings.mlr.press/v242/lee24a.html
- date added to LUP
- 2024-12-05 11:21:11
- date last changed
- 2025-04-04 15:24:05
@inproceedings{8872082d-821d-48cf-a157-3a7ed508cdda,
abstract = {{<p>The strategy of pre-training a large model on a diverse dataset, then fine-tuning for a particular application has yielded impressive results in computer vision, natural language processing, and robotic control. This strategy has vast potential in adaptive control, where it is necessary to rapidly adapt to changing conditions with limited data. Toward concretely understanding the benefit of pre-training for adaptive control, we study the adaptive linear quadratic control problem in the setting where the learner has prior knowledge of a collection of basis matrices for the dynamics. This basis is misspecified in the sense that it cannot perfectly represent the dynamics of the underlying data generating process. We propose an algorithm that uses this prior knowledge, and prove upper bounds on the expected regret after T interactions with the system. In the regime where T is small, the upper bounds are dominated by a term that scales with either poly(log T) or <sup>√</sup>T, depending on the prior knowledge available to the learner. When T is large, the regret is dominated by a term that grows with δT, where δ quantifies the level of misspecification. This linear term arises due to the inability to perfectly estimate the underlying dynamics using the misspecified basis, and is therefore unavoidable unless the basis matrices are also adapted online. However, it only dominates for large T, after the sublinear terms arising due to the error in estimating the weights for the basis matrices become negligible. We provide simulations that validate our analysis. Our simulations also show that offline data from a collection of related systems can be used as part of a pre-training stage to estimate a misspecified dynamics basis, which is in turn used by our adaptive controller.</p>}},
author = {{Lee, Bruce D. and Rantzer, Anders and Matni, Nikolai}},
booktitle = {{Proceedings of Machine Learning Research}},
issn = {{2640-3498}},
language = {{eng}},
pages = {{980--992}},
publisher = {{ML Research Press}},
series = {{Proceedings of Machine Learning Research}},
title = {{Nonasymptotic Regret Analysis of Adaptive Linear Quadratic Control with Model Misspecification}},
url = {{https://proceedings.mlr.press/v242/lee24a.html}},
volume = {{242}},
year = {{2024}},
}