Dual Control by Reinforcement Learning Using Deep Hyperstate Transition Models
Rosdahl, Christian LU
; Cervin, Anton
LU
and Bernhardsson, Bo
LU
(2022)
14th IFAC Workshop on Adaptive and Learning Control Systems, ALCOS 2022
In IFAC-PapersOnLine
55.
p.395-401
- Abstract
In dual control, the manipulated variables are used to both regulate the system and identify unknown parameters. The joint probability distribution of the system state and the parameters is known as the hyperstate. The paper proposes a method to perform dual control using a deep reinforcement learning algorithm in combination with a neural network model trained to represent hyperstate transitions. The hyperstate is compactly represented as the parameters of a mixture model that is fitted to Monte Carlo samples of the hyperstate. The representation is used to train a hyperstate transition model, which is used by a standard reinforcement learning algorithm to find a dual control policy. The method is evaluated on a simple nonlinear... (More)
In dual control, the manipulated variables are used to both regulate the system and identify unknown parameters. The joint probability distribution of the system state and the parameters is known as the hyperstate. The paper proposes a method to perform dual control using a deep reinforcement learning algorithm in combination with a neural network model trained to represent hyperstate transitions. The hyperstate is compactly represented as the parameters of a mixture model that is fitted to Monte Carlo samples of the hyperstate. The representation is used to train a hyperstate transition model, which is used by a standard reinforcement learning algorithm to find a dual control policy. The method is evaluated on a simple nonlinear system, which illustrates a situation where probing is needed, but it can also scale to high-dimensional systems. The method is demonstrated to be able to learn a probing technique that reduces the uncertainty of the hyperstate, resulting in improved control performance.
(Less)
- author
- Rosdahl, Christian
LU
; Cervin, Anton
LU
and Bernhardsson, Bo
LU
- organization
- publishing date
- 2022
- type
- Chapter in Book/Report/Conference proceeding
- publication status
- published
- subject
- keywords
- adaptive control, adaptive control by neural networks, Bayesian methods, nonlinear adaptive control, reinforcement learning control, stochastic optimal control problems
- host publication
- IFAC papers online
- series title
- IFAC-PapersOnLine
- volume
- 55
- edition
- 12
- pages
- 7 pages
- conference name
- 14th IFAC Workshop on Adaptive and Learning Control Systems, ALCOS 2022
- conference location
- Casablanca, Morocco
- conference dates
- 2022-06-29 - 2022-07-01
- external identifiers
-
- scopus:85137169393
- ISSN
- 2405-8963
- DOI
- 10.1016/j.ifacol.2022.07.344
- project
- Efficient Learning of Dynamical Systems
- language
- English
- LU publication?
- yes
- additional info
- Publisher Copyright: © 2022 Elsevier B.V.. All rights reserved.
- id
- 41440028-0669-4123-819b-5a09056d6662
- date added to LUP
- 2022-10-07 22:08:31
- date last changed
- 2023-11-16 21:42:02
@inproceedings{41440028-0669-4123-819b-5a09056d6662,
abstract = {{<p>In dual control, the manipulated variables are used to both regulate the system and identify unknown parameters. The joint probability distribution of the system state and the parameters is known as the hyperstate. The paper proposes a method to perform dual control using a deep reinforcement learning algorithm in combination with a neural network model trained to represent hyperstate transitions. The hyperstate is compactly represented as the parameters of a mixture model that is fitted to Monte Carlo samples of the hyperstate. The representation is used to train a hyperstate transition model, which is used by a standard reinforcement learning algorithm to find a dual control policy. The method is evaluated on a simple nonlinear system, which illustrates a situation where probing is needed, but it can also scale to high-dimensional systems. The method is demonstrated to be able to learn a probing technique that reduces the uncertainty of the hyperstate, resulting in improved control performance.</p>}},
author = {{Rosdahl, Christian and Cervin, Anton and Bernhardsson, Bo}},
booktitle = {{IFAC papers online}},
issn = {{2405-8963}},
keywords = {{adaptive control; adaptive control by neural networks; Bayesian methods; nonlinear adaptive control; reinforcement learning control; stochastic optimal control problems}},
language = {{eng}},
pages = {{395--401}},
series = {{IFAC-PapersOnLine}},
title = {{Dual Control by Reinforcement Learning Using Deep Hyperstate Transition Models}},
url = {{http://dx.doi.org/10.1016/j.ifacol.2022.07.344}},
doi = {{10.1016/j.ifacol.2022.07.344}},
volume = {{55}},
year = {{2022}},
}