ERA : Enhanced Rational Activations
(2022) 17th European Conference on Computer Vision, ECCV 2022 In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 13680 LNCS. p.722-738- Abstract
Activation functions play a central role in deep learning since they form an essential building stone of neural networks. In the last few years, the focus has been shifting towards investigating new types of activations that outperform the classical Rectified Linear Unit (ReLU) in modern neural architectures. Most recently, rational activation functions (RAFs) have awakened interest because they were shown to perform on par with state-of-the-art activations on image classification. Despite their apparent potential, prior formulations are either not safe, not smooth, or not “true” rational functions, and they only work with careful initialisation. Aiming to mitigate these issues, we propose a novel, enhanced rational function, ERA, and... (More)
Activation functions play a central role in deep learning since they form an essential building stone of neural networks. In the last few years, the focus has been shifting towards investigating new types of activations that outperform the classical Rectified Linear Unit (ReLU) in modern neural architectures. Most recently, rational activation functions (RAFs) have awakened interest because they were shown to perform on par with state-of-the-art activations on image classification. Despite their apparent potential, prior formulations are either not safe, not smooth, or not “true” rational functions, and they only work with careful initialisation. Aiming to mitigate these issues, we propose a novel, enhanced rational function, ERA, and investigate how to better accommodate the specific needs of these activations, to both network components and training regime. In addition to being more stable, the proposed function outperforms other standard ones across a range of lightweight network architectures on two different tasks: image classification and 3d human pose and shape reconstruction.
(Less)
- author
- Trimmel, Martin LU ; Zanfir, Mihai ; Hartley, Richard and Sminchisescu, Cristian LU
- organization
- publishing date
- 2022
- type
- Chapter in Book/Report/Conference proceeding
- publication status
- published
- subject
- keywords
- Activation function, Deep learning, Neural networks, Rational activation
- host publication
- Computer Vision – ECCV 2022 - 17th European Conference, Proceedings
- series title
- Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
- editor
- Avidan, Shai ; Brostow, Gabriel ; Cissé, Moustapha ; Farinella, Giovanni Maria and Hassner, Tal
- volume
- 13680 LNCS
- pages
- 17 pages
- publisher
- Springer Science and Business Media B.V.
- conference name
- 17th European Conference on Computer Vision, ECCV 2022
- conference location
- Tel Aviv, Israel
- conference dates
- 2022-10-23 - 2022-10-27
- external identifiers
-
- scopus:85144577051
- ISSN
- 1611-3349
- 0302-9743
- ISBN
- 9783031200434
- DOI
- 10.1007/978-3-031-20044-1_41
- language
- English
- LU publication?
- yes
- id
- 9241041b-e6df-4972-911e-a4ff7bc67534
- date added to LUP
- 2023-01-12 10:35:28
- date last changed
- 2024-04-18 17:47:15
@inproceedings{9241041b-e6df-4972-911e-a4ff7bc67534,
abstract = {{<p>Activation functions play a central role in deep learning since they form an essential building stone of neural networks. In the last few years, the focus has been shifting towards investigating new types of activations that outperform the classical Rectified Linear Unit (ReLU) in modern neural architectures. Most recently, rational activation functions (RAFs) have awakened interest because they were shown to perform on par with state-of-the-art activations on image classification. Despite their apparent potential, prior formulations are either not safe, not smooth, or not “true” rational functions, and they only work with careful initialisation. Aiming to mitigate these issues, we propose a novel, enhanced rational function, ERA, and investigate how to better accommodate the specific needs of these activations, to both network components and training regime. In addition to being more stable, the proposed function outperforms other standard ones across a range of lightweight network architectures on two different tasks: image classification and 3d human pose and shape reconstruction.</p>}},
author = {{Trimmel, Martin and Zanfir, Mihai and Hartley, Richard and Sminchisescu, Cristian}},
booktitle = {{Computer Vision – ECCV 2022 - 17th European Conference, Proceedings}},
editor = {{Avidan, Shai and Brostow, Gabriel and Cissé, Moustapha and Farinella, Giovanni Maria and Hassner, Tal}},
isbn = {{9783031200434}},
issn = {{1611-3349}},
keywords = {{Activation function; Deep learning; Neural networks; Rational activation}},
language = {{eng}},
pages = {{722--738}},
publisher = {{Springer Science and Business Media B.V.}},
series = {{Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)}},
title = {{ERA : Enhanced Rational Activations}},
url = {{http://dx.doi.org/10.1007/978-3-031-20044-1_41}},
doi = {{10.1007/978-3-031-20044-1_41}},
volume = {{13680 LNCS}},
year = {{2022}},
}