Improving the Performance of Shallow Neural Networks Using Norm-Preserving Regularization
(2026)Department of Automatic Control
- Abstract
- The Shallow Neural Network (SNN) is one of the most basic forms of machine learning models. It has a high theoretical representational capacity, but this is often not realized in practice, in part due to overfitting. This thesis proposes a new regularization method and compares it against existing techniques across a range of tasks using SNNs. The proposed method, norm-preserving regularization (NP-reg), penalizes ℓ2-norm difference between the input vector and hidden representation. Adding this penalty term to the objective function was found to affect the generalization performance of SNNs, particularly in heavily overparametrized settings.
Performance improvements were observed for larger models, both in standalone SNNs and when used... (More) - The Shallow Neural Network (SNN) is one of the most basic forms of machine learning models. It has a high theoretical representational capacity, but this is often not realized in practice, in part due to overfitting. This thesis proposes a new regularization method and compares it against existing techniques across a range of tasks using SNNs. The proposed method, norm-preserving regularization (NP-reg), penalizes ℓ2-norm difference between the input vector and hidden representation. Adding this penalty term to the objective function was found to affect the generalization performance of SNNs, particularly in heavily overparametrized settings.
Performance improvements were observed for larger models, both in standalone SNNs and when used as a classifier head in transfer learning settings. For smaller models, NP-reg generally had negligible or negative effects on performance. Among existing regularization methods, Batch Norm and Layer Norm showed both positive and negative effects depending on the task, while dropout and weight decay had limited impact.
The mechanisms underlying the performance gains remain unclear. NP-reg produced a rougher loss landscape while still improving performance. This was also observed for Batch Norm, suggesting that smoother optimization alone does not explain the performance gains in the shallow settings. In some tasks, NP-reg reduced rank degradation of feature representations similarly to what Batch Norm was found to do, though neither this effect was consistently associated with improved performance.
Despite the lack of a complete theoretical explanation, NP-reg showed practical value and was shown to be robust across different settings, functioning with multiple activation functions while remaining more computationally feasible than similar approaches. (Less)
Please use this url to cite or link to this publication:
https://lup.lub.lu.se/student-papers/record/9249001
- author
- Stoopendahl, Daniel
- supervisor
- organization
- year
- 2026
- type
- H3 - Professional qualifications (4 Years - )
- subject
- report number
- TFRT-6317
- other publication id
- 0280-5316
- language
- English
- id
- 9249001
- date added to LUP
- 2026-08-25 10:30:31
- date last changed
- 2026-08-25 10:30:31
@misc{9249001,
abstract = {{The Shallow Neural Network (SNN) is one of the most basic forms of machine learning models. It has a high theoretical representational capacity, but this is often not realized in practice, in part due to overfitting. This thesis proposes a new regularization method and compares it against existing techniques across a range of tasks using SNNs. The proposed method, norm-preserving regularization (NP-reg), penalizes ℓ2-norm difference between the input vector and hidden representation. Adding this penalty term to the objective function was found to affect the generalization performance of SNNs, particularly in heavily overparametrized settings.
Performance improvements were observed for larger models, both in standalone SNNs and when used as a classifier head in transfer learning settings. For smaller models, NP-reg generally had negligible or negative effects on performance. Among existing regularization methods, Batch Norm and Layer Norm showed both positive and negative effects depending on the task, while dropout and weight decay had limited impact.
The mechanisms underlying the performance gains remain unclear. NP-reg produced a rougher loss landscape while still improving performance. This was also observed for Batch Norm, suggesting that smoother optimization alone does not explain the performance gains in the shallow settings. In some tasks, NP-reg reduced rank degradation of feature representations similarly to what Batch Norm was found to do, though neither this effect was consistently associated with improved performance.
Despite the lack of a complete theoretical explanation, NP-reg showed practical value and was shown to be robust across different settings, functioning with multiple activation functions while remaining more computationally feasible than similar approaches.}},
author = {{Stoopendahl, Daniel}},
language = {{eng}},
note = {{Student Paper}},
title = {{Improving the Performance of Shallow Neural Networks Using Norm-Preserving Regularization}},
year = {{2026}},
}