Volltext-Downloads (blau) und Frontdoor-Views (grau)

The Power of Linear Combinations: Learning with Random Convolutions

  • Following the traditional paradigm of convolutional neural networks (CNNs), modern CNNs manage to keep pace with more recent, for example transformer-based, models by not only increasing model depth and width but also the kernel size. This results in large amounts of learnable model parameters that need to be handled during training. While following the convolutional paradigm with the accordingFollowing the traditional paradigm of convolutional neural networks (CNNs), modern CNNs manage to keep pace with more recent, for example transformer-based, models by not only increasing model depth and width but also the kernel size. This results in large amounts of learnable model parameters that need to be handled during training. While following the convolutional paradigm with the according spatial inductive bias, we question the significance of \emph{learned} convolution filters. In fact, our findings demonstrate that many contemporary CNN architectures can achieve high test accuracies without ever updating randomly initialized (spatial) convolution filters. Instead, simple linear combinations (implemented through efficient 1×1 convolutions) suffice to effectively recombine even random filters into expressive network operators. Furthermore, these combinations of random filters can implicitly regularize the resulting operations, mitigating overfitting and enhancing overall performance and robustness. Conversely, retaining the ability to learn filter updates can impair network performance. Lastly, although we only observe relatively small gains from learning 3×3 convolutions, the learning gains increase proportionally with kernel size, owing to the non-idealities of the independent and identically distributed (\textit{i.i.d.}) nature of default initialization techniques.…zeige mehrzeige weniger

Metadaten exportieren

Weitere Dienste

Suche bei Google Scholar

Statistik

frontdoor_oas
Metadaten
Dokumentart:Preprint
Zitierlink: https://opus.hs-offenburg.de/8404
Bibliografische Angaben
Titel (Englisch):The Power of Linear Combinations: Learning with Random Convolutions
Verfasserangaben:Paul GavrikovStaff MemberORCiDGND, Janis KeuperStaff MemberORCiDGND
Erscheinungsjahr:2023
Seitenanzahl:14, 6
DOI:https://doi.org/10.48550/arXiv.2301.11360
Sprache:Englisch
Inhaltliche Informationen
Fakultäten / Einrichtungen:Fakultät Elektrotechnik, Medizintechnik und Informatik (EMI) (ab 04/2019)
Forschung:IMLA - Institute for Machine Learning and Analytics
Sammlungen der Hochschule Offenburg:Bibliografie
Formale Angaben
Relevanz für "Jahresbericht über Forschungsleistungen":Keine Relevanz
Open-Access-Status: Open Access 
 Diamond 
Lizenz (Deutsch):License LogoCreative Commons - CC BY-SA - Namensnennung - Weitergabe unter gleichen Bedingungen 4.0 International
ArXiv-Id:http://arxiv.org/abs/2301.11360