On the unreasonable vulnerability of transformers for image restoration – and an easy fix
- Following their success in visual recognition tasks, Vision Transformers(ViTs) are being increasingly employed for image restoration. As a few recent works claim that ViTs for image classification also have better robustness properties, we investigate whether the improved adversarial robustness of ViTs extends to image restoration. We consider the recently proposed Restormer model, as well asFollowing their success in visual recognition tasks, Vision Transformers(ViTs) are being increasingly employed for image restoration. As a few recent works claim that ViTs for image classification also have better robustness properties, we investigate whether the improved adversarial robustness of ViTs extends to image restoration. We consider the recently proposed Restormer model, as well as NAFNet and the "Baseline network" which are both simplified versions of a Restormer. We use Projected Gradient Descent (PGD) and CosPGD for our robustness evaluation. Our experiments are performed on real-world images from the GoPro dataset for image deblurring. Our analysis indicates that contrary to as advocated by ViTs in image classification works, these models are highly susceptible to adversarial attacks. We attempt to find an easy fix and improve their robustness through adversarial training. While this yields a significant increase in robustness for Restormer, results on other networks are less promising. Interestingly, we find that the design choices in NAFNet and Baselines, which were based on iid performance, and not on robust generalization, seem to be at odds with the model robustness.…
Document Type: | Conference Proceeding |
---|---|
Conference Type: | Konferenzartikel |
Zitierlink: | https://opus.hs-offenburg.de/8392 | Bibliografische Angaben |
Title (English): | On the unreasonable vulnerability of transformers for image restoration – and an easy fix |
Conference: | IEEE/CVF International Conference on Computer Vision Workshops (02-06 October 2023 : Paris, France) |
Author: | Shashank Agnihotri, Kanchana Vaishnavi Gandikota, Julia GrabinskiStaff MemberORCiD, Paramanand Chandramouli, Margret Keuper |
Year of Publication: | 2023 |
Publisher: | IEEE |
First Page: | 3709 |
Last Page: | 3719 |
Parent Title (English): | Proceedings : 2023 IEEE/CVF International Conference on Computer Vision Workshops : ICCVW 2023 |
ISBN: | 979-8-3503-0744-3 (Elektronisch) |
ISBN: | 979-8-3503-0745-0 (Print on Demand) |
ISSN: | 2473-9944 (Elektronisch) |
ISSN: | 2473-9936 (Print on Demand) |
DOI: | https://doi.org/10.1109/ICCVW60793.2023.00398 |
Language: | English | Inhaltliche Informationen |
Institutes: | Forschung / IMLA - Institute for Machine Learning and Analytics |
Institutes: | Bibliografie |
Tag: | Image restoration |
Funded by (selection): | Deutsche Forschungsgemeinschaft |
Funded by (textarea): | DFG research unit 5336 - Learning to Sense | Formale Angaben |
Relevance: | Konferenzbeitrag: h5-Index > 30 |
Open Access: | Closed |
Licence (German): | Urheberrechtlich geschützt |