SASA: Sequence-Aware Shadow Attacks via Attention Alignment for Traffic Sign Recognition

Jun 1, 2025· Amir Salarpour, Pedram MohajerAnsari, David FernandezDavid Fernandez , Mert D. Pese
PDF Cite

Key Contributions & Takeaways

A car does not see a traffic sign once; it sees it across a stream of frames as it drives closer. Most attacks fool a single still image with pixel noise that flickers and gets filtered out before it matters. This paper casts a single, physically plausible shadow, the kind a pole or a cloud would throw, and holds it steady across the whole clip. It places that shadow exactly where a frozen vision transformer is looking, and it never once queries the model it is trying to fool. The result is a strict black-box attack that transfers across very different classifiers and drops accuracy far more than per-frame scores would suggest.

  • A shadow attack for video, not single frames, that never touches the target. SASA optimizes one shared shadow mask applied identically across all 30 frames, guided only by frozen DINO and DeiT attention maps. It uses no weights, gradients, or predictions from the model it attacks, yet transfers across CNNs, STNs, EfficientNet, and ViTs.
  • Three physically grounded shadow shapes, all differentiable. A differentiable generator casts Blob (clouds, tree cover), Strip (poles, signposts), and Side (barriers, parked cars) shadows from a handful of geometric parameters, darkening only the luminance channel in CIELAB so the result stays plausible. The elongated StripShadow guided by DeiT is the strongest across every architecture.
  • Sequential recognition is far more fragile than frame scores suggest. By aligning the shadow to the model’s most salient pixels, SASA drops frame accuracy by up to 86 points on ViT and over 60 on EfficientNet, and sequence-level accuracy (seq@50) by over 90 points. Attention maps confirm the shadow displaces where the model looks, not just how the pixels appear.
publications

01  ·  The Problem

Most attacks fool a single still image with pixel noise and need to probe the model. But a car reads a sign across a video stream, and temporal filtering wipes flickering noise out before it reaches the classifier.

02  ·  The Approach

Cast one physically plausible shadow, the kind a pole or cloud throws, and hold it steady across all 30 frames. Place it where frozen DINO and DeiT transformers look, and never query the target model.

03  ·  The Finding

The shadow transfers across four unseen classifiers, cutting frame accuracy by up to 86 points on ViT and sequence-level accuracy by over 90, with the elongated StripShadow the strongest variant.

How SASA Builds The Attack
Two frozen transformers say where the model looks; one shared shadow is optimized to sit there, then tested on classifiers it never touched.
Clean sequenceT = 30 frames1DINOself-supervisedobject-centricDeiTsupervisedclass-discriminativefrozen · never trained here2Fused saliencymax-pool across frames3Shadow generator + optimizerBlobStrip ★Sidealigned to attention · top-k loss0 queries to the target model4Adversarialsame mask, all frames5Target modelsCNNSTNEffB0ViTheld-out · black-boxnever queried during the attack

The attack is optimized entirely against the frozen attention maps. The classifiers it ultimately fools are only seen at evaluation time, which is what makes it a strict black-box attack.

The pipeline in five steps
1Read the clip

A 30-frame sign sequence is fed to two frozen vision transformers.

2Fuse attention

DINO and DeiT maps are max-pooled over time into one saliency heatmap.

3Guide the shadow

A top-k loss steers the shadow onto the most salient pixels, no target needed.

4Apply everywhere

One optimized mask is painted onto every frame for temporal consistency.

5Test unseen

The clip is scored against four classifiers the attack never queried.

Three Physically Grounded Shadows
A differentiable generator casts each shape from a few geometric parameters, darkening only luminance so the shadow reads as natural shading.
Blob radial

Cast by clouds and tree cover: a soft, diffuse patch.

θ = (cₓ, cᵧ, s)
Strip ★ strongest

Cast by poles, signposts, fences: a sharp directional band.

θ = (cₓ, cᵧ, α, s)
Side lateral

Cast by barriers and parked cars: an oblique, asymmetric shade.

θ = (cₓ, cᵧ, α, f)

All three are rendered in CIELAB and applied only to the luminance channel, L′ = L · (1 − γM), so hue and saturation stay untouched and a single intensity γ dials the shadow from barely visible to heavy.

What The Shadow Does On GTSRB
One attention-aligned shadow, no target access, and accuracy collapses, most of all on the transformers.
86 pts
frame accuracy lost on ViT
>90 pts
sequence accuracy lost (seq@50)
0
queries to the target model
4
unseen classifiers fooled
Accuracy drop at high intensity (γ = 0.8), StripShadow guided by DeiT frame   sequence (seq@50)
CNN
40.6 / 47.5
STN
46.9 / 47.5
EffB0
62.0 / 62.5
ViT
86.3 / 92.5

Bars are percentage-point drops from the clean baseline. The transformer (ViT) loses the most; the convolutional model resists best, matching the paper’s finding that global-attention models are the most exposed.

Why sequences fall harder than frames

Sequence accuracy (seq@50) counts a clip as recognized only if at least half of its 30 frames are right. Because a single mask is held steady across the whole clip, the sign is not just occasionally misread, it is lost for the entire approach. That is why sequence-level drops run ahead of frame-level drops, and why frame-only evaluations understate the real risk.

What wins, what breaks
StripShadow wins everywhere

The elongated band overlaps the high-attention regions on structured signs, and tops all four architectures.

DeiT beats DINO for aiming

Class-discriminative attention places shadows more effectively than object-centric attention.

Transformers are most exposed

ViT and DeiT fall hardest; the CNN, with local receptive fields, holds up the best.

No target access needed

Covering the top 50% of attention gives the strongest attacks, and even a plain L1 loss suffices.

Why It Matters
A realistic threat model

The attack uses no weights, gradients, or queries, only public frozen transformers. That mirrors what an attacker with no access to the deployed model could actually do.

Video defenses are not enough

Pipelines that filter flickering per-frame noise still fall to a stable, temporally coherent shadow. Frame-only benchmarks make these systems look safer than they are.

Nothing looks tampered

Shadows from poles, clouds, and barriers belong in ordinary driving scenes. There is no sticker or printed patch, so the sign never looks altered to a human.

About this page

The diagrams redraw the paper’s pipeline and its three shadow styles in this site’s style; the numbers are the high-intensity (γ = 0.8) StripShadow results guided by DeiT from the paper’s tables. Evaluation is on GTSRB with 30-frame sequences against four held-out classifiers (CNN, STN, EfficientNet-B0, ViT), with DINO and DeiT used only as frozen attention guides. The work is limited to GTSRB under simulated video; real driving validation and defenses are the stated next steps. Full method, ablations, and results are in the paper.

Abstract

“One physically plausible shadow, placed where a frozen vision transformer looks and held steady across all thirty frames, cuts traffic-sign accuracy by up to 86 points, without ever querying the model it fools.”

We propose SASA (Sequence-Aware Shadow Attack), a black-box adversarial framework that uses physically realistic, differentiable shadow patterns to deceive traffic sign recognition systems. Unlike prior image-based attacks, SASA targets video sequences by generating smooth, temporally consistent shadows that remain visually plausible and imperceptible to humans. Guided by attention maps from frozen vision transformers, SASA aligns shadow placement with semantically salient regions without querying the target model. Evaluated on the GTSRB dataset, SASA reduces classification accuracy by up to 86% and sequence-level accuracy by over 90% on black-box models, including CNNs and ViTs.
Type Conference paper
Venue 6th Workshop on Adversarial Machine Learning on Computer Vision: Safety of Vision-Language Agents (AdvML@CVPR)
Adversarial Attacks AI Security Computer Vision Traffic Sign Recognition
David Fernandez
Authors
David Fernandez
PhD Candidate in Computer Science

David Fernandez is a PhD candidate in Computer Science at Clemson University, working on safe, efficient, and explainable AI for safety-critical systems. His research spans perception, adversarial robustness, and on-device deployment of large foundation models, including LLMs and VLMs, with five first-authored publications on component-level explainability, zero-shot reasoning, and adversarial scenario analysis, alongside collaborative work on edge AI for industrial agentic systems. Much of this research is grounded in autonomous driving, where trustworthiness, latency, and robustness constraints are unforgiving, but the underlying methods transfer broadly to other high-stakes domains.

As a member of Clemson’s VIPR-GS Research Program, he develops hierarchical LLM reasoning frameworks and VLM evaluation systems for the U.S. Army’s Next Generation Combat Vehicle (NGCV) program, focusing on zero-shot reasoning and component-level explainability under real-world deployment constraints.

At BMW Group, he designs agentic AI systems for enterprise environments, building autonomous prompt optimization pipelines that enable continual agent improvement without model retraining and context-aware moderation frameworks that detect coordinated multi-turn adversarial attacks in production deployments.