Attention-Aware Temporal Adversarial Shadows on Traffic Sign Sequences

Jun 1, 2025· Pedram MohajerAnsari, Amir Salarpour, David FernandezDavid Fernandez , Cigdem Kokenoz, Bing Li, Mert D. Pesé
PDF Code DOI Cite

Key Contributions & Takeaways

A self-driving car reads a traffic sign across a stream of frames as it drives closer, not from a single snapshot. This paper shows that a plain cast shadow, with no sticker or printed patch, can hold that sign at the wrong label for most of the approach. The trick is to evolve one shadow that falls exactly where the classifier is looking, then measure the attack over the whole clip rather than one lucky frame.

  • The first shadow attack that works over a whole sequence. Prior attacks fool a single still frame. This casts one shadow whose scale grows across all 30 frames, like natural shading on an approaching sign, so it stays plausible while the classifier stays wrong frame after frame.
  • It aims at the model’s attention, not just its label. A term based on DINO ViT attention maps steers the shadow onto the exact region the model relies on. That adds 11 to 18% attack success and, surprisingly, produces smaller, less visible shadows than a label-only attack.
  • A metric built for video, not snapshots. Sequence-Level Attack Success (SL-ASR) counts an attack as a win only when at least τ of the 30 frames flip, capturing persistence that single-frame scores miss. On GTSRB it reaches 52.3 to 87.5%.
publications

01  ·  The Problem

Earlier shadow attacks fool a sign in one still frame. But a car sees the same sign in a stream of frames as it drives closer, and a filter that compares frames can shrug off a one-off glitch.

02  ·  The Approach

Cast one shadow that grows across the whole sequence, like natural shading on an approaching sign. A genetic algorithm shapes it to both flip the label and land where the model looks.

03  ·  The Finding

The attack flips a sign across most of the sequence 52.3–87.5% of the time. Aiming at the model’s attention adds 11–18% and, oddly, makes the shadows smaller.

The Framework
One optimization loop that evolves a shadow until it fools the model across the whole sequence.
6Update parametersearly stop when ≥ τ of 30 frames are misclassifiedDINO ViTAttention Extractorclean focusshadowedRaw GTSRBGTSRB Loader& Preprocessorcrop · resize · sequence120Clean sequenceGenetic AlgorithmOptimizerpopulation · crossover · mutation1candidate shadowparams (shape, opacity)Shadow GeneratorLAB luminance · polygon / triangletemporal scaling over frames2120Shadowed sequence3Traffic Sign ClassifierCNNclean acc. 97.3% · 43 classes45Multi-ObjectiveLossClassification errorAttention disruptionL = L​cls− λ·Lattn

A candidate shadow is rendered across the sequence, scored by the CNN and DINO ViT, and updated through the dual loss until enough frames are misclassified.

The loop, step by step
1Propose a shadow

The GA holds a population of shadow shapes and opacities and picks candidates to try.

2Render it over time

The generator darkens only luminance and scales the shadow across all 30 frames.

3Look through two lenses

A CNN reads the label; a DINO ViT reads where the model is looking.

4Score against clean

Predictions and attention maps are compared to cached clean references for the sign.

5Combine both goals

One loss adds the classification error and the attention shift, weighted by λ.

6Update and repeat

Crossover and mutation breed the next generation, until enough frames flip.

How the Shadow Is Built
A plausible cast shadow that grows over time and is aimed at the model’s focus.
The Core Idea

One shadow, the same shape and opacity throughout, is scaled up frame after frame to mimic a sign coming into view. It stays subtle, yet it holds the wrong label for most of the clip.

frame 1
frame 12
frame 24
frame 30
Classifier reads
class 36 → class 40
“go straight” misread as “turn right”, frame after frame.
Two Shadow Shapes, Aimed at the Focus

Shadows darken only the luminance channel in LAB space, so colour stays natural. Two shapes are searched, and both are steered onto the bright, high-attention regions the model relies on. Soft edges, motion blur, and brightness normalisation keep them looking like real cast shadows.

Polygon 4 corners
30

Covers several angles at once; the strongest attacker overall.

Triangle 3 corners
30

Mimics a wedge cast by a roadside pole or structure.

opacity α ∈ [0.1, 0.7] LAB luminance only Gaussian smoothing + motion blur temporal scale 0.6 → 1.0
The dual objective
// averaged over the T = 30 frames
Ltotal = Lcls − λ · Lattn
L​cls

Fool the label. Push the CNN to lose confidence in the true class.

L​attn

Move the gaze. Maximise the gap between the clean and shadowed DINO ViT attention maps.

λ

Balances the two. λ = 0.5 is the full attack; λ = 0 ablates attention.

A sequence-level metric · SL-ASR

An attack counts as a success only if at least τ of the T = 30 frames are misclassified, so it must fail persistently, not just once.

misclassified frame correct frame

Thresholds tested: τ = 17, 27, 29 out of 30.

Results
Evaluated on GTSRB, against a clean classifier that is right 97.3% of the time.
87.5%
best sequence-level
attack success
+11–18%
lift from aiming
at attention
30
frames per sequence,
1,306 sequences
↓ L2
attention shadows
are also stealthier
A · Aiming at Attention Pays Off

Turning the attention term on (λ = 0.5) beats the label-only attack (λ = 0) at every setting. Shown here at the τ = 17 threshold.

Polygon shadow +17.1 pts
λ = 0
70.4%
λ = 0.5
87.5%
Triangle shadow +18.5 pts
λ = 0
65.7%
λ = 0.5
84.2%
B · SL-ASR at Every Threshold (Table 1)

A lower τ asks fewer frames to flip, so success is easier; a higher τ demands a near-constant failure. The attention lift (Δ) holds up across the board.

Frames to flip (τ)Triangle shadowPolygon shadow
λ = 0λ = 0.5Δλ = 0λ = 0.5Δ
29 (strictest)40.452.3+11.945.556.9+11.4
2750.664.2+13.655.168.3+13.2
17 (loosest)65.784.2+18.570.487.5+17.1
Values are SL-ASR (%). Δ = points gained by adding attention supervision. Highlighted cell is the peak, 87.5%.
C · Stronger and Less Visible at Once

The surprise: guiding the shadow toward the model’s focus lets it be smaller. Because it lands on the few pixels that matter, it does not need to darken half the sign, so its L2 distance from the clean image is lower.

50
Label only λ = 0

A large, obvious shadow darkens nearly half the sign.

50
Attention guided λ = 0.5

A small patch on the key stroke does more damage, and hides better.

Why It Matters
Threat

No sticker, no printed patch, just a shadow. It looks like ordinary shading, holds the wrong label across most of an approach, and so slips past defences that vote over several frames before acting.

Attention

Where a model looks is itself an attack surface. Steering the shadow onto the high-attention region makes the attack both stronger and smaller, so the map meant to explain a model can also be turned against it.

Evaluation

Single-frame scores flatter a model. Real perception runs on video, so robustness has to be judged over sequences. SL-ASR is a step toward measuring how long a failure actually lasts.

Evaluated on GTSRB: 1,306 traffic-sign sequences of 30 frames each across 43 classes, split 85/15 at the sequence level. The attacked classifier is a GTSRB-CNN (97.3% clean accuracy); attention comes from a fine-tuned DINO ViT-small (6 heads, final layer, 128×128 map). Shadows are searched by a genetic algorithm with early stopping. The diagrams above rebuild the paper’s Figures 1–4 in this site’s style; the exact tables and equations are in the PDF.

Abstract

“A single moving shadow, shaped to fall exactly where the model looks, can flip a traffic sign's label across a whole video, not just one frame, up to 87.5% of the time.”

We present a framework for black-box adversarial attacks on traffic signs using dynamic, temporally coherent shadows. Unlike prior work that focuses on single-image attacks or relies on conspicuous physical artifacts, our method operates over entire image sequences, mimicking realistic scenarios where a traffic sign is observed from varying distances. We design a non-differentiable shadow generator that casts a single fixed-shape, fixed-opacity shadow whose spatial scale evolves over time to simulate natural environmental shading. A genetic algorithm is used to optimize shadow geometry and opacity, guided by a dual loss that jointly maximizes classification error and visual attention disruption. Attention perturbation is measured using DINO ViT attention maps between clean and shadowed frames. Evaluated on the GTSRB dataset, our method achieves a sequence-level attack success rate (SL-ASR) — defined as the percentage of sequences where at least τ out of T frames are misclassified — ranging from 52.3% to 87.5%, depending on the threshold and shadow type. Furthermore, incorporating attention supervision yields consistent SL-ASR gains of 11–18% over purely classification-based attack.
Type Conference paper
Venue IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW 2025), pp. 3600–3608
Adversarial Attacks Computer Vision Traffic Sign Recognition