Attention-Aware Temporal Adversarial Shadows on Traffic Sign Sequences
David Fernandez
Key Contributions & Takeaways
A self-driving car reads a traffic sign across a stream of frames as it drives closer, not from a single snapshot. This paper shows that a plain cast shadow, with no sticker or printed patch, can hold that sign at the wrong label for most of the approach. The trick is to evolve one shadow that falls exactly where the classifier is looking, then measure the attack over the whole clip rather than one lucky frame.
- The first shadow attack that works over a whole sequence. Prior attacks fool a single still frame. This casts one shadow whose scale grows across all 30 frames, like natural shading on an approaching sign, so it stays plausible while the classifier stays wrong frame after frame.
- It aims at the model’s attention, not just its label. A term based on DINO ViT attention maps steers the shadow onto the exact region the model relies on. That adds 11 to 18% attack success and, surprisingly, produces smaller, less visible shadows than a label-only attack.
- A metric built for video, not snapshots. Sequence-Level Attack Success (SL-ASR) counts an attack as a win only when at least τ of the 30 frames flip, capturing persistence that single-frame scores miss. On GTSRB it reaches 52.3 to 87.5%.
Earlier shadow attacks fool a sign in one still frame. But a car sees the same sign in a stream of frames as it drives closer, and a filter that compares frames can shrug off a one-off glitch.
Cast one shadow that grows across the whole sequence, like natural shading on an approaching sign. A genetic algorithm shapes it to both flip the label and land where the model looks.
The attack flips a sign across most of the sequence 52.3–87.5% of the time. Aiming at the model’s attention adds 11–18% and, oddly, makes the shadows smaller.
A candidate shadow is rendered across the sequence, scored by the CNN and DINO ViT, and updated through the dual loss until enough frames are misclassified.
The GA holds a population of shadow shapes and opacities and picks candidates to try.
The generator darkens only luminance and scales the shadow across all 30 frames.
A CNN reads the label; a DINO ViT reads where the model is looking.
Predictions and attention maps are compared to cached clean references for the sign.
One loss adds the classification error and the attention shift, weighted by λ.
Crossover and mutation breed the next generation, until enough frames flip.
attack success
at attention
1,306 sequences
are also stealthier
Turning the attention term on (λ = 0.5) beats the label-only attack (λ = 0) at every setting. Shown here at the τ = 17 threshold.
A lower τ asks fewer frames to flip, so success is easier; a higher τ demands a near-constant failure. The attention lift (Δ) holds up across the board.
| Frames to flip (τ) | Triangle shadow | Polygon shadow | ||||
|---|---|---|---|---|---|---|
| λ = 0 | λ = 0.5 | Δ | λ = 0 | λ = 0.5 | Δ | |
| 29 (strictest) | 40.4 | 52.3 | +11.9 | 45.5 | 56.9 | +11.4 |
| 27 | 50.6 | 64.2 | +13.6 | 55.1 | 68.3 | +13.2 |
| 17 (loosest) | 65.7 | 84.2 | +18.5 | 70.4 | 87.5 | +17.1 |
The surprise: guiding the shadow toward the model’s focus lets it be smaller. Because it lands on the few pixels that matter, it does not need to darken half the sign, so its L2 distance from the clean image is lower.
A large, obvious shadow darkens nearly half the sign.
A small patch on the key stroke does more damage, and hides better.
No sticker, no printed patch, just a shadow. It looks like ordinary shading, holds the wrong label across most of an approach, and so slips past defences that vote over several frames before acting.
Where a model looks is itself an attack surface. Steering the shadow onto the high-attention region makes the attack both stronger and smaller, so the map meant to explain a model can also be turned against it.
Single-frame scores flatter a model. Real perception runs on video, so robustness has to be judged over sequences. SL-ASR is a step toward measuring how long a failure actually lasts.
Evaluated on GTSRB: 1,306 traffic-sign sequences of 30 frames each across 43 classes, split 85/15 at the sequence level. The attacked classifier is a GTSRB-CNN (97.3% clean accuracy); attention comes from a fine-tuned DINO ViT-small (6 heads, final layer, 128×128 map). Shadows are searched by a genetic algorithm with early stopping. The diagrams above rebuild the paper’s Figures 1–4 in this site’s style; the exact tables and equations are in the PDF.
“A single moving shadow, shaped to fall exactly where the model looks, can flip a traffic sign's label across a whole video, not just one frame, up to 87.5% of the time.”