Budget-Aware Adaptive Adversarial Patches for Black-Box Object Detection
David Fernandez
,
Mert D. PeséKey Contributions & Takeaways
An adversarial patch is a small pattern that can make a detector lose track of what it sees. Creating one without access to the model’s internals requires repeated queries, and simply making the patch larger can hide how expensive or conspicuous an attack is. PatchBandit adapts placement, texture, and size together, measuring success alongside those costs.
- One loop controls where, what, and how large. Contextual Thompson sampling chooses a location, Natural Evolution Strategies updates the pixels using detector scores, and the patch grows only when progress stalls. No model weights or gradients are required.
- Success is measured on the plain image. The original top-class confidence must fall by at least 0.25 and end at 0.10 or below. The baseline achieves 77.5% strict suppression on YOLOv5, 89.7% on Faster R-CNN, and 59.1% on YOLOS; robustness across transformed views is assessed separately.
- Faster growth can save queries without enlarging the typical patch. On YOLOv5, GP5 reduces median queries from 1,079 to 529, with nearly unchanged success (77.5% → 77.7%) and the same 8.32% median image area. Other detectors show different trade-offs.
Every query costs. Every pixel shows.
A patch can fool an object detector, but its practical cost depends on how many times the attacker queries the model and how much of the image the patch covers.
Adapt three decisions together
PatchBandit searches for a useful location, improves the pattern, and increases its size only after progress stalls, all through a score-based black-box interface.
Detector architecture changes the trade-off
The baseline suppresses the original top class in 77.5–89.7% of tested cases on the two CNN detectors and 59.1% on the transformer detector.
Same object. Different prediction.
A localized pattern changes the detector’s output while the object remains in view.
Physical example
160 px patchDigital example
60 px patchInside PatchBandit
A detector returns boxes, classes, and confidence scores. Its weights and gradients stay hidden.
Choose where
A contextual Thompson-sampling policy explores candidate grid cells. Features describe proximity to annotated objects, texture variation, and image borders.
Improve the pattern
Natural Evolution Strategies tests pixel perturbations and uses the resulting confidence changes to update the patch, without backpropagating through the detector.
Grow only if stalled
If the reward stops improving for a set number of iterations, the patch moves to the next size and the placement grid is rebuilt. Otherwise, its footprint stays small.
What counts as success?
The target is the highest-confidence class in the clean image. On the patched, untransformed image, both conditions must hold:
Confidence is the maximum score for that class across detected boxes, or zero if the class is absent. Suppressing that class does not mean every object disappears. Expectation over Transformation (EOT) tests transformed views separately; an EOT average is not a replacement for this plain-image test.
Success, footprint, and query cost
Three detector families expose different vulnerabilities. These baseline results use the paper’s strict plain-image criterion.
YOLOv5
77.5%One-stage CNN · strict successFaster R-CNN
89.7%Two-stage CNN · strict successYOLOS
59.1%ViT transformer · strict success| Detector | Strict success | Median patch area | Median queries |
|---|---|---|---|
| YOLOv5 | 77.5% | 8.32% | 1,079 |
| Faster R-CNN | 89.7% | 0.92% | 177 |
| YOLOS | 59.1% | 8.32% | 290 |
Area is a percentage of the image. These numbers are reproduced from Table 2; the fixed-size ablation in Table 3 reports different query counts for the CNN detectors.
On YOLOv5, earlier growth roughly halves queries
GP5 shortens the patience before increasing patch size. It retains almost the same success rate and median footprint as the baseline.
The trade-off changes on YOLOS: GP5 cuts median queries from 290 to 47.5 and raises success from 59.1% to 60.8%, but increases median area from 8.32% to 14.79%.
Explore the trade-off
Higher success, smaller patch, fewer queries: no single configuration wins every measure.
YOLOv5
↑ More success · ← Less image areaBaseline
- Success
- 77.5%
- Image area
- 8.32%
- Queries
- 1,079
GP5
- Success
- 77.7%
- Image area
- 8.32%
- Queries
- 529
Fixed 160 px
- Success
- 71.1%
- Image area
- 14.80%
- Queries
- 133
Baseline and GP5 nearly overlap: the same 8.32% footprint and almost identical success, but GP5 uses about half the median queries. Fixed 160 px uses fewer queries with a larger footprint and lower success.
From digital optimization to a printed patch
A small print–capture pilot tests a bottle patch under changes in distance, viewpoint, and object instance.
YOLOv5 detected the bottle with confidence above 0.90 in clean shots. With the printed patch, the reported conditions produced different kinds of errors:
Original bottle
40 cm · 0°Original bottle
50 cm · 10°New bottle #1
54 cm · 0°New bottle #2
54 cm · 0°Original schematic illustrations of the four conditions in Table 4. Predictions, scores, distances, and viewing angles are reported measurements; bottle shapes, patch textures, and box positions are illustrative.
These are individual pilot observations, not a physical success-rate benchmark. A wrong bounding box or class change is distinct from satisfying the digital strict-suppression test.
What the evidence supports—and its limits
Score access and scene context
The attack needs detector confidence scores. Placement features also use ground-truth boxes; a label-only API or an unannotated scene does not provide the same information.
Measured suppression, not universal failure
The digital metric targets the original top class. It does not establish that all detections vanish, or that one universal patch achieves the same rates across arbitrary scenes.
Preliminary physical transfer
The bottle pilot supports transfer in the tested conditions. Wider claims about lighting, long distances, and deployment environments need broader physical evaluation.
The practical lesson: evaluate adversarial patches against a joint budget of queries and visible area, and keep plain-image suppression separate from robustness across transformations.
Source: the paper, arXiv:2606.18318v1, Sections 3–4 and Tables 2–4. The DOI and BibTeX refer to the arXiv version; the venue is listed as accepted to ICIP 2026.
“PatchBandit learns where to place a patch, how to change its pixels, and when to enlarge it, using only detector outputs. Its success depends on both the query budget and the space the patch occupies.”
David Fernandez is a PhD candidate in Computer Science at Clemson University and an AI Researcher at BMW Group. His research focuses on safe, efficient, and explainable AI, spanning large language models, vision-language models, agentic systems, and adversarial robustness. His 10 papers include five first-authored studies on component-level explainability, zero-shot reasoning, and adversarial scenario analysis.
At BMW Group, he develops enterprise AI agents, automated prompt optimization, and defenses against multi-turn adversarial attacks. His work also includes edge AI for industrial agentic systems.
Through Clemson’s Virtual Prototyping of Ground Systems (VIPR-GS) program, he develops explainable reasoning and multimodal evaluation systems for the U.S. Army’s Next Generation Combat Vehicle program.