Budget-Aware Adaptive Adversarial Patches for Black-Box Object Detection

Tue, 16 Jun 2026 00:00:00 +0000

01 · The problem

Every query costs. Every pixel shows.

A patch can fool an object detector, but its practical cost depends on how many times the attacker queries the model and how much of the image the patch covers.

02 · The approach

Adapt three decisions together

PatchBandit searches for a useful location, improves the pattern, and increases its size only after progress stalls, all through a score-based black-box interface.

03 · The finding

Detector architecture changes the trade-off

The baseline suppresses the original top class in 77.5–89.7% of tested cases on the two CNN detectors and 59.1% on the transformer detector.

Same object. Different prediction.

A localized pattern changes the detector’s output while the object remains in view.

Physical example

160 px patch

Clean view

Top predictionBicycle · 0.95

With patch

Top predictionMotorbike · 0.36

Digital example

60 px patch

Clean view

Top predictionBoat · 0.91

With patch

Top predictionCat · 0.92

Original schematic drawings of the examples reported in Figure 1. Labels, confidence scores, and nominal patch sizes are from the paper; scene geometry, boxes, and patch patterns are illustrative and not to scale. These top-label changes do not by themselves establish strict suppression.

Inside PatchBandit

A detector returns boxes, classes, and confidence scores. Its weights and gradients stay hidden.

Choose where

A contextual Thompson-sampling policy explores candidate grid cells. Features describe proximity to annotated objects, texture variation, and image borders.
Improve the pattern

Natural Evolution Strategies tests pixel perturbations and uses the resulting confidence changes to update the patch, without backpropagating through the detector.
Grow only if stalled

If the reward stops improving for a set number of iterations, the patch moves to the next size and the placement grid is rebuilt. Otherwise, its footprint stays small.

Repeat with feedback: update the placement policy from observed rewards, then search again. Stop at strict success or the query cap.

Conceptual flow of the method in Section 3 and Algorithm 1. The digital attack optimizes each image separately; it does not assume access to detector parameters.

What counts as success?

The target is the highest-confidence class in the clean image. On the patched, untransformed image, both conditions must hold:

Confidence drop ≥ 0.25 and Remaining confidence ≤ 0.10

Confidence is the maximum score for that class across detected boxes, or zero if the class is absent. Suppressing that class does not mean every object disappears. Expectation over Transformation (EOT) tests transformed views separately; an EOT average is not a replacement for this plain-image test.

Success, footprint, and query cost

Three detector families expose different vulnerabilities. These baseline results use the paper’s strict plain-image criterion.

YOLOv5

77.5%One-stage CNN · strict success

Faster R-CNN

89.7%Two-stage CNN · strict success

YOLOS

59.1%ViT transformer · strict success

Baseline PatchBandit (ctx-strict), Table 2
Detector	Strict success	Median patch area	Median queries
YOLOv5	77.5%	8.32%	1,079
Faster R-CNN	89.7%	0.92%	177
YOLOS	59.1%	8.32%	290

Area is a percentage of the image. These numbers are reproduced from Table 2; the fixed-size ablation in Table 3 reports different query counts for the CNN detectors.

On YOLOv5, earlier growth roughly halves queries

GP5 shortens the patience before increasing patch size. It retains almost the same success rate and median footprint as the baseline.

Baseline1,079 queries

GP5 · faster growth529 queries

YOLOv5, Table 2: median queries fall by approximately 51%; strict success changes from 77.5% to 77.7%, with median patch area unchanged at 8.32%.

The trade-off changes on YOLOS: GP5 cuts median queries from 290 to 47.5 and raises success from 59.1% to 60.8%, but increases median area from 8.32% to 14.79%.

Explore the trade-off

Higher success, smaller patch, fewer queries: no single configuration wins every measure.

YOLOv5

↑ More success · ← Less image area

Baseline

Success: 77.5%
Image area: 8.32%
Queries: 1,079

GP5

Success: 77.7%
Image area: 8.32%
Queries: 529

Fixed 160 px

Success: 71.1%
Image area: 14.80%
Queries: 133

Baseline and GP5 nearly overlap: the same 8.32% footprint and almost identical success, but GP5 uses about half the median queries. Fixed 160 px uses fewer queries with a larger footprint and lower success.

Recreated from Table 2. Each marker represents a configuration; shapes and colors identify it, and labels report median queries. The small image tiles show median patch area to scale, with illustrative placement and texture. All detector views use the same axes.

From digital optimization to a printed patch

A small print–capture pilot tests a bottle patch under changes in distance, viewpoint, and object instance.

YOLOv5 detected the bottle with confidence above 0.90 in clean shots. With the printed patch, the reported conditions produced different kinds of errors:

01 · Frontal

Original bottle

40 cm · 0°

Top predictionChair · 0.59Misclassified

02 · Tilted

Original bottle

50 cm · 10°

Top predictionTV monitor · 0.35Misclassified

03 · Transfer

New bottle #1

54 cm · 0°

Top predictionBottle · 0.41Wrong box / low confidence

04 · Transfer

New bottle #2

54 cm · 0°

Top predictionBottle · 0.81Wrong box

Original schematic illustrations of the four conditions in Table 4. Predictions, scores, distances, and viewing angles are reported measurements; bottle shapes, patch textures, and box positions are illustrative.

These are individual pilot observations, not a physical success-rate benchmark. A wrong bounding box or class change is distinct from satisfying the digital strict-suppression test.

What the evidence supports—and its limits

Score access and scene context

The attack needs detector confidence scores. Placement features also use ground-truth boxes; a label-only API or an unannotated scene does not provide the same information.

Measured suppression, not universal failure

The digital metric targets the original top class. It does not establish that all detections vanish, or that one universal patch achieves the same rates across arbitrary scenes.

Preliminary physical transfer

The bottle pilot supports transfer in the tested conditions. Wider claims about lighting, long distances, and deployment environments need broader physical evaluation.

The practical lesson: evaluate adversarial patches against a joint budget of queries and visible area, and keep plain-image suppression separate from robustness across transformations.

Source: the paper, arXiv:2606.18318v1, Sections 3–4 and Tables 2–4. The DOI and BibTeX refer to the arXiv version; the venue is listed as accepted to ICIP 2026.

Black-Box Optimization | David Fernandez

Budget-Aware Adaptive Adversarial Patches for Black-Box Object Detection

Every query costs. Every pixel shows.

Adapt three decisions together

Detector architecture changes the trade-off

Physical example

Digital example

Choose where

Improve the pattern

Grow only if stalled

What counts as success?

YOLOv5

Faster R-CNN

YOLOS

On YOLOv5, earlier growth roughly halves queries

YOLOv5

Baseline

GP5

Fixed 160 px

Original bottle

Original bottle

New bottle #1

New bottle #2

Score access and scene context

Measured suppression, not universal failure

Preliminary physical transfer