Visual Prompting is an interaction technique for computer vision models — where users provide visual cues (points, boxes, scribbles, or reference images) as inputs to guide the model's prediction, rather than relying solely on text or fixed classes.

What Is Visual Prompting?

• Definition: Using visual signals to specify the target or task.
• Examples:
• Spatial: Drawing a box around a car to track it.
• Example-based: Showing an image of a screw and asking "Find all of these".
• Inpainting: Masking an area to say "fill this space".

Why Visual Prompting Matters

• Precision: Text ("the red car") is ambiguous; a click on the pixel is precise.
• New Tasks: Can define tasks that are hard to describe in words (e.g., "count cells that look abnormal like this one").
• CV-Native: Aligns the input modality (visual) with the task modality (visual).

Models:

• SAM: Accepts points/boxes.
• SEEM (Segment Everything Everywhere All at Once): Accepts audio, visual, and text prompts.
• Visual Prompting (VP): Learning pixel-level perturbations to adapt frozen models to new tasks.

Visual Prompting is the mouse-click of the AI era — allowing intuitive, non-verbal communication with intelligent visual systems.