Look Where It Matters

Abstract

Overview

AwaRes processes a low-resolution global view by default. When additional detail is required, it invokes a structured tool-call to request only specific high-resolution sub-regions — then answers conditioned on both views.

Figure 1: Left: Given a low-resolution image, AwaRes uses tool-calling to request only the needed crops. Right: Accuracy vs. retained visual tokens — AwaRes achieves Qwen2.5-VL accuracy (80.3%) using only 36% of the visual tokens.

Main Results

AwaRes matches full-resolution performance while using only 36% of visual tokens, outperforming all efficient baselines across six benchmarks.

Table 1: RTR = fraction of visual tokens retained (lower is better). AwaRes matches full-resolution at 36% compute.

Performance vs. Wall Clock Time

Figure 3: AwaRes achieves sub-second latency across benchmarks via short tool calls, while VisionThink's reasoning traces increase decoding time (4.3s vs. 0.6s on ChartQA).

Qualitative Results

Examples of AwaRes adaptively selecting crops across charts, documents, and natural images. Each shows the question, tool call, low-res input, and retrieved high-res crop.

Chart: Zooming into center trend-line data points to compute the median value (14.5) of the red graph.

Chart: Cropping the bottom to compare median values between China and Angola trend lines.

Natural image: Zooming into the bottom of a wine bottle to read the label — "Amarone".

Natural scene: Cropping the right side of a garden to identify the black color of a broom handle.

Scene understanding: Cropping the bottom of a rainy street to count 2 pedestrians within 20 meters.

Object counting: Cropping the bottom of a parking lot to identify and count 2 traffic cones.

Method

AwaRes learns a coupled-decision policy (CDP) that jointly decides whether additional resolution is needed and where to acquire it, by selecting a subset of crops from a predefined candidate set.

Data Curation Pipeline

Figure 2: Each sample is processed at two resolutions. An LLM judge determines sufficiency; insufficient cases are routed to an oracle for crop localization, producing multi-turn tool-use trajectories.

Crop Annotation Examples

The oracle grounding model localizes answer regions, which are mapped to discrete crops for training.

Example 1: "What website is being advertised on the plane?" — The oracle localizes the text region; the selected crop reveals "Jetstar.com".

Example 2: "What is the Grand Total for Net Block As of 31.3.2012?" — The oracle targets the relevant table cell; the crop enables reading "52708.86".

Training Pipeline

AwaRes is trained in two stages: supervised cold-start (SFT) to teach the tool-calling protocol, followed by multi-turn GRPO to optimize the accuracy–efficiency trade-off.

Stage 1 Cold-Start SFT

We fine-tune on a mixture of direct-answer trajectories (single turn, low-res only) and tool-call-then-answer trajectories (two turns with crop retrieval). This teaches the coupled-decision policy (CDP)—jointly deciding whether to escalate and where to crop—and yields a reference policy $\pi_{\text{ref}}$ for the RL stage.

We minimize a weighted negative log-likelihood over assistant tokens:

where $h_t$ is the dialogue history at step $t$ and $w_t$ is a per-token weight. We set $w_t{=}5$ for the tool-call turn (1 otherwise) to stabilize learning of the first-turn decision—this single change boosts accuracy from 77.9→79.7 and achieves 100% tool-call formatting validity. We use trajectory-level SFT, optimizing the full two-turn interaction as a single sample. After SFT, the model is frozen as $\pi_{\text{ref}}$.

Stage 2 Multi-Turn GRPO

The SFT model has learned the tool-calling protocol but still uses it inefficiently, often requesting crops when the low-resolution view already suffices. We apply GRPO on full multi-turn interactions to improve tool usage and overall efficiency, using a composite reward:

$R_{\text{ans}}$ is the cosine similarity between sentence-transformer embeddings of the predicted and ground-truth answers. The tool cost is asymmetric—missing a necessary crop is penalized more heavily than making an unnecessary request:

For each prompt, we sample $G{=}8$ trajectories and compute group-relative advantages. The policy is optimized with a PPO-style clipped objective regularized toward $\pi_{\text{ref}}$:

GRPO shifts the tool-usage distribution closer to the oracle annotations while also discovering new crop strategies not present in the supervision. The resulting policy matches full-resolution accuracy (80.3 vs. 80.5) while using only 36% of the visual tokens.

Full training hyperparameters and ablation details are provided in the paper.

Citation

If you find AwaRes useful in your research, please consider citing:

@article{shabtay2026look,
title={Look Where It Matters: High-Resolution Crops Retrieval for Efficient VLMs},
author={Shabtay, Nimrod and Kimhi, Moshe and Spector, Artem and Haray, Sivan and Rivlin, Ehud and Baskin, 
Chaim and Giryes, Raja and Schwartz, Eli},
journal={arXiv preprint arXiv:2603.16932},
year={2026}
}