LandMarkVLN

Do Navigation VLA Models Ground What They Are Told?An Empirical Audit of Landmark-Vision Alignment in VLN

Josh Qixuan Sun

Xiaoying Xing

Alibaba Group

xiaoyingxy0505@gmail.com

Yurun Chen

Huaiyuan Weng

Wentao Zhu

Mark Crowley

WACV 2027 · Datasets Track

Code Explore the audit BibTeX

7,228evaluation samples

8navigation VLA models

2complementary audit views

The question behind the benchmark

Correct actions do not establish landmark grounding.

Two predictions can agree with the reference action while their visual attribution tells different stories.

InstructionTurn slight left and move towards the black chair.

Landmark-localized attributionAttribution misses the landmark

Attribution focuses on the instructed black chair, with an acceptable local action.

Attribution misses the instructed black chair, despite an acceptable local action.

Drag to compare the two attribution views from the paper’s motivating example. Both actions are correct; only one attribution map localizes the named landmark.

Overview

Vision-and-language navigation requires an agent to connect route instructions with what it sees. Standard navigation metrics measure how well it moves, but do not reveal which visual evidence supports its actions.

LandMarkVLN connects each instructed landmark to current and historical observations, masks of the same physical object, and local reference actions. We use it to audit eight released navigation VLA models from two complementary perspectives: where visual attribution falls, and whether editing the landmark changes the output more than a matched control.

The central finding: acceptable local actions and broad visual responsiveness are not, by themselves, reliable evidence that a model relies on the instructed landmark.

Two views of landmark–vision alignment

Localization and output sensitivity answer different questions. The audit reports both, alongside local action agreement.

One landmark, linked through time

Each sample joins a navigation instruction and a singular landmark phrase with a current RGB observation, recent observation history, temporally aligned masks of the same physical landmark, and a local expert-action horizon. The dataset is constructed from the continuous-environment releases of R2R and RxR, using English instructions.

Instruction + landmarkCurrent + historical viewsSame-instance masksLocal reference actions

Three stages of construction

Stage	What is checked	Output
Text-only selection	The phrase denotes one resolvable physical object instance.	118,706 candidate identities
Current-view grounding	The target is visible, unique, and geometrically usable.	19,742 current-grounded identities
Temporal verification	Visibility and same-instance identity are checked across history.	7,228 intervention-ready samples

The current-view stage evaluates 350,660 candidate–frame pairs generated from the text-selected identities. Counts across stages use different units.

What the audit finds

The evaluation covers OneVLA, InternVLA-N1, OmniVLA, Uni-NaVid, FantasyVLN, StreamVLN, VAMOS, and TIC-VLA, spanning current-only and history-aware inputs, and discrete and continuous actions.

2 / 60

Localization rarely tracks acceptable actions.

Two of 60 vision-action evaluations support a positive association between landmark localization and acceptable local actions after model-wise Holm correction.

23 / 23

Models respond to broad visual changes.

Every evaluated intervention configuration shows a positive mean response to the full-frame edit.

6 / 23

Landmark-specific sensitivity has limited support.

Six configurations support a larger response to landmark edits than to matched non-landmark control edits after Holm correction within each native output family.

These denominators count evaluations and configurations, not individual models. The 60 vision-action evaluations combine 20 model–temporal-scope pairs with three attribution estimators. The intervention analysis uses 23 configurations, including separate output branches for InternVLA-N1.

A shared scene, different diagnoses

Current-scope attention × gradient attribution for eight navigation VLA models on the same black-chair instruction, with local action and landmark-localization diagnostics. — The same instruction and observation realize all four combinations of action acceptability and landmark localization across eight models. Heatmaps are normalized independently within each model. View full-size figure ↗

Evaluate the evidence behind the action. Navigation performance, visual attribution, and matched intervention responses provide complementary information; none should be treated as interchangeable evidence of landmark grounding.

Code and reproducibility

The official LandMarkVLN repository provides CPU analysis code and frozen, sample-level measurements for all eight models. It reproduces the 60 vision-action panels, 23 matched intervention analyses, 23 full-frame response summaries, and 23 attribution–intervention correlations.

See the reproducibility guide for validation, tests, and analysis commands. The release starts from model measurements; the dataset-construction and GPU inference pipelines, raw images, instruction text, pixel masks, and full attribution maps are not included.

Citation

@inproceedings{sun2027landmark,
  title = {Do Navigation VLA Models Ground What They Are Told? An Empirical Audit of Landmark-Vision Alignment in VLN},
  author = {Josh Qixuan Sun and Xiaoying Xing and Yurun Chen and Huaiyuan Weng and Wentao Zhu and Mark Crowley},
  booktitle = {Winter Conference on Applications of Computer Vision},
  year = {2027}
}