acceptodds
Under review as a conference paper at ICLR 2027

AnchorNav: Long-Horizon 3D Game Navigation with Persistent Memory and Waypoint-Conditioned Control

Abstract

Recent advances in multimodal agents have steadily improved autonomous game-playing, yet reliable long-range navigation in photorealistic 3D games remains challenging due to open terrain, extended horizons, and the absence of privileged localization. Here we present AnchorNav, a *Vision-Language-Navigation* framework built around two complementary anchors: a local visual anchor for grounding route-level decisions into short-horizon actions, and a persistent global anchor for maintaining spatial consistency over time. The local anchor projects the next navigation target into the current view as a 2D waypoint, which guides a waypoint-conditioned visuomotor controller together with the language instruction. The global anchor is provided by a persistent topological memory, where visual place recognition periodically re-anchors visual odometry to previously established locations. In photorealistic 3D games, AnchorNav effectively completes long-horizon navigation and remains competitive on RxR under the VLN-CE protocol.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language-Navigation, Long-horizon, 3D game

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_i9JJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AnchorNav studies long-horizon navigation from game images and language instructions. It builds a landmark-and-route map from gameplay videos, uses place recognition to correct online position estimates, and projects the next target into the current image. A local controller follows this waypoint together with a language sub-instruction and produces keyboard–mouse actions. The controller is trained on 350 hours of gameplay. Experiments cover long-horizon traversal and short instruction following in Black Myth: Wukong, together with an evaluation on RxR.

Critical strengths
  • The problem is useful, and the system design is clear. Long-horizon navigation requires both remembering visited locations and translating route instructions into actions in the current view. The map and image-space waypoint address these two needs.
  • The long-horizon experiment demonstrates system usability. The full system completes the spawn-to-boss traversal in five of eight runs.
  • The evaluation includes a public benchmark with a strong reported success rate. On RxR val-unseen, the paper reports a 61.65% success rate, higher than the reference results listed in Table 3.
Critical weaknesses
  1. Sections 3.2–3.3 select the next target through graph search and project its 3D position into the image. However, Section 4.3 states that the graph contains no nodes for the RxR validation scenes. The paper does not explain how the target's 3D position and image-space waypoint are generated at test time in this setting.
  2. AnchorNav uses a map built from offline gameplay videos, while the baselines receive map prompts and can consult the in-game map. The differences in coverage, available information, and construction cost are not quantified. Table 1 therefore does not establish whether the advantage persists when the methods receive the same environmental information.
  3. The game-navigation metrics are not sufficiently specified. The paper does not clearly specify the calculation procedures for PCR and RVR or the arrival criterion for OSR in the game experiments.
  4. Table 1 reports only the Chapter 1 navigation task, with eight runs per method. The evidence is insufficient to establish reliability across a broader range of long-horizon tasks.
  5. The long-horizon experiment lacks existing navigation-method baselines. Table 1 compares only four zero-shot general-purpose VLMs. The retrained DualVLN and ActiveVLN baselines in Table 2 are not evaluated on the long-horizon task. The results demonstrate an advantage over these VLMs but do not establish an advantage over existing navigation methods.
Questions for authors
  1. Without stored nodes for the current RxR validation scene, how are the next target's 3D position and image-space waypoint generated at test time?
  2. Section 4.2 reports 250 trials per difficulty, but multiplying 250 by the reported success rate of 53.8% gives 134.5 successes, which is not an integer. How was this success rate calculated?
  3. Are the VLM baselines in Table 1 allowed to build, update, or query a topological map? Can at least one baseline receive the same landmark and route information and query interface as AnchorNav for a navigation comparison?
  4. How are PCR, RVR, and game OSR calculated?
  5. How much video is used for map construction, and how long does processing it take?
  6. Please specify each VLM baseline's prompt, observation history and memory management, action interface, and available tools in Table 1. What reasoning settings, token budgets, and model-call budgets are used?
  7. Can at least one navigation baseline already retrained for Table 2 be evaluated on the long-horizon task in Table 1, with its adaptation for long-horizon goals and action execution specified?
8 comments
AuthorsOpenReview ↗

Response to Reviewer i9JJ(1/2)

We thank Reviewer i9JJ for the valuable comments. We have included examples of actual code execution in the Supplementary Material and revised the manuscript according to the comments. Please find our response below.

Q1.Without stored nodes for the RxR validation scene, how are the next target's 3D position and image-space waypoint generated at test time?

For a fair comparison on RxR, we do not provide our model with topological memory or stored 3D node coordinates. At test time, the global-anchor LLM selects the next visual target from the task instruction and current observation, then outputs a language sub-instruction and the target’s image-space coordinates. For example, when the agent must leave a room, the planner places the waypoint on the visible door. The waypoint is then rendered directly onto the current frame and passed, together with the sub-instruction, to the local controller. The controller subsequently issues VLN-CE atomic actions, such as MOVE_FORWARD and TURN_RIGHT.

Q2.How was the Section 4.2 success rate of 53.8% calculated, given that 250 trials times this rate is not an integer?

Thank you for pointing out this issue; the number of trials reported in the paper is a typo. For each difficulty level, we used 50 tasks and tested each task 10 times, i.e., 500 trials per difficulty level (1,500 trials across the three difficulty levels), rather than the 250 trials per difficulty stated in the paper. With 500 trials per difficulty, the reported success rate of 53.8% corresponds to 269 successful trials, which is an integer. This has been corrected in the paper.

Q3.Are the Table 1 VLM baselines allowed to build, update, or query a topological map, and can one receive the same landmark and route information as AnchorNav?

The geometry-guided topological memory is a contribution of this work and is not available to the four general-purpose VLM baselines in Table 1; they neither build, update, nor query this graph. Instead, we provide each baseline with position, route, and destination information in its prompt. Specifically, we annotate the route and destination on a saved game map and include it in the VLM’s context. During execution, the baselines may also open the in-game character map to estimate their current position and compare it with the annotated route.

Q4.How are PCR, RVR, and game OSR calculated?

We have provided a more detailed explanation of the evaluation metrics in the Appendix.

PCR denotes the path coverage rate, measuring the proportion of the reference route traversed by the agent. We uniformly sample points along the ground-truth route and count a point as covered if its minimum distance to the test trajectory is within 15m. PCR is computed as the number of covered reference points divided by the total number of sampled reference points. PCR@15 reports coverage after 15 minutes of navigation, whereas PCR-F reports the final coverage at the end of the full evaluation horizon.

RVR is the revisit rate, which measures the proportion of backtracking. It is computed on the test trajectory: we divide the trajectory into N grid cells and traverse the trajectory from start to end, where entering a new cell counts as one entry and later re-entering a previously left cell counts as one revisit. RVR = number of revisits / total number of merged entries. A lower RVR indicates less backtracking.

OSR is the oracle success rate, which measures whether the goal is reached, i.e., whether the character trajectory reaches the goal point. If the agent reaches the goal but does not stop, it is counted in OSR; if it reaches the goal and stops, it is counted in SR. OSR = number of test samples reaching the goal / total number of samples. A higher OSR indicates a higher probability of passing the goal and thus stronger navigation ability.

Q5.How much video is used for map construction, and how long does processing it take?

We used 75 offline videos totaling 1.84 hours for map construction. Processing these videos to generate the geometry-guided topological memory took 4 hours, of which gemini-3.1-pro spent 2.5 hours on node selection, node annotation, and edge annotation, and VGGT-Omega spent 1.5 hours on four GPUs adding geometry information to the topological memory. This information has been added to the Appendix.

AuthorsOpenReview ↗

Response to Reviewer i9JJ(2/2)

Q6.What are each Table 1 VLM baseline's prompt, observation history, memory, action interface, tools, and reasoning, token, and model-call budgets?

The four VLM baselines in Table 1 use the same prompt, observation history, memory, tools, and action interface; the complete prompt is provided in the Appendix E.2. It describes the game environment, output format, keyboard–mouse action mapping, recovery behavior when the agent becomes stuck, stopping criterion, and use of the map. Each model receives the 20 most recent frames together with the corresponding action history. Its memory contains a complete game map annotated with the start, goal, and a hand-drawn optimal route. In addition, the system opens the in-game map every two minutes, allowing the model to verify the character’s current position. The action interface outputs strings in the format X Y;k1 k2;k3;k4 k5..., where X and Y specify horizontal and vertical view rotations mapped to mouse movements, and each subsequent segment specifies the keyboard or mouse buttons pressed simultaneously. Every output contains at least five actions, each executed for 100 ms. The reasoning instruction asks the model to briefly describe only the obstacles immediately ahead and the direction indicated by the map. Each request uses approximately 2,900 input tokens and 50 output tokens, while a full-map evaluation run consumes approximately 5 million tokens. The estimated cost per run is \16forKimi−K3,16 for Kimi-K3, \\27 for GPT-5.5, \4forGemini−3.8−Flash,and4 for Gemini-3.8-Flash, and \\27 for Claude-Opus-5.

Q7.Can at least one navigation baseline retrained for Table 2 be evaluated on the long-horizon task in Table 1?

We thank the reviewer for this suggestion. We conducted an ablation study in which our local anchor was replaced with ActiveVLN while the global anchor remained unchanged. We evaluated each controller over eight independent runs on the same long-horizon route, following the protocol and two-hour per-run time budget of the main Table 1 experiment.

CasePCR-F (↑)Succ. (↑)TTS (min) (↓)
Local anchor replaced with ActiveVLN11.50--
Ours96.962.562.29

The substantial performance drop indicates that ActiveVLN struggles to ground and execute the language-only instructions issued by the global anchor. One key source of error is referential ambiguity. As illustrated in Figure 6, when several similar pillars are visible, the instruction “Move near the stone pillar at the front left” may not uniquely identify the intended target. Our local anchor resolves such ambiguity by jointly grounding the text instruction and an image-space waypoint, leading to substantially better long-horizon navigation performance.

Thank you for the detailed response. I now better understand the experimental setup and the role of each component, and the authors have addressed my original questions.

My remaining concern is the limited evaluation scope. The main experiments are conducted in a single game, while the proposed framework relies on game-specific gameplay data, an offline topological map, and a trained local controller. It would therefore be helpful to clarify whether the framework can transfer to other games, or whether the map and local controller need to be rebuilt and retrained for each new game.

I would also like to ask whether the proposed tools and interfaces can be directly used with stronger frontier models, such as GPT-6 Astra. In particular, can such a model replace the current planner or local controller while using the same topological memory, re-anchoring, and waypoint interface? If so, it would help clarify whether the framework provides a model-agnostic navigation architecture and whether stronger foundation models could achieve substantially better performance.

Reviewer_KKpWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AnchorNav combines persistent topological memory with a waypoint-conditioned controller for long-horizon navigation in 3D games. Place recognition corrects visual odometry drift, while image-space waypoints and language guide local actions. Experiments in Black Myth: Wukong show improved traversal and instruction following. An additional RxR evaluation shows higher success rates, though path quality remains below several baselines.

Critical strengths
  1. The paper tackles a practical and challenging setting. Completing an approximately hour-long route through native keyboard-mouse controls is a meaningful systems result.

  2. The language-waypoint interface is motivated. Table 4 provides useful evidence that directional guidance and semantic instructions serve complementary roles.

Critical weaknesses
  1. The main long-horizon comparison does not isolate the proposed contribution. AnchorNav uses a trained controller and a video-derived spatial memory, while the baselines use different environment priors. The paper acknowledges this, but the results still leave unclear how much the dual-anchor design adds beyond these advantages. The memory ablation also removes several components together, so it does not establish the value of re-anchoring itself.

  2. Key implementation details are missing. VGGT-Omega is described as specially trained for game distractors, but its training data and procedure are not specified. The paper also leaves unclear how target coordinates, camera intrinsics, and behind-camera targets are handled during online waypoint projection. These details are central to reproducing the method.

  3. Closely related work is missing from the discussion: LiteVLoc [1] studies hierarchical visual localization and image-goal navigation using lightweight topometric maps, while OpenNavMap [2] addresses structure-free topometric mapping and multi-session alignment using geometric foundation models. These works with open-sourced code are directly relevant to AnchorNav’s persistent spatial memory, localization, and cross-video map integration. Yet neither is cited, and the paper does not explain what its global anchor contributes beyond this existing line of work.

[1] LiteVLoc: Map-lite visual localization for image goal navigation. ICRA 2025

[2] Opennavmap: Structure-free topometric mapping via large-scale collaborative localization. T-RO 2027

Questions for authors

With the graph, planner, and controller fixed, how much does disabling place-based re-anchoring affect long-horizon performance?

How is VGGT-Omega trained, and how are target geometry and camera intrinsics obtained at inference, including when the target is behind the camera?

6 comments
AuthorsOpenReview ↗

Response to Reviewer KKpW(1/2)

We thank Reviewer KKpW for the valuable comments. We have included examples of actual code execution in the Supplementary Material and revised the manuscript according to the comments. Please find our response below.

Q1.With the graph, planner, and controller fixed, how much does disabling place-based re-anchoring affect long-horizon performance?

The purpose of re-anchoring is to eliminate accumulated error. Since the accumulated error grows as the system runs, leaving it uncorrected degrades the route awareness of the memory. To this end, we add an experiment showing the effect of removing re-anchoring. We evaluated each setting over eight independent runs on the same long-horizon route, using the same protocol and two-hour per-run time budget as the main Table 1 experiment:

MethodPCR-F (↑)Succ. (↑)TTS (min) (↓)
w/o re-anchor18.90--
Ours96.962.562.29

Without re-anchoring, the accumulated error makes it impossible to complete a task that runs continuously for about one hour.

Q2.How is VGGT-Omega trained, and how are target geometry and camera intrinsics obtained at inference, including when the target is behind the camera?

VGGT-Ω is a feed-forward geometry model. In this paper, we use the released vggt_omega_1b_512 checkpoint [1], which adapts to our game scenes without retraining. We choose this checkpoint because [1] trains it on 4 million scenes with camera and depth annotations, about one third of which contain dynamic content; therefore, the model naturally generalizes to dynamic scenes, which makes it well suited to the game scenes we handle.

At inference, VGGT-Ω predicts the camera center, orientation, and field of view for each frame. The focal lengths are recovered from the predicted field of view as fx=(W/2)/tan⁡(fovw/2)f_x=(W/2)/\tan(\mathrm{fov}_w/2) and fy=(H/2)/tan⁡(fovh/2)f_y=(H/2)/\tan(\mathrm{fov}_h/2), with the principal point set to (W/2,H/2)(W/2,H/2). We then use the fixed intrinsics estimated at this stage to render the waypoint. At a resolution of 800×448800\times448 (width ×\times height), the estimated values are fx=573.854048f_x=573.854048 and fy=563.833107f_y=563.833107, with the principal point at the image center, (cx,cy)=(400,224)(c_x,c_y)=(400,224).

The next target is the center pp of route node n∗n^\ast stored in the graph, expressed in the same map coordinate system as the current camera center cc. We transform it into the OpenCV camera frame and then project it onto the image plane using the fixed intrinsics.

When Z>0Z>0, we apply the perspective projection defined in the main text and clamp an out-of-frame waypoint to the image boundary. When Z≤0Z\leq0, indicating that the target is behind the camera or at the camera center, perspective projection would reverse left and right. We therefore cast a ray from the image center in the direction (X,Y)(X,Y) and place the waypoint at its intersection with the image boundary, prompting the agent to turn toward that edge. If both XX and YY are close to zero, we place the waypoint at the midpoint of the bottom edge.

AuthorsOpenReview ↗

Response to Reviewer KKpW(2/2)

Q3.How does our global anchor differ from LiteVLoc and OpenNavMap?

We thank the reviewer for pointing out these closely related works. LiteVLoc focuses on hierarchical visual localization and image-goal navigation using lightweight topometric maps, whereas OpenNavMap focuses on scalable, structure-free topometric mapping, multi-session alignment, and collaborative localization. Both provide important foundations for persistent visual localization and map integration.

Our novelty does not lie in topometric mapping or hierarchical visual localization alone, but in transforming persistent spatial memory into a task-oriented anchor for long-horizon, language-conditioned navigation. First, in addition to visual references and geometry, the global anchor stores semantic landmark descriptions, forming a structured semantic-topological representation that the LLM can query and reason over as long-term memory. The LLM can therefore retrieve a language-specified destination as a graph node, plan a route from the current location, and periodically re-anchor the online trajectory through visual place recognition. Second, the global anchor converts the next route node into a semantic sub-instruction and an image-space waypoint for the waypoint-conditioned local controller: language specifies what to reach and when to stop, while the waypoint specifies where to move, thereby resolving the referential ambiguity of language-only instructions among visually similar landmarks. AnchorNav thus closes the loop from language-goal retrieval, persistent localization, and route planning to native keyboard–mouse control—an interface that a pose estimate or topological route alone does not provide.

The operating setting is also different. LiteVLoc primarily studies metric localization and image-goal navigation, while OpenNavMap emphasizes general multi-session map construction and maintenance. AnchorNav instead targets navigation lasting tens of minutes from RGB-only game observations containing dynamic characters, visual effects, and screen-fixed UI, without privileged pose or depth measurements.

[1] VGGT-Ω
[2] LiteVLoc: Map-Lite Visual Localization for Image Goal Navigation
[3] OpenNavMap: Structure-Free Topometric Mapping via Large-Scale Collaborative Localization

Thanks the author solve some of my doubts about its use of vgg-omega, but I still believe that the novelty of this paper is far from sufficient. To be honest, when using Anchor, I didn't feel any significant difference from the previous work. I agree with Reviewer iTRe on this point. Moreover, this paper leans more towards a systematic approach, that is, designing a system using existing modules and verifying it in a game. It is interesting, but It seems more like an interesting demo rather than a solid contribution to the learning community. After reading this paper, I did not gain any further insights into learning, nor did I think the methods presented in this paper could be applied to a broader range of applications. This seems more like a specific design for a specific topic. The author replied to me that “this paper offers native keyboard-mouse control - an interface that cannot be provided by a pose estimation or topological route alone.” But this is an application and engineer thing, not any sort of academic novelty.

It would be more suitable for conferences like IROS, WACV with demo in some games, but I think there won't be many people interested in such work at ICLR. I will maintain my original score.

Reviewer_iTReOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AnchorNav is a vision-language framework for 3D game navigation. It uses a local anchor (that projects the target into the current view) for short-horizon decisions. It constructs a global anchor graph (using VPR decisions) to maintain spatial consistency over time. Data collected from a 3D game (Black Myth: Wukong) is used in the experiments.

Critical strengths

The anchoring approach is interesting, while not very novel.

The use of VPR and semantic features to generate a merge score nicely accounts for vision and language cues.

Critical weaknesses

The experimental validation is limited, both in terms of the datasets used and the comparisons. Please consider comparing with more zero-shot VLN methods.

The gains that the proposed approach is expected to deliver are not seen in the RxR experiments. This weakens the claims. It also highlights the generalization gap between synthetic and real data.

The anchor memory size and the window size both influence performance. A discussion about this is missing. Table 4 only lists the memoryless case.

Questions for authors

Why is performance comparison limited to the chosen datasets and methods?

Is the comparison against general-purpose VLMs fair?

What is the impact of the VPR algorithm on performance?

Why is the generalization performance on RxR not as expected (for SPL and nDTW)?

What is the impact on performance of the threshold in (2)?

5 comments
AuthorsOpenReview ↗

Response to Reviewer iTRe(1/3)

We thank Reviewer iTRe for the valuable comments. We have included examples of actual code execution in the Supplementary Material and revised the manuscript according to the comments. Please find our response below.

Q1.The experimental validation is limited.

We thank the reviewer for the suggestion. To evaluate the effectiveness of our method more thoroughly, we additionally include experiments on three routes of different difficulty levels in the main text. The routes are as follows:

DifficultyRouteGame DistanceSkilled Human Time#Nodes on Shortest Route#Loops#Dead Ends
EasyOutside the Forest → Wandering Wight~170 m~0.5 min822
MediumWhiteclad Noble → Elder Jinchi~380 m~1.5 min12113
HardFront Hills → Outside the Forest~680 m~3.5 min2462
HardestStart → Black Bear Guai~3,900 m~20.0 min14118612

The Hardest route corresponds to the long-horizon task evaluated in Table 1 of the paper. For the other three routes, we additionally conduct experiments with 10 runs each, using per-run time budgets of 20, 40, and 90 minutes for the Easy, Medium, and Hard routes, respectively. The results are as follows:

MethodDifficultyPCR-F (↑)Succ. (↑)TTS (min) (↓)
Kimi-K3Easy93.480.04.51
Gemini-3.8-FlashEasy92.860.02.43
GPT-5.5Easy93.480.04.07
Claude-Opus-5Easy89.240.016.17
OursEasy96.090.00.96
Kimi-K3Medium34.30--
Gemini-3.8-FlashMedium49.70--
GPT-5.5Medium69.820.022.40
Claude-Opus-5Medium5.00--
OursMedium89.280.05.88
Kimi-K3Hard54.120.080.50
Gemini-3.8-FlashHard83.860.064.64
GPT-5.5Hard69.560.031.89
Claude-Opus-5Hard44.540.076.11
OursHard87.570.06.86

PCR-F denotes average route coverage, Succ. the success rate, and TTS the average completion time over successful runs; “--” indicates that the destination was never reached. Our method performs best across all three routes. Its advantage becomes more pronounced as route complexity increases: the baselines either fail to reach the destination or require substantially more time, whereas our method maintains high coverage and success rates with efficient completion.

Q2.Please consider comparing with more zero-shot VLN methods.

We thank the reviewer for the suggestion. We add comparisons with MapGPT [1] and PIVOT [2]. Since the action interfaces of these two methods cannot directly output keyboard-mouse operations, we deploy them in the position of the global anchor for route-level planning; to ensure a fair comparison, the local anchor uniformly uses our local model. We evaluated each method over ten independent runs on the Whiteclad Noble → Elder Jinchi scenario, using the same 40-minute per-run time budget as the medium-difficulty route experiment. The results are as follows:

CasePCR-F (↑)Succ. (↑)TTS (min) (↓)
MapGPT51.220.06.51
PIVOT20.20--
Ours89.280.05.88

Our method substantially outperforms both baselines in path coverage (PCR-F), success rate (Succ.), and completion time (TTS). Because all three methods use the same local anchor, the performance gap primarily reflects differences in route-level planning. Our global anchor grounds each planning decision in a persistent geometry-guided topological memory, performs explicit shortest-path planning, and uses visual place recognition to re-anchor the agent during execution. This combination provides stable long-term spatial context and limits the accumulation of localization and planning errors. In contrast, MapGPT and PIVOT are not designed around this persistent graph-based re-anchoring mechanism, making their route targets less stable over long-horizon navigation.

AuthorsOpenReview ↗

Response to Reviewer iTRe(2/3)

Q3.Is the comparison against general-purpose VLMs fair?

To ensure a fair comparison, we provide the general-purpose VLM baselines with explicit route and localization information through both prompt memory and the in-game map interface. Specifically, each baseline receives a saved game map annotated with the route and destination as part of its context. During execution, the baseline can also open the in-game character map to estimate its current position and compare it with the annotated route. In contrast, our method cannot access the in-game map for localization and must infer its position from rendered game frames through the global anchor.

To further control for low-level action execution, we use GPT-5.5 as the global planner and connect it to our local anchor, evaluating this hybrid baseline on routes of all three difficulty levels. We evaluated each method over ten independent runs per route, using per-run time budgets of 20, 40, and 90 minutes for the Easy, Medium, and Hard routes, respectively. GPT-5.5 relies on the annotated route map and in-game map queries to generate textual navigation instructions. By contrast, our global anchor plans with its geometry-guided topological memory and provides the local anchor with both language instructions and image-space waypoints. The results below show that our full method consistently achieves higher route coverage and success rates while requiring less completion time. Since the local anchor is shared, these results support the combined benefit of topology-grounded global planning and waypoint conditioning, which helps reduce ambiguity during instruction execution.

MethodDifficultyPCR-F (↑)Succ. (↑)TTS (min) (↓)
GPT-5.5 + Our Local AnchorEasy92.160.03.16
OursEasy96.090.00.96
GPT-5.5 + Our Local AnchorMedium71.20.0--
OursMedium89.280.05.88
GPT-5.5 + Our Local AnchorHard68.460.032.03
OursHard87.570.06.86

Q4.What is the impact of the VPR algorithm on performance?

VPR is a key component of route planning: only after VPR matches the current frame to a topological node and completes re-anchoring can the agent continue planning the subsequent path. When the threshold is too low, false matches easily occur and the agent cannot determine its own position; when the threshold is too high, no node is matched for a long time and re-anchoring cannot be performed, so the accumulated error causes the estimated trajectory to deviate from the actual position. In practice, we set the VPR confidence threshold to 0.53. As a supplement, we set the threshold to 0.2 and 0.9 on the Whiteclad Noble → Elder Jinchi route and ran each setting ten times. The results are as follows:

CasePCR-F (↑)Succ. (↑)TTS (min) (↓)
VPR-0.280.070.06.92
VPR-0.943.820.07.21
Ours89.280.05.88

The results show that a too-low threshold leads to a longer arrival time, while a too-high threshold makes the task harder to complete. Notably, navigation can still be completed with a lower threshold because we have already filtered out overly similar nodes when building the geometry-guided topological memory, which reduces the requirement on the discriminative accuracy of VPR.

Q5.Why is the generalization performance on RxR not as expected for SPL and nDTW?

Our results reveal a trade-off between goal-reaching performance and adherence to the reference path. On RxR, our method achieves a lower navigation error (NE) than StreamVLN (5.51 m vs. 6.22 m) and a higher success rate (61.65% vs. 52.9%), indicating that it more reliably approaches and stops near the goal. However, its SPL and nDTW are lower (42.12 and 52.56, respectively) than those of StreamVLN (46.0 and 61.9). This is because the val-unseen scenes contain no nodes from our topological memory. The route planner therefore selects the next target directly from the instruction and current observation, and the local anchor renders that target as an image-space waypoint for the controller to follow. This mechanism guides the agent toward the goal but does not explicitly constrain it to the annotated RxR reference route. Consequently, successful trajectories may be longer, reducing SPL, and may deviate in shape from the reference path, reducing nDTW. Thus, the lower SPL and nDTW reflect weaker reference-path fidelity rather than poorer goal-reaching ability.

AuthorsOpenReview ↗

Response to Reviewer iTRe(3/3)

Q6. How do the merge threshold in Eq. (2) and the resulting anchor memory size affect performance?

Eq. (2) determines whether two observations should be merged into the same topological node. The merge score combines visual and semantic similarities with weights of 0.7 and 0.3, respectively; observations are merged when the score exceeds the default threshold of 0.72. This weighting prioritizes visual evidence while retaining semantic cues. Overemphasizing visual similarity may prevent the same location from being associated across substantially different viewpoints, whereas overemphasizing semantic similarity may incorrectly merge spatially distinct but semantically similar locations, such as different areas containing trees or bridges.

Because the threshold directly controls how many nodes are retained, it also determines the effective size of the topological memory. We therefore evaluate its impact on the medium-difficulty route. We evaluated each threshold over ten independent runs, using the same 40-minute per-run time budget as the main medium-difficulty route experiment. A high threshold imposes a stricter merging criterion, retaining too many near-duplicate nodes and producing an unnecessarily large and redundant graph. In contrast, a low threshold over-merges spatially distinct locations, leaving too few nodes for reliable localization. The results show that the default threshold of 0.72, which yields 489 nodes, achieves the best balance between memory size and navigation performance.

Threshold#Memory NodesPCR-F (↑)Succ. (↑)
0.506434.90.0
0.7248989.280.0
0.9055672.370.0

Q7. How does the window size affect performance?

We use VGGT-Omega to incorporate geometric information into the topological memory. By default, it processes sliding windows of eight frames with a three-frame overlap, using the shared frames to align adjacent local reconstructions within a common memory coordinate system. We also evaluate a larger window of 30 frames. In dynamic game scenes, this longer temporal span makes consistent pose estimation and alignment more difficult, introducing residual geometric errors that accumulate as local reconstructions are integrated into the route map and ultimately degrade navigation performance. We evaluated each window size over ten independent runs on the medium-difficulty route, using the same 40-minute per-run time budget as the main experiment:

Window size (frames)PCR-F (↑)Succ. (↑)TTS (min) (↓)
889.280.05.88
3044.030.07.91

[1] MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigation
[2] PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs

Reviewer_bKjYOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ANCHORNAV introduces dual-scale anchoring for long-horizon 3D game navigation. A global anchor builds persistent topological memory from offline gameplay video and uses AnyLoc place recognition plus VGGT-Omega odometry for periodic re-anchoring and route-target selection. A local anchor projects the next target as a 2D image-space waypoint and combines it with a language sub-instruction to drive a waypoint-conditioned Qwen3.5-4B policy with native keyboard–mouse control. Results show ANCHORNAV is the only system completing the long-horizon route, leads instruction following on SR, SPL, and nDTW at all difficulty levels, and reaches NE 5.51 m / SR 61.65 on RxR, though without SPL/nDTW gains. Ablations indicate memory sustains progress and coverage, language supplies stopping, and waypoints provide directional grounding.

Critical strengths
  1. Language–image-space waypoint interface: The paper projects the next route target as a 2D waypoint directly onto the input frame and pairs it with a language sub-instruction, so the controller receives both where to move and what arrival means in the same visual space. This is the paper’s most substantive interface-level innovation, though it is mainly validated in short-horizon and 15-minute PCR settings.

  2. Persistent topological memory with periodic re-anchoring: ANCHORNAV organizes offline walkthrough video, AnyLoc place recognition, and VGGT-Omega odometry into a reusable spatial reference, and periodically re-anchors local trajectories to the memory coordinate system to limit drift. The novelty lies in the system-level combination and re-anchoring workflow rather than in any single component.

3.Long-horizon evaluation in a photorealistic commercial game with native keyboard–mouse control: The agent navigates a roughly 20-minute human-expert route using only RGB frames and native controls, and the paper uses continuous metrics such as coverage, backtracking, and R/C to distinguish failure modes. The setting is novel and relevant, but evidence is limited to one route and eight runs, and the memory may overlap with the evaluation area.

4.R/C as a diagnostic metric: By normalizing backtracking by progress, the paper prevents a stalled agent from appearing efficient simply because it backtracks little, revealing the difference between “not daring to move” and “moving wrongly and returning.” This is a methodologically useful contribution, though the metric itself is not entirely new and the sample size remains small.

Critical weaknesses
  1. Long-horizon instruction is not specified. Section 4.1 only states the agent must “reach the boss arena” but never gives the actual instruction text, its stopping condition, or how the boss arena is represented as a retrievable node in the topological memory.

  2. Route complexity is not characterized. The long-horizon track uses a single route, with no quantification of branches, dead ends, landmark density, or visually repetitive regions. One route cannot support generalization claims, and the paper does not analyze how memory coverage interacts with route complexity.

  3. Efficiency and real-time capability are not measured. The paper claims a “real-time execution loop”, but reports no end-to-end latency, policy inference frequency, hardware configuration, or worst-case delay.

  4. Baseline fairness is insufficient. ANCHORNAV is a trained multi-component system with an environment-specific topological memory, while long-horizon baselines are zero-shot general VLMs; training data, component count, and priors all differ. On RxR, baseline numbers are quoted rather than rerun with the same adapter, and control frequency and stopping are unmatched.

  5. Sample sizes are small. The long-horizon track has one route and eight runs; PCR@15 uses five runs; ablation instruction following uses fourteen instructions. No confidence intervals, significance tests, or per-task variance are reported, and “baseline success rate undefined” should be replaced by distributions of coverage, progress, and time-to-failure.

Questions for authors

Q1. What is the instruction text for the long-horizon task, and how is end-point confirmation validated?

Section 4.1 only states “reach the boss arena” but does not give the actual instruction text, the stopping condition, or how the boss arena is represented as a retrievable target node in the topological memory. Can the authors provide a complete instruction example and the trigger conditions for termination?

Q2. The long-horizon track uses only one route. How is route complexity characterized?

The paper does not quantify the number of branches, proportion of dead ends, landmark density, or distribution of visually repetitive regions along this route. Success on a single route cannot support generalization claims. Can the authors provide multiple routes, route-complexity metrics, and a difficulty stratification for long-horizon tasks corresponding to the easy/medium/hard levels in the short-horizon track?

Q3. What are the end-to-end latency and hardware configuration?

The paper claims a “real-time execution loop” and 20 Hz action replay, but 20 Hz is the interface dispatch rate, not the model inference rate. Can the authors report: the per-step inference latency of the local policy (Qwen3.5-4B, 20 frames, 448×800); the call latency of AnyLoc, VGGT-Omega, and the VLM route planner; GPU model and memory; the global-anchor update period; and worst-case latency and frame-drop handling?

Q4. Can the authors add baselines matched on backbone, observation, control frequency, and training budget?

In the long-horizon track, ANCHORNAV is a trained multi-component system while the baselines are zero-shot general VLMs. Can the authors provide controlled comparison: equip a baseline with a local policy trained under the same budget, or remove training from ANCHORNAV and run it zero-shot, to isolate the effect of training data?

Q5. Will the code, data, and models be released?

The method relies on multiple proprietary VLM endpoints, the custom 350-hour gameplay dataset is not public, and graph size, prompts, thresholds, and training hyperparameters are insufficiently reported. Can the authors state their release plan for code/data/models, or at least provide sufficient details to reproduce graph construction and local-policy training?

4 comments
AuthorsOpenReview ↗

Response to Reviewer bKjY(1/4)

We thank Reviewer bKjY for the valuable comments. We have included examples of actual code execution in the Supplementary Material and revised the manuscript according to the comments. Please find our response below.

Q1.What is the instruction text for the long-horizon task (“reach the boss arena”), and how are the stopping condition and end-point confirmation validated?

The phrase “reach the boss arena” in Section 4.1 is only a description of the task rather than the actual instruction given to the agent. In the long-horizon task, we provide goal instructions that name the destination, such as “Go find the Black Bear Guai” (the Chapter 1 boss) or “Go to the Keeper's Shrine outside the forest”. Each destination corresponds to a landmark node in the topological memory, which stores a semantic description, nearby reference views, and visual descriptors. After receiving the instruction, the global anchor retrieves this node and sets it as the goal, then performs shortest-path planning on the topological graph and passes the next route target together with a language sub-instruction to the local controller. When visual place recognition matches the current game frame to the goal node, the agent is considered to have arrived; the global policy then outputs <stop> and ends the navigation.

Q2.The long-horizon track uses only one route. How is route complexity characterized, and can multiple routes with difficulty stratification be provided?

We thank the reviewer for the suggestion. We add tasks of different difficulty levels, from easy to hard: Outside the Forest → Wandering Wight, Whiteclad Noble → Elder Jinchi, Front Hills → Outside the Forest, and Start → Black Bear Guai. We characterize route complexity using the game distance, the completion time of a skilled human player, the number of topological memory nodes, the number of loops, and the number of dead ends.

DifficultyRouteGame DistanceSkilled Human Time#Nodes on Shortest Route#Loops#Dead Ends
EasyOutside the Forest → Wandering Wight~170 m~0.5 min822
MediumWhiteclad Noble → Elder Jinchi~380 m~1.5 min12113
HardFront Hills → Outside the Forest~680 m~3.5 min2462
HardestStart → Black Bear Guai~3,900 m~20.0 min14118612

The Hardest route corresponds to the long-horizon task evaluated in Table 1 of the paper. For the other three routes, we additionally conduct experiments with 10 runs each, using per-run time budgets of 20, 40, and 90 minutes for the Easy, Medium, and Hard routes, respectively. The results are as follows:

MethodDifficultyPCR-F (↑)Succ. (↑)TTS (min) (↓)
Kimi-K3Easy93.480.04.51
Gemini-3.8-FlashEasy92.860.02.43
GPT-5.5Easy93.480.04.07
Claude-Opus-5Easy89.240.016.17
OursEasy96.090.00.96
Kimi-K3Medium34.30--
Gemini-3.8-FlashMedium49.70--
GPT-5.5Medium69.820.022.40
Claude-Opus-5Medium5.00--
OursMedium89.280.05.88
Kimi-K3Hard54.120.080.50
Gemini-3.8-FlashHard83.860.064.64
GPT-5.5Hard69.560.031.89
Claude-Opus-5Hard44.540.076.11
OursHard87.570.06.86

PCR-F denotes average route coverage, Succ. the success rate, and TTS the average completion time over successful runs; “--” indicates that the destination was never reached. Our method performs best across all three routes. Its advantage becomes more pronounced as route complexity increases: the baselines either fail to reach the destination or require substantially more time, whereas our method maintains high coverage and success rates with efficient completion.

AuthorsOpenReview ↗

Response to Reviewer bKjY(2/4)

Q3.What are the end-to-end latency and hardware configuration, given that 20 Hz is the interface dispatch rate rather than the model inference rate?

Our system runs on six NVIDIA H200 GPUs. The global anchor uses five GPUs: Qwen3.8-27B-FP8 is distributed across four GPUs, and VGGT-Omega runs on the fifth; the local anchor uses the remaining GPU. The global anchor has a time to first token (TTFT) of 122 ms (287 ms worst case) and an average end-to-end latency of 570 ms (673 ms worst case). It is queried at 1 Hz, with each query either producing a new instruction or returning an empty response to indicate that the current instruction should remain unchanged. The average latency of AnyLoc is 109.05 ms (110.10 ms worst case), while VGGT-Omega inference takes 177.89 ms and Dijkstra route planning takes 0.201 ms on average. The local anchor has a TTFT of 88.04 ms (103 ms worst case) and typically generates 12–14 actions per query. Once the first action is decoded, subsequent actions are streamed and executed concurrently with decoding. Thus, 20 Hz refers to the rate at which decoded actions are dispatched to the keyboard–mouse interface, rather than the model inference rate, with each action executed for 50 ms. After completing an action sequence, the local anchor either starts a new inference using the latest observation or reports completion to the global anchor. We use a latest-wins policy for dropped frames: if a consumer falls behind, it discards all intermediate frames and processes only the most recent one.

Q4.Can the authors add baselines matched on backbone, observation, control frequency, and training budget, to isolate the effect of training data?

We thank the reviewer for this suggestion. A zero-shot variant without supervised fine-tuning (SFT) is not directly applicable in our setting because the model must learn to map navigation instructions to executable keyboard–mouse actions. To provide a more controlled comparison, we replaced our local anchor with ActiveVLN while keeping the global anchor unchanged. ActiveVLN was trained on the same data and under the same training budget as our local anchor but did not receive waypoint-conditioned inputs.

Local ControllerPCR-F (↑)Succ. (↑)TTS (min) (↓)
ActiveVLN11.50--
Ours96.962.562.29

The substantial performance drop after replacing our local anchor indicates that ActiveVLN struggles to ground and execute the language-only instructions issued by the global anchor. A key challenge is referential ambiguity: as illustrated in Figure 6, when several similar pillars are visible, the instruction “Move near the stone pillar at the front left” may not uniquely identify the intended target. Our local anchor mitigates this ambiguity by jointly grounding the language instruction and an image-space waypoint, resulting in substantially better long-horizon navigation performance.

AuthorsOpenReview ↗

Response to Reviewer bKjY(3/4)

Q5.Will the code, data, and models be released, or can sufficient details be provided to reproduce graph construction and local-policy training?

We will release the data, code, and models once the necessary permissions have been obtained. In the meantime, we provide below the implementation details required to reproduce the topological graph construction and local-policy training; these details have also been added to the Appendix.

The geometry-guided topological memory is constructed automatically from offline videos. Gemini first identifies candidate landmarks and generates a name, note, and semantic label for each one. Within a temporal window around each landmark, we retain as reference images only frames whose AnyLoc similarity is at least 0.55. Because the videos comprise multiple segments and Gemini may annotate the same location repeatedly, duplicate landmarks are merged using Eq. (2). Specifically, the merge score combines visual and semantic similarity with weights λ_vis=0.7 and λ_sem=0.3 (K=3). The visual descriptor is a 49,152-dimensional L2-normalized DINOv2 VLAD feature, while the semantic descriptor is a 1,024-dimensional embedding of the landmark labels and notes. A candidate with a weighted score above 0.72 is merged into the existing node; otherwise, a new node is created. Camera geometry is estimated using VGGT-Omega (vggt_omega_1b_512) at a resolution of 512, with eight-frame windows and a three-frame overlap, and is then aligned to a common map coordinate system. For visual place recognition during re-anchoring, node reference images are indexed in FAISS using DINOv2 ViT-g/14 with VLAD aggregation. Route planning performs a shortest-path search over the resulting graph using cost_seconds as the edge weight. The final graph contains 489 nodes and 691 edges.

The local policy is based on Qwen3.5-4B and is trained for 30,000 steps with the vision encoder frozen. Training uses eight GPUs, a per-GPU batch size of 4, a gradient-accumulation factor of 2, a learning rate of 5×10⁻⁶, a weight decay of 0.01, and a warm-up ratio of 0.01. Input images are resized to 448×800, and the navigation instructions use three formats—text with cursor, cursor only, and text only—sampled at a ratio of 4:3:3. The training data are drawn from 350 hours of gameplay recordings; before annotation, Gemini removes low-quality segments such as cutscenes and wall-collision sequences. We sample one frame out of every ten from the 20 Hz videos and construct a 20-frame history window. During training, waypoints are obtained by projecting future trajectory points onto the image; at inference, they are projected from the global target node. In both cases, the waypoints are rendered directly on the input frames. The model is optimized with cross-entropy loss over the action-token vocabulary: mouse movements are represented by relative-angle tokens, key presses by their corresponding key tokens (e.g., W and Space), and termination by a dedicated <stop> token. At inference, the model predicts variable-length action sequences, which are dispatched to the keyboard–mouse interface at 20 Hz, with each action held for 50 ms.

Q6. The baseline success rate is undefined and should be replaced by distributions of coverage, progress, and time to failure.

In Table 1, we evaluate each method over eight independent runs, with a two-hour time budget per run. None of the baselines reaches the destination within this budget; their success rates are therefore 0%, while their average time to success (TTS) cannot be computed because there are no successful runs. To characterize their navigation behavior despite the absence of successful runs, we additionally report route coverage after 15 minutes (PCR@15) and average final route coverage over the full two-hour runs (PCR-F):

CasePCR@15 (↑)PCR-F (↑)
GPT-5.59.347.6
Claude-ops54.344.0
Gemini3.8flash14.231.2
kimi-k34.439.3
Ours30.396.9
AuthorsOpenReview ↗

Response to Reviewer bKjY(4/4)

Q7.No confidence intervals, significance tests, or per-task variance are reported.

We thank the reviewer for this suggestion. We run each method repeatedly on the Easy, Medium, and Hard routes and report the mean PCR-F together with the variance (Var) and standard deviation (SD), all expressed in percentage points:

MethodDifficultyMean (%)Var (%²)SD (%)
Kimi-K3Easy93.4216.014.70
Gemini-3.8-FlashEasy92.898.09.90
GPT-5.5Easy93.4173.013.15
Claude-Opus-5Easy89.2147.012.12
OursEasy96.063.07.94
Kimi-K3Medium34.3483.021.98
Gemini-3.8-FlashMedium49.7320.017.89
GPT-5.5Medium69.8761.027.59
Claude-Opus-5Medium5.092.09.59
OursMedium89.2579.024.06
Kimi-K3Hard54.11770.042.07
Gemini-3.8-FlashHard83.8581.024.10
GPT-5.5Hard69.51992.044.63
Claude-Opus-5Hard44.52603.051.02
OursHard87.5919.030.31

Our method achieves the highest mean PCR-F on every difficulty level. In terms of run-to-run stability, its variance is the lowest among all methods on the Easy route and remains within the range spanned by the baselines on the Medium and Hard routes, indicating that the advantage is consistent across independent runs rather than attributable to run-to-run randomness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.