CoFL-S: Spatially Queryable Sector Flow Fields for Local Language-Conditioned Navigation

VLN studies how an embodied agent follows natural-language instructions under partial observability. Canonical benchmarks such as R2R [1], RxR [2], and VLN-CE [3] established instruction following in photo-realistic indoor environments and continuous simulation [20, 21]. Recent VLN methods have advanced instruction grounding, mapping, subgoal prediction, memory, and instruction decomposition, while also examining visual and physical embodiment gaps [4, 5, 6, 7, 22, 23, 24, 25]. This paper takes a complementary perspective: using fine-grained annotations from Fine-Grained R2R [8] and Landmark-RxR [9, 19], it factors out high-level route decomposition and studies how local language-conditioned intent should be represented for low-level execution.

VLN commonly uses discrete navigation actions such as moving forward, turning, and stopping, or predicts relative waypoint targets that are typically realized by an additional tracking or local-planning module [6, 3, 10, 11]. Beyond navigation, robot policies often use action-token policies [12, 26, 27] or action-chunk decoders [13, 14, 28, 29]. This distinction guides our baseline choice: action tokens represent the current-state decision interface, while action chunks provide an end-to-end counterpart to waypoint interfaces by predicting both a local target and the motion rollout. Together, these baselines cover current-state decisions and finite-horizon rollouts, but neither represents how desired motion should vary across nearby workspace locations. Classical navigation methods offer a complementary spatial-guidance view, representing control over workspace locations or velocity states rather than only at the current robot state [15, 16, 17, 18, 30, 31]. Recent learned flow-field policy [32] connects perception, semantics, and control by representing motion as a flow field over a globally observed BEV workspace, enabling dense workspace-level supervision and stronger navigation feedback than action or trajectory interfaces. However, its third-person BEV observation assumption bypasses the partial-observability and onboard-sensing constraints faced by robots that must infer control from first-person sensory input. CoFL-S brings this principle to a more restrictive and practical setting, predicting a language-conditioned visible-sector field from ego-centric RGB-D observations for onboard closed-loop execution.

CoFL-S is designed as a low-level planner for the visible local workspace within the robot camera’s field of view. Given an ego-centric RGB observation $I$, depth $D$, and a low-level language instruction $\ell$, the desired output is a local continuous trajectory $\tau(t)\in\mathbb{R}^{2}$ parameterized by normalized time $t\in[0,1]$. The trajectory is represented in the robot-centric ground-plane Cartesian frame as

Instead of regressing the whole trajectory, CoFL-S generates it through a first-order motion model:

Here $\mathbf{v}_{\phi}$ is a learnable flow field conditioned on the current observation and instruction, and $\Pi_{\mathrm{polar}}$ maps a Cartesian point $\tau(t)$ on the local ground plane to its normalized sector coordinate $\mathbf{q}$. We define the normalized polar sector as $\Omega=[-1,1]\times[0,1]$ and $\mathbf{q}=(\tilde{\theta},\tilde{r})\in\Omega$. This polar sector is naturally aligned with the robot camera’s field of view.

For a queried sector coordinate, the policy predicts a Cartesian velocity:

Thus, the field is queried in normalized polar coordinates, while both the predicted velocity and the generated trajectory are expressed in the ego-centric ground-plane Cartesian frame.

Fig. 2 summarizes the architecture. A frozen SigLIP2 encoder [33] $(f_{\mathrm{vis}},f_{\mathrm{txt}})$ first maps the RGB observation $I$ and low-level instruction $\ell$ into vision tokens $\mathbf{Z}_{v}$ and text tokens $\mathbf{Z}_{\ell}$ in a shared embedding space:

In the RGB-D setting, a convolution $f_{\mathrm{dep}}$ first patchifies the depth into features aligned with the vision token by matching the kernel size and stride. Then, a FiLM pathway $f_{\mathrm{FiLM}}$ [34] injects these depth features into $\mathbf{Z}_{v}$ through bounded residual modulation (illustrated in Fig. 3(a)). The modulated visual tokens and text tokens are then fused by Transformer decoder-style blocks $f_{\mathrm{fuse}}$ [35], where visual tokens serve as queries and text tokens serve as keys and values:

For field prediction, each sector query $\mathbf{q}\in\Omega$ is embedded as a coordinate token with Gaussian Fourier feature embedding $\psi$ followed by a linear projection [36]. A query-based Transformer decoder refines these coordinate tokens over the fused context $\mathbf{C}$:

In addition to the sector-field decoder, CoFL-S includes a lightweight action module for termination prediction, as illustrated in Fig. 3(b). This module shares the fused context $\mathbf{C}$ with the decoder and uses the same query-based Transformer-decoder design [35], but replaces coordinate queries with $N_{id}$ learnable action queries $[\mathrm{a}_{1},\ldots,\mathrm{a}_{N_{id}}]$, corresponding to the discrete VLN-CE action IDs.

The decoder output is passed to an MLP that produces one logit for each action query:

During training, we sample sector queries $\mathbf{q}_{i}\in\Omega$ from an area-uniform distribution over the physical local sector, map them to Cartesian positions $\mathbf{p}_{i}$, and bilinearly sample the target Cartesian velocity $\mathbf{v}_{i}^{*}$ from the annotated dense flow field. The training objective is

Here $\mathcal{L}_{\mathrm{dir}}$ supervises the local motion direction at each queried position, indicating where the robot should move under the current instruction. $\mathcal{L}_{\mathrm{mag}}$ supervises the distance-to-go scale, which reflects the remaining obstacle-aware geodesic distance to the instruction-specific goal. $\mathcal{L}_{\mathrm{act}}$ supervises the auxiliary action module used for termination. $p_{\mathrm{act}}^{*}$ is a one-hot distribution derived from the ground truth action id $\mathrm{a}^{*}$.

At each inference cycle, if the STOP action has the highest probability in $p_{\mathrm{act}}$, execution terminates; otherwise, motion is generated by integrating the predicted sector field. For efficiency, we query the decoder once on a regular polar grid and cache the resulting field; rollout then samples velocities from this grid by bilinear interpolation instead of repeatedly invoking the decoder. Rollout is performed in Cartesian space with $T$ inference steps. Starting from $\mathbf{p}_{0}=\tau(0)$, we repeatedly map the current Cartesian state to a sector query and integrate the predicted Cartesian velocity:

where $\tilde{\mathbf{v}}_{j}=s(j,T)\,{\mathbf{v}}(\mathbf{q}_{j}),\quad\mathbf{q}_{j}=\Pi_{\mathrm{polar}}(\mathbf{p}_{j})$. Here ${\mathbf{v}}(\mathbf{q}_{j})$ denotes the bilinearly interpolated velocity from the cached sector grid. $s(j,T)$ is a bounded inverse-time rescaling that converts distance-to-go-scaled vectors into fixed-horizon rollout steps. The final output is the Cartesian trajectory $\tau=\{\mathbf{p}_{0},\mathbf{p}_{1},\ldots,\mathbf{p}_{T}\}$. Appendix B gives thorough implementation details.

Simulation protocol. Training data are constructed from the official R2R-CE and RxR-CE [3] training episodes by deriving interface-matched discrete, finite-horizon, and dense flow-field targets from the same replayed frames and adding the instruction-diverse slots described in Sec. 4. Fig. 5 summarizes the composition of the augmented training set. We use the official R2R [1] and RxR [2] val-unseen episodes route and instruction annotations for evaluation, but do not follow the standard discrete VLN-CE action protocol. Instead, all methods are evaluated in a continuous-time Habitat environment [21], where the agent state is updated at $60\text{\,}\mathrm{Hz}$, RGB-D observations are received at $30\text{\,}\mathrm{Hz}$, and planner outputs are converted into velocity commands executed by a shared low-level controller. At each control step, we map the current Habitat pose of the agent to the closest progress point on the reference trajectory and select the associated Fine-Grained R2R [8] or Landmark-RxR [9] sub-instruction. All methods receive the same ego-centric RGB-D observation stream and aligned local sub-instruction, and are evaluated with the same controller and evaluation episodes. Inference latency is measured on an NVIDIA GeForce RTX 4090. More details are provided in Appendix D.1.1.

Baselines. All methods use the same vision-language encoder (Sec. 3.2) and matched training. Action Token, representing discrete action-class prediction, follows the standard VLN-CE discrete action interface, predicting forward, turn-left, turn-right, and stop actions [3, 6], but uses continuous velocity control instead of discrete jumps. For architectural matching, it uses the same query-based action module as CoFL-S (Sec. 3.2), but uses its predicted action distribution for all low-level decisions rather than only for termination. Action Chunk, representing current-state-anchored future-motion prediction, predicts a finite-horizon future action sequence from the current observation and instruction with a Transformer-style diffusion policy head [14], and uses the same query-based action module as CoFL-S (Sec. 3.2) for STOP prediction (Horizon: $100$, denoise steps: $5$). CoFL-S generates motion by integrating the predicted ego-centric sector flow field, using a $40\times 40$ query grid and $100$ rollout steps. More details are provided in Appendix D.1.2.

Metrics. We report standard VLN metrics, including navigation error (NE), oracle success (OS), success rate (SR), and Success weighted by Path Length (SPL) (we use the reference path length as the ground truth rather than the shortest-path length). We further report two metrics beyond standard VLN metrics: Blocked-Step Rate (BSR), computed as the fraction of control steps in which the agent is blocked by the navigation mesh, and Heading Smoothness (HS), computed from heading changes along the executed trajectory. More details are provided in Appendix D.1.3.

Table 1 compares the action-token, action-chunk, and the CoFL-S interface on R2R-CE and RxR-CE val-unseen [3] under the same simulation protocol. A qualitative example is provided in Fig. 6. Each method is tested at low, moderate, and high frequencies ($2\text{\,}\mathrm{Hz}$ / $5\text{\,}\mathrm{Hz}$ / $10\text{\,}\mathrm{Hz}$).

Across planner frequencies and both datasets, CoFL-S achieves the strongest task-completion performance, obtaining the best NE, OS, SR, and SPL in every frequency group. This result is consistent with the representation–supervision view of flow-field policies observed in [32]: workspace-level supervision trains the field over many queried local states, effectively providing a family of corrective motion targets rather than a single current-state action or one rollout for action token or action chunk training. The gains are especially clear over Action Chunk, which has the largest prediction head but remains weaker in task completion (SR/SPL) and geometry awareness (BSR). At $10\text{\,}\mathrm{Hz}$, Action Token achieves the lowest BSR and highest HS, indicating that high-frequency discrete updates with low latency can reduce blocked steps and produce smooth heading changes. CoFL-S remains competitive on these local diagnostics while achieving higher SR/SPL, suggesting that the sector-field interface, together with its dense workspace-level supervision, improves task completion without causing a clear loss in practical local stability.

Detailed ablation studies are provided in Appendix D.2. The results show that instruction-diverse supervision provides moderate gains, while the action module and depth modulation complement the sector-field interface by improving termination prediction and geometry-aware execution.

We further deploy CoFL-S on a physical mobile robot equipped with an ego-centric Intel RealSense D435i camera (Frequency: $30\text{\,}\mathrm{Hz}$; FoV: $70^{\circ}$) and an NVIDIA Jetson AGX Orin $64\text{\,}\mathrm{GB}$ (details are provided in Appendix D.3.1). Notably, we directly use the simulation-trained model without any real-world fine-tuning. The real-world evaluation follows the same local-instruction setting: local subtasks are manually assigned and updated by an operator according to the robot’s progress. We evaluate zero-shot deployment across eight task sequences spanning three real-world areas, including indoor, outdoor, and hybrid indoor–outdoor settings. Each sequence is repeated three times, and the inference frequency is fixed at $5\text{\,}\mathrm{Hz}$. Target objects are chosen from the Matterport3D label list [20]. One example is shown in Fig. 7. The maximum linear velocity is set to $0.5\text{\,}\mathrm{m}\text{/}\mathrm{s}$ and the maximum angular velocity is set to $1.0\text{\,}\mathrm{rad}\text{/}\mathrm{s}$.

Table 2 reports success rate (SR), manually counted collision times averaged over each trial (CT), success-conditioned completion time (T-S), success-conditioned path length (PL-S) and inference latency (Lat.). Under this zero-shot cross-domain deployment setting, CoFL-S achieves the highest SR and the lowest CT, T-S, and PL-S, indicating stable transfer beyond the indoor simulation distribution. Compared with Action Token, which remains usable in open areas but suffers from oscillations near obstacles, likely because its current-state decisions are more sensitive to the onboard latency gap ($\sim 100\text{\,}\mathrm{ms}$ vs. $\sim 10\text{\,}\mathrm{ms}$ in simulation), and Action Chunk, which more often collides or gets stuck, CoFL-S better couples local instruction semantics with surrounding geometric constraints. Despite its higher latency, CoFL-S remains compatible with the $5\text{\,}\mathrm{Hz}$ onboard closed-loop setting, as its rollout provides a short-horizon geometric command that is less sensitive to moderate observation–control latency than single-step action prediction. Failure analyses are provided in Appendix D.3.3.