A Survey of Spatial Intelligence Benchmarks for MLLMs, VLAs, and World Models

Version v0.5

1. Introduction

Recent multimodal models have demonstrated strong visual perception capabilities, including recognizing and counting objects, identifying their attributes, and understanding events from images and videos. These capabilities provide an important foundation for understanding the physical world. Spatial problems, however, present challenges that go beyond recognizing or locating objects in images. They may demand a more precise understanding of geometry and spatial relations, as well as the ability to integrate information across views and time and reason about changes in the environment.

Spatial intelligence is the ability to understand and use information about the spatial world. It involves perceiving objects and their relations, organizing observations into persistent representations, simulating possible changes, and selecting actions under spatial and physical constraints. These abilities are closely related, but success in one task does not establish reliable performance in another. A model may recognize an object while misjudging its distance, recover a scene while failing to predict its future, or answer a spatial question without completing the corresponding action [13], [23], [77].

Multimodal large language models (MLLMs) provide an important foundation for visual perception and understanding. Extending these capabilities to spatial problems requires methods for encoding geometry, maintaining spatial state across observations, and reasoning about changes in the environment. Action supervision trains models to generate actions from observations and instructions [56], [57], while world-modeling objectives teach models to predict how spatial states change over time and in response to actions [59], [61]. These learning signals can be combined, as illustrated by models that jointly learn action generation and environment dynamics [62].

This survey asks: How do models learn to perceive, represent, simulate, and act in the physical world, and how should these capabilities be evaluated? We examine how spatial information is supplied and represented, how architectures and training data support perception, memory, simulation, and action, and how existing benchmarks assess progress in these abilities.

We organize representative benchmarks from four perspectives: (1) inputs and interfaces, describing the observations and tools available to a model; (2) spatial capabilities, describing the tasks it must complete; (3) evaluation, describing access, interaction, outputs, and scoring; and (4) data sources, describing the origin of observations and supervision. Within these perspectives, we connect benchmark findings with studies of representation, learning, memory, simulation, and action.

Our coverage includes foundational evaluations of spatial and physical reasoning, recent multimodal and robotic benchmarks, and studies that examine how models acquire and use spatial capabilities.

Fig. 1 summarizes the taxonomy. Sections 2 and 3 introduce the background and organization. Sections 4–7 review the four perspectives. Section 8 discusses spatial representations, capability transfer, simulation for action, and generalization.

2. Background

2.1 Multimodal Large Language Models

Early visual foundation models, such as CLIP [3] and DINO [4], established reusable representations through contrastive and self-supervised learning. MLLMs such as LLaVA [5] connect visual encoders with large language models, enabling visual question answering and instruction following. Video-LLaVA [6] and LLaVA-OneVision [7] extend this interface to multiple images and video. Thinking in Space [13] examines the spatial understanding of video-based MLLMs through VSI-Bench.

An MLLM receives spatial information through the features, tokens, or structured observations supplied to its language backbone. Its ability to use that information also depends on alignment and task training. SpatialVLM [8], for example, introduces spatial question-answer supervision, while DepthVLM [9] studies dense metric depth estimation. These approaches specialize multimodal learning toward spatial targets without requiring all spatial information to be expressed in the same form.

2.2 Vision-Language-Action Models

Vision-language-action (VLA) models connect visual and language understanding with action generation. They are trained to produce actions from visual observations and language instructions. RT-2 [56] represents actions as discrete tokens and combines robot demonstrations with visual-language training. OpenVLA [57] also adapts a vision-language backbone for robot control. Other designs use continuous action modules: π0 [58] combines a vision-language model with a flow-matching action expert.

Spatial information supports action by helping models identify targets, determine feasible movements, and respond to changes in the environment. Related systems connect understanding with action through skill selection, as in SayCan [73], or spatial objectives for motion planning, as in VoxPoser [74].

2.3 World Models

A world model is a learned model of an environment that predicts how it evolves over time and in response to actions [59]. Simulation can operate on latent features, images, explicit states, or executable programs. World Models [59] separates visual representation, temporal dynamics, and control. DreamerV3 [60] learns latent dynamics and uses imagined trajectories to train an actor. DINO-WM [88] predicts pretrained visual features and searches for actions that reach an image goal.

Simulation provides a way to examine possible outcomes before acting, helping models select appropriate actions. V-JEPA 2 [61] connects video representation learning with an action-conditioned predictor for planning. WorldCoder [19] constructs executable transition and reward models that a planner can query. These examples emphasize predicting the consequences of actions, while a policy selects which actions to take. The two learning targets can be combined: WorldVLA [62] jointly learns action generation and environment prediction within one model.

2.4 Benchmarks

General multimodal benchmarks evaluate a broad range of visual-language tasks. MME [50] and MMBench [51] cover perception and reasoning, while MM-Vet [52] examines combinations of visual-language capabilities. MMMU [53] extends evaluation to academic subjects, Video-MME [54] covers video understanding, and MEGA-Bench [55] includes diverse tasks and output formats.

Capability-specific benchmarks evaluate particular abilities, such as spatial understanding. For example, 3DSRBench [10] examines three-dimensional relations in images, while VSI-Bench [13] evaluates spatial understanding from videos. CLEVRER [22], Physion [23], and PHYRE [24] provide foundational evaluations of physical reasoning, future outcomes, and action selection. Robot suites such as LIBERO [94] and CALVIN [95] examine task execution, while 4DCodeBench [16] requires reconstruction as an executable dynamic scene.

These benchmarks differ in their observations, tasks, outputs, and data. We also review methods that explain how the evaluated spatial abilities are learned and used.

2.5 Spatial Intelligence

Building on multimodal perception and understanding, spatial intelligence requires information about geometry, relations, and change. Relevant quantities include object position, size, orientation, camera pose, and motion. A useful representation must connect these quantities across reference frames and observations. Spatial supervision teaches models to recover or reason about them, while memory retains information that is no longer visible [8], [38], [65].

Different spatial operations place different demands on this information. Moving a camera changes the observed projection without necessarily moving an object. Moving the object changes the environment state. Simulating its subsequent motion requires a transition model, while grasping it requires an action expressed in the controller's coordinates. World modeling learns the consequences of changes; action supervision connects observations and instructions with control [56], [61].

We follow spatial information from its source and encoding to learning and use. Evaluations then test whether a change in input, representation, or training improves measurement, integration, memory, simulation, or action under a stated protocol.

3. Taxonomy

This survey organizes spatial intelligence evaluation along four complementary dimensions. Inputs and Interfaces describe what a model observes and how it obtains feedback. Spatial Capabilities describe the operation required, including perception, memory, simulation, action, and world construction. Evaluation describes when information becomes available, which outputs are required, and how performance is measured. Data Sources describe where observations, training signals, and reference answers originate.

A benchmark can involve several capabilities and access settings. We therefore classify concrete tasks and reported configurations rather than assigning every paper to one category. Model inputs, training resources, and evaluator-only information are recorded separately.

4. Inputs and Interfaces

Spatial tasks begin with the information available to a model. Images provide projected observations, multiple views expose additional geometry, and interactions reveal the effects of actions. Models must also encode this information in a form that supports the required computation. We review image and video observations, spatial inputs and their encoding, and interaction interfaces.

4.1 Image

4.1.1 Single Image

Single-image tasks require models to solve problems using a single image as visual input. A single image provides only a two-dimensional projection, so three-dimensional quantities such as object size and distance generally need to be estimated from visual cues and prior knowledge. 3DSRBench [10] evaluates height, orientation, location, and multi-object relations. Spatial457 [28] further distinguishes two-dimensional relations, three-dimensional pose, and six-dimensional relations. NuScenes-SpatialQA [11] constructs image questions using associated driving annotations.

Spatial information can be introduced through the visual encoder or through training supervision. SpatialVLM [8] estimates depth, lifts image content into an estimated three-dimensional representation, and constructs spatial question-answer data. The estimated geometry supplies training information; it does not make ground-truth point clouds a default deployment input. This route should be distinguished from a system that receives measured depth at inference.

4.1.2 Multiple Images

Multi-image tasks require models to solve problems using multiple images as visual input, without requiring temporal continuity between them. Multi-view tasks are a subset in which the images show the same object or environment from different viewpoints. These tasks can involve entity correspondence, reference-frame conversion, and fusion. MMSI-Bench [31] combines multiple real images, SCOPE [32] examines viewpoint and perspective changes, and CityCube [33] connects observations across urban platforms.

Methods can align views through learned features or explicit geometric estimates. VGGT [64] predicts several geometric quantities jointly from image sets. What Makes a Good Visual Representation for Spatial Understanding in MLLMs? [65] studies how geometric representations transfer into spatial understanding. Together, these works motivate separate questions about the quality of recovered geometry and the ability of a downstream model to use it. More views increase available evidence, but do not determine the effectiveness of the fusion mechanism.

4.2 Video

Video consists of a temporally ordered sequence of frames captured over time. Unlike a general collection of images, its frames come from a continuous temporal process and may capture changes in objects, the environment, or the camera viewpoint. Video-based tasks require models to solve problems using information from this sequence. VSI-Bench [13] uses recorded indoor routes for spatial questions, while STI-Bench [14] includes size, distance, pose, displacement, speed, and acceleration. Video-LLaVA [6] and LLaVA-OneVision [7] provide examples of models that accept frame sequences through a multimodal interface.

Frame sampling determines which evidence reaches the model. The image-to-video representation study [81] compares matched image and video diffusion backbones. At the same model size, video training improves all ten evaluated readout tasks, with larger changes in point tracking, camera pose, and motion recognition than in scene and object classification. The training objective therefore affects which spatial information can be recovered from the features.

4.3 Spatial Inputs

Spatial inputs include depth maps, three-dimensional geometry such as point clouds, camera or object poses, and agent states. These quantities may be measured, estimated from visual observations, or provided through textual descriptions.

Textual representations express spatial quantities and relations in language. SiT-Bench [67] evaluates spatial reasoning from such descriptions.

The construction of the description matters: comparisons with visual inputs should account for differences in the information provided.

4.3.1 Depth and Geometry

Explicit geometry may be measured, estimated, or used only to construct reference answers. ScanRefer [18] supplies a point cloud and referring expression for object localization. By contrast, NuScenes-SpatialQA [11] uses three-dimensional annotations to construct questions about camera images. OSI-Bench [35] uses multiple sensors and reconstruction tools to establish metric references without requiring all tested systems to receive those sensors.

Geometry may also be estimated within a system. SpatialVLA [99] uses estimated depth to provide three-dimensional position information, while GEN3C [39] constructs an estimated three-dimensional cache to guide video generation. Evaluations should distinguish measured geometry from estimated geometry and identify the information available during inference.

4.3.2 Pose and State

Camera poses describe the camera's position and orientation, while agent states may include end-effector pose, joint positions, and gripper state. Past or candidate actions can provide additional conditions for predicting subsequent observations. V-JEPA 2-AC [61] uses actions and end-effector states for visual prediction; DINO-WM [88] uses actions and, where available, proprioception to predict future visual features.

State inputs can contain information that is difficult to recover from RGB alone. Physion [23], for example, reports visual models separately from models with privileged physical state. Likewise, using true camera poses to retrieve memory differs from estimating poses during deployment. These conditions help explain performance and must remain visible in comparisons.

4.4 Interactive Environment

An interactive environment allows models to explore, take actions, and receive new observations and feedback. These actions may change the viewpoint, reveal previously unseen regions, or alter the environment. The simulated component of S³-Bench [15] permits movement, rotation, and scanning to gather spatial information. PHYRE [24] allows models to place objects and observe the resulting physical outcomes. Navigation and manipulation environments provide subsequent observations as models move through a scene or interact with objects.

For generated scenes and programs, execution provides feedback such as errors, rendered images, simulation states, and test results. 4DCodeBench [16], PhysCodeGym [17], and PhysCodeBench [43] involve generating and executing scene or simulation code. Code World Models [92] uses unit tests and environment trajectories to guide program search. Models can inspect these outputs and use the feedback to revise their programs.

5. Spatial Capabilities

This section reviews the spatial capabilities developed in existing models and the tasks they support. We organize these capabilities into perception and understanding, memory and reasoning, and action.

5.1 Perception and Understanding

State recovery asks what entities are present, where they are located, and how their geometry or motion can be measured. ScanRefer [18] studies localization in a scan. VSI-Bench [13], STI-Bench [14], and OSI-Bench [35] include distance, dimensions, orientation, or motion questions. Their reference quantities may come from sensors and reconstruction even when the model receives only RGB.

Spatial relation tasks ask how objects are positioned or oriented relative to each other and to a reference frame. 3DSRBench [10] separates height, orientation, location, and multi-object relations. Spatial457 [28] distinguishes simpler image relations from three-dimensional pose and six-dimensional relations. NuScenes-SpatialQA [11] additionally includes multi-hop questions.

Cross-view integration requires observations to refer to a common environment. Models must identify the same entity, align reference frames, and preserve relations as the camera changes. Simultaneously supplied views test this integration; retaining the information after a view is no longer accessible additionally requires memory. MMSI-Bench [31], SCOPE [32], and CityCube [33] evaluate related aspects through multi-image tasks. In MMSI-Bench, GPT-5 is reported at 41.9%, while humans exceed 97% under its choice-question protocol [31].

5.2 Memory and Reasoning

Spatial memory tasks ask models to retain locations, layouts, and relations after they leave the current view. VSI-Bench [13] includes route-level questions, while LongSpace [68] focuses on long environment videos and recall. OpenEQA [104] distinguishes questions answered from observation history from questions requiring exploration; its spatial tasks are part of a broader embodied-question set. KilometerVision [105] extends spatial integration to long walking tours and map-related tasks, without measuring navigation execution. OVO-S-Bench [20] separates several levels of streaming spatial tasks, including global allocentric mapping, which is the lowest-scoring level for 32 of 38 evaluated systems in v2.

Answering a recall question and generating a consistent revisit are complementary tests. MBench [36] evaluates memory in generated continuations, and LOCI [37] studies streaming world memory. A comparison across these tasks should match observed history and storage conditions, rather than attributing every gain from additional frames or true poses to the memory mechanism.

Persistent spatial information must be updated when objects move, a new viewpoint resolves ambiguity, or an earlier estimate proves wrong. S³-Bench [15] and MemAct [69] connect accumulating observations with spatial questions. WorldMem [38] uses time conditions to distinguish states associated with different moments.

Spatial transformation tasks ask how a configuration changes under rotation, movement, folding, or a change of reference frame. SpatialViz-Bench [71] studies spatial visualization through several geometric tasks. MentalGrid [72] examines initial perception, intermediate state evolution, and terminal answers, while SpaceCast-Bench [12] specifies scene transformations and asks about resulting relations.

Some properties must be inferred from interaction rather than appearance. ComPhy [30] asks about hidden physical attributes and their effects on observed motion. IntPhys [29] uses physically possible and impossible events, while Physion [23] tests consequences across several physical scenarios.

Future prediction withholds the target outcome. Physion [23] provides a video prefix and asks whether two objects will later contact. CLEVRER [22] includes future event questions. These tasks score an outcome judgment and can be approached through direct inference or an explicit simulation. MPMWorlds [42] instead fits an observed simulation prefix and evaluates its continuation on a hidden suffix. Physics-IQ [87] compares generated continuations with recordings of real physical experiments.

Intervention tasks specify a change and ask how its consequences differ. CLEVRER [22] uses event-level counterfactuals, CausalVQA [34] includes hypothetical variants from real video, and SpaceCast-Bench [12] applies specified geometric transformations. What-If World [49] tests intervention-conditioned generated videos.

Executable interfaces permit variables to be changed directly. ScratchWorld [45] evaluates consequences of state changes in an executable setting, while PhysCodeGym [17] tests selected reconstructed scenes under altered forces, contacts, or initial conditions. Each evaluation must state what changed and what remained fixed. A correct response to one intervention does not establish all physical mechanisms, but paired changes can reveal failures concealed by a reconstruction of the original observation.

Methods retain different content. MemAct [69] organizes history for streaming understanding and active exploration. WorldMem [38] retrieves stored image information for view generation. Spatial Memory Agent [70] instead stores experience-grounded procedures for solving tasks. Environment memory records what is where; procedure memory records how to approach a class of problems. Their gains should be evaluated against the content each is intended to preserve.

5.3 Action

Spatial grounding connects instructions with entities, locations, and target configurations. A robot must identify both the intended object and the spatial condition to satisfy. Spa-Bench [76] uses real-robot tasks and matched instruction controls to distinguish relational grounding from the remaining manipulation problem. ESI-Bench [75] connects spatial judgments with interaction in a simulated environment.

Navigation and manipulation require spatial information to support feasible actions. SayCan [73] combines language-based proposals with skill feasibility, while VoxPoser [74] constructs three-dimensional value maps for motion planning. VLAs such as RT-2 [56], OpenVLA [57], and π0 [58] learn action generation from demonstrations. LIBERO [94], CALVIN [95], and RoboCasa [96] provide different environments and task structures for evaluating execution.

Active evidence acquisition asks which observation or interaction would reduce uncertainty. The simulated component of S³-Bench [15] allows movement, rotation, and scanning, whereas its real-video component follows fixed trajectories. MemAct [69] combines remembered information with active camera exploration. Mine Odyssey [78] extends evaluation to long-horizon spatial agent tasks in an interactive world.

Scene reconstruction externalizes spatial understanding as geometry, a digital twin, or an executable world. 4DCodeBench [16] reconstructs dynamic scenes from reference videos and evaluates appearance, geometry, trajectories, and events. Vid2Sim [40] and PhysTwin [41] recover geometric and physical representations from captured observations under their respective object and material assumptions.

Executable representations express scenes or transition rules as programs. A scene program can construct objects and constraints; an animation program can specify trajectories; a simulator configuration can provide geometry and physical parameters; and a transition program can implement the effect of actions. Their expressiveness overlaps, but the behavior they encode differs.

Executable construction may begin from observations, textual requirements, or interaction traces. PhysCodeBench [43] supplies simulation specifications and engine documentation. ExeWorldBench [44] includes construction tasks at different levels of abstraction.

WorldCoder [19] and Code World Models [92] generate environment rules that planners can call. Code2Worlds [93] constructs simulation code from language and refines rendered dynamics through feedback. 4DCodeBench [16] instead evaluates reconstruction from reference videos. These works connect program synthesis with spatial modeling through different tasks and supervision.

The Code World Models benchmark [92] includes 18 environments with descriptions and trajectories. Evaluation should identify whether a program is expected to reproduce appearance, construct a mechanism, predict transitions, or support a planner.

Model revision incorporates new observations or feedback into a constructed world. WorldCoder [19] updates transition programs after interaction. PACE-Bench [46] evaluates code adaptation when physical conditions change, while PhysCodeGym [17] includes execution and physical inspection of reconstructed scenes.

6. Evaluation

Evaluation protocols determine what information is available to the model, which metrics are used, and what capabilities are evaluated. In this section, we review evaluation settings, outputs, metrics, and reliability.

6.1 Evaluation Settings

6.1.1 Static Evaluation

Static evaluation supplies the evidence before the answer or artifact is produced. It includes images, multiple views, scans, offline videos, and fixed prefixes. 3DSRBench [10] and VSI-Bench [13] are examples of prescribed-observation tasks. Physion [23] supplies a fixed prefix but hides the target future. Reports should state the supplied frames, query time, and whether history can be revisited.

6.1.2 Streaming Evaluation

Streaming evaluation supplies observations sequentially. The system may maintain state, retain a bounded cache, or repeatedly process an available prefix. OVO-S-Bench [20], Stream3D-Bench [21], and S³-Bench [15] use different incremental-access conditions. A complete description includes ingestion rate, history access, storage, query visibility, and response timing.

Some tasks permit a delayed answer after a condition becomes visible. This tests timely recognition and response. Stream3D-Bench [21] scores answer content and timing separately, distinguishing an incorrect answer from one delivered at the wrong moment. Forecasting instead requires an answer before the target evidence arrives. The distinction should be made at the task level rather than inferred from the word “forward” or “streaming.”

An observation history can be supplied directly as context, retrieved as selected frames, or compressed into a persistent state. LongSpace [68] studies spatial recall over long environment videos. MemAct [69] combines memory with active observation, while WorldMem [38] retrieves stored image information using pose and time conditions for generation.

These designs differ in what is retained and how it is queried. A longer context exposes more past evidence; a memory module may instead organize a fixed amount of evidence more effectively. History access and storage budgets are therefore reported alongside the representation.

6.1.3 Interactive Evaluation

Interactive evaluation returns feedback after actions. Actions may move a camera, change an environment, or execute a generated program. PHYRE [24] limits physical trials, robot suites evaluate repeated execution, and coding benchmarks provide tool or render feedback [16], [17], [43]. These feedback channels have different information content.

Protocols should specify what is reset between trials and what carries over. Hidden-future targets must be excluded from generation and candidate selection. Controlled reruns should record changed variables, and revision tests should identify the new evidence. Interaction budgets include observations, environment steps, candidate programs, and execution time.

The value of exploration depends on both acquired information and its cost. Evaluation should report whether the chosen action reveals necessary evidence, how many actions are used, and whether the final task is completed. WorldCoder [19] and PHYRE [24] provide related settings in which action outcomes inform later choices, although successful trial-and-error need not imply an explicit learned dynamics model.

6.2 Evaluation Outputs

Answers may be multiple-choice, free-form, numerical, or structured. A relation answer exposes a conclusion, whereas a coordinate or pose exposes an estimate that can be checked more directly. Scene graphs, trajectories, and maps additionally expose relationships across entities or time. The evaluator should score the requested quantities rather than treating all outputs as generic text [11], [13], [14].

A plan states intended steps; an action output specifies a control or command. Offline action matching can test agreement with a recorded sequence, while environmental execution tests its actual consequences. The available controller, action normalization, execution frequency, and use of intermediate goals should accompany reported results [56], [58], [61], [63].

Generated video exposes future or revisited observations. Geometry, digital twins, and programs expose different amounts of structure. Vid2Sim [40] and PhysTwin [41] recover parameterized models, while 4DCodeBench [16] and PhysCodeGym [17] require executable scenes. The evaluation should follow the properties actually inspected: appearance, geometry, state, transitions, or response to changed conditions.

Reconstruction provides a reusable artifact, but its validation should follow its intended use. Matching an observed trajectory tests fitting. Held-out views test additional geometry, while hidden futures or new interactions test behavior beyond the fitting observations. These settings should be reported separately even when they use the same recovered scene.

For code-to-3D and code-to-video systems, the generated artifact should therefore be examined directly. Correct scene layout, a prescribed animation, and a model that responds correctly to new conditions are distinct achievements. When an external engine supplies physical dynamics, the learned contribution may lie in recovering the scene, selecting mechanisms, or estimating parameters. That contribution should be evaluated alongside the resulting simulation.

Revision should be evaluated through both correction and retention. A repaired program may run while still encoding an incorrect trajectory, and a changed parameter may improve a new observation while damaging earlier fits. Reporting the new evidence, edited components, retained behavior, and revision cost helps distinguish syntax repair from updating an environment model.

6.3 Evaluation Metrics

Spatial accuracy includes relation correctness, numerical error, localization, pose error, and geometric reconstruction. VSI-Bench [13] separates numerical MRA from choice accuracy. CLEVRER [22] reports both per-option and stricter per-question scores. These quantities should retain their definitions and should not be averaged across benchmarks as one spatial-intelligence percentage.

The answer format also matters. A qualitative relation, a distance interval, and a continuous estimate are different targets. VSI-Bench uses numerical MRA and choice accuracy for different questions. STI-Bench v6 reports Gemini-2.5-Pro at 41.4% on five-way questions [14]; this measures option selection under that protocol rather than continuous metric calibration.

Temporal consistency measures whether identities, locations, and state changes remain coherent across observations or generated frames. Tracking error and revisit consistency examine whether the same entity or place is maintained; MBench [36] evaluates persistence in generated content. Recall accuracy tests the accessibility of stored information and can complement consistency measures. Memory results should be accompanied by observed duration, revisit interval, storage, and historical access.

Updating can be evaluated through both new and previously established information. A model should incorporate the changed object while retaining unaffected relations. Final-answer accuracy alone may hide failures of entity binding or stale state. Tests that identify what changed, when it was observed, and which earlier facts should remain valid provide more direct evidence of state maintenance.

Prediction metrics should correspond to hidden events, future states, or action-conditioned trajectories. Physion [23] uses future contact, Physics-IQ [87] compares physical continuations, and MPMWorlds [42] evaluates a hidden suffix. Image similarity can supplement these targets, but may include background and decoder effects that are weakly related to an action-critical state.

Action evaluation includes success, path efficiency, attempts, contact accuracy, and time. PHYRE [24] uses success under an attempt budget. Navigation can report both reaching the goal and path efficiency. Robot suites measure task or sequence completion, but their environments, control interfaces, and training access differ [94], [95], [96].

Offline action scores should remain distinct from execution. The JEPA study [89], for example, uses an offline action score on DROID alongside success rates in other environments. Similarly, the inference time of one dynamics call is not the latency of a planner that evaluates many candidate sequences. Both outcome and computation belong in deployment-oriented evaluation.

6.4 Evaluation Reliability

Performance depends on supplied information and computation. True geometry, estimated geometry, robot state, historical frames, goal images, and oracle execution can each change a task. Reports should identify which are available to the model, which supervise training, and which are reserved for scoring.

Matched comparisons should also account for model training, context, storage, trials, and wall-clock time. A system with additional observations or a stronger planner may improve without changing the underlying representation. Cross-paper results can motivate hypotheses, while controlled within-paper comparisons provide more direct evidence about a component.

Reference quality and aggregation can affect conclusions. ReSTI [47] audits STI-Bench against source geometry, coordinate frames, and timestamps. Physics-IQ Verified [48] revises prompts, references, and scoring; its six-model ranking comparison yields Kendall τ = 0.46. These changes improve the measurement procedure rather than the evaluated models themselves.

Robot benchmarks also require interpretation. What Are We Actually Benchmarking in Robot Manipulation? [98] demonstrates strong task success with a comparatively simple shortcut probe on several LIBERO subsets. LIBERO-PRO [97] tests perturbations of objects, layouts, language, and environments. Such findings motivate stronger diagnostic controls; they do not prove that every successful policy lacks spatial capability. Benchmark version, prompts, splits, aggregation, and uncertainty should accompany results.

Feature readouts provide additional diagnostics. Their interpretation depends on the target, label source, probe capacity, and whether the backbone is frozen [80], [83], [84]. A nonlinear readout trained on geometric labels can perform substantial computation. A good probe score therefore complements, rather than replaces, tests of the model's original outputs.

Spatial transformation tasks permit a useful separation between recovering the initial configuration and applying a transformation. Intermediate-state evaluation can show whether an incorrect final answer follows from a perception error or from subsequent state updates. Geometric simulation may use text, features, diagrams, or explicit operations; success does not require a physical dynamics model.

Connections with spatial understanding require direct evidence. Beyond Spatial Benchmarks [77] studies whether spatially specialized models improve navigation in a fixed framework and investigates navigation-aligned training. VCN-Bench [79] separately diagnoses target identification from prior video and subsequent navigation. Its MV-DualVLN system reaches 37.8% navigation success on the subset with correct first target identification. Correct identification is therefore insufficient in that setting; the conditional statistic is not a training intervention showing how much better identification would improve navigation.

Reference answers may be obtained from sensors, human annotation, reconstruction, or simulation state. NuScenes-SpatialQA [11] constructs questions from associated three-dimensional annotations, while OSI-Bench [35] uses multisensor measurements and reconstruction to establish metric references. These reference sources remain distinct from the observations supplied to the tested model.

Estimated labels inherit errors from depth, camera assumptions, object boundaries, and language generation. ReSTI [47] shows how coordinate and timestamp mismatches can affect evaluation questions. Independent evaluation should therefore avoid relying only on the estimator that produced training supervision. Reports should distinguish estimated labels, measured quantities, and engine-derived state.

Held-out observations, new interactions, or calibrated measurements provide additional validation. The appropriate test depends on whether the claim concerns appearance, geometry, parameter recovery, or behavior. An alternative parameter setting may fit the same video, so reproducing the fitting observations alone need not identify the true physical parameters.

Spatial457 provides a useful within-model contrast: GPT-4o scores 74.46% on single-object questions, 42.41% on 3D pose, and 37.01% on 6D relations in the reported setting [28]. These differences identify task-dependent weaknesses, but do not alone isolate their architectural cause. Combining them with controlled changes to spatial inputs or supervision can distinguish failures of observation, encoding, and relation composition.

A readout maps features to a target such as depth or pose. When trained for diagnosis, it is commonly called a probe. Probe results describe information accessible under that mapping and its supervision. They complement input and training analyses, but do not by themselves establish that the original model uses the recovered quantity in reasoning or action [80], [83], [84].

Official reports and model cards provide evaluation results for specified model versions and configurations [1], [2], [61], [100], [103]. These sources should be considered alongside third-party evaluations, with benchmark protocols and model versions recorded separately.

7. Data Sources

Spatial data come from real-world observations and synthetic environments. These sources provide images, videos, geometry, and interaction records for training and evaluation.

7.1 Real-world Data

Real images and videos provide natural appearance, occlusion, scale variation, and motion. ScanNet [25] and ARKitScenes [26] pair indoor observations with geometry, while Ego4D [27] provides long egocentric video. VSI-Bench [13], STI-Bench [14], and OSI-Bench [35] use real observations to construct spatial tasks.

Natural data introduce calibration and annotation challenges. A scan-derived distance must correspond to the correct image, timestamp, entity, and coordinate frame. ReSTI [47] shows why these links should be checked. Real data increase ecological coverage, but do not automatically provide complete or error-free physical state.

Robot trajectories connect visual observations with actions and subsequent states. RT-2 [56], OpenVLA [57], and π0 [58] use demonstrations to learn control, while V-JEPA 2-AC [61] uses robot trajectories for action-conditioned prediction. These sources expose effects that are not available in independent image-caption pairs.

Their value depends on action coverage, embodiment, camera geometry, and recording quality. Task labels, rewards, state measurements, and action information are separate annotations. Describing a trajectory dataset as unlabeled does not imply that no state or action information is used. Evaluation should state which parts of this experience are required by each learning method.

7.2 Synthetic Data

Procedural scenes allow controlled changes to geometry, layout, viewpoint, and object combinations. They provide exact construction information and can generate spatial questions systematically. SpatialTree [63] combines existing tasks with constructed data, while transformation benchmarks use controlled scene changes to obtain resulting relations [12].

Coverage should be considered alongside size. A dataset can contain many samples while repeating a narrow range of relations or views. Synthetic data also support targeted tests in which one factor changes and others remain fixed. Such tests are useful for identifying what a model learned, even when natural data are needed to assess external validity.

Synthetic data are generated with rendering tools and physics simulators. CLEVRER [22] uses Bullet to simulate object motion and Blender to render the resulting trajectories into videos. Physion [23] uses ThreeDWorld (TDW), a Unity3D-based simulation environment. For manipulation, CALVIN [95] uses PyBullet, while RoboCasa [96] builds on the MuJoCo-based robosuite framework. These tools provide the simulated scenes and observations used to construct evaluation tasks and collect interaction data.

Simulation provides states, actions, contacts, and physical parameters that are difficult to measure in real videos. CLEVRER [22], Physion [23], and PHYRE [24] use this access for different reasoning and action tasks. LIBERO [94], CALVIN [95], and RoboCasa [96] support robot learning and execution in controlled environments.

8. Discussion

8.1 Spatial Representations

The route by which spatial information enters a model matters alongside its form. SpatialVLM [8] converts estimated geometry into language supervision; SpatialVLA [99] encodes estimated positions in visual tokens; [65] transfers external features and geometry through training targets. These routes introduce different spatial constraints even when the deployed system begins from RGB observations.

The controlled comparisons in [65] show why the interface matters. Latent fusion and distillation give comparable average gains, while explicit geometry benefits more from distillation through the tested pretrained decoding head. The task profile also changes: on VSI-Bench, the baseline, feature-distilled, geometry-distilled, and joint models score 57.64, 60.17, 60.42, and 62.15. On SQA3D, however, geometry distillation scores 56.58 and the joint model 56.44. Combining targets improves some tasks without making the combined representation best on every task.

DINO-WM [88] provides a complementary route: it retains frozen spatial patch features and learns their action-conditioned evolution. Its control results connect representation granularity with execution, whereas [65] connects representation transfer with spatial answers. The shared question is which spatial structure the downstream computation needs. A useful next comparison would vary the encoding or supervision within a fixed system, then test both the targeted spatial task and transfer to memory or action.

Feature granularity also affects downstream use. How Much 3D Do Video Foundation Models Encode? [80] distinguishes cross-view correspondence from recovering global geometry. Rethinking Implicit Spatial Representation [86] compares pooled features with spatially organized encodings in control and identifies limitations on fine manipulation tasks. DINO-WM's encoder comparison tests this distinction through planning outcomes rather than geometric readouts alone.

Representation studies distinguish possible bottlenecks. In [65], cross-view feature consistency is associated with gains from representation transfer, while latent and geometric supervision produce different task profiles. How Much 3D Do Video Foundation Models Encode? [80] shows that strong feature correspondence and strong global 3D recovery need not coincide. Together, these results separate matching observations from recovering a common spatial structure. Evaluations can then test whether improving either property also improves cross-view answers.

Video features provide another diagnostic interface. Does Physics Live in the Activations? [83] studies readouts of physical quantities and limited activation interventions. Why Do Video Diffusion Models Violate Physics? [90] examines attention components and changes to generation. Together, these studies connect represented quantities with behavior, but the strength of evidence depends on the input and intervention: recovering a quantity from a complete video is different from inferring it before an unseen outcome.

8.2 Capability Transfer

The studies form a progression from training effects to system use. SpatialTree [63] tests which tasks improve or decline after targeted supervision. Beyond Spatial Benchmarks [77] tests whether spatial specialization helps navigation within a fixed framework. VCN-Bench [79] separates target identification from later execution, showing where a system can still fail after a correct intermediate result. A cross-model correlation, a training intervention, and conditional execution success answer different questions. Together with existing task and method surveys [101], [102], these results motivate evaluating both the acquired spatial skill and its contribution to the complete system.

8.3 Simulation for Action

Simulation for action connects an environment representation, a transition model, and a decision procedure. The representation determines what state is available. WorldMem [38] retrieves image information for revisits, whereas GEN3C [39] projects estimated geometry for view-aligned generation. These mechanisms support persistence and viewpoint consistency. Hidden-future tests such as Physics-IQ [87] additionally examine evolution beyond observed evidence; GEN3C's dynamic novel-view setting instead uses corresponding-time video evidence.

The next distinction is how simulated transitions affect behavior. DreamerV3 [60] trains an actor on imagined trajectories. DINO-WM [88] and V-JEPA 2-AC [61] evaluate candidate action sequences through learned feature dynamics during planning. WorldVLA [62] tests the benefit of joint action and world prediction during training. The reported gains from the auxiliary objective do not establish that the deployed policy searches simulated futures. These are distinct ways for world modeling to support action.

Programmatic models expose the transition computation to a planner. WorldCoder [19] and Code World Models [92] synthesize rules that can be queried under candidate actions. Code2Worlds [93] and 4DCodeBench [16] instead construct rendered environments from different specifications. Their connection is executable structure, not an identical decision loop. Evaluation should check the recovered state, the response to unseen conditions, and any resulting improvement in action, while separating the contributions of the generated program, external engine, and planner.

Predictive accuracy and decision performance should be connected through explicit experiments. The JEPA world-model study [89] examines encoders, action conditioning, multistep training, and planners. Its checkpoint-level prediction–performance relationships vary by environment. These results support task-dependent evaluation of a predictor rather than a universal assumption that lower reconstruction error guarantees better control.

Feedback can also improve external representations without updating the foundation model's weights. WorldCoder [19] revises programs from transitions, and Code World Models [92] uses program search with execution-related feedback. Procedure memory [70] stores lessons for later tasks. Weight learning, state estimation, memory accumulation, and program revision are different routes to adaptation and should be described through the information each receives.

8.4 Generalization

Generalization depends on which aspect of the spatial problem changes. Viewpoint changes test observation robustness, layout changes test spatial configuration, parameter changes test dynamics, and new goals test task reuse. Cross-View Action Consistency [85], LIBERO-PRO [97], and the controlled physical-law study [91] examine different members of this set.

Synthetic and real data add another distinction. In S³-Bench, Qwen3-VL-8B exceeds InternVL3.5-8B on the simulated split, 41.1 versus 37.7, while their order reverses on real data, 39.0 versus 40.5 [15]. The result motivates reporting both domains. It does not isolate the cause of reversal, because appearance, scene structure, observations, and annotations can all differ.

Future comparisons should identify the changed factor and retain the relevant controls. A new viewpoint is different from a new physical parameter, and both differ from a new task that requires an unseen skill. Reporting these changes separately is more informative than one generalization average.

Physical generalization can be tested by changing parameter ranges or combinations. How Far Is Video Generation from World Model [91] controls a two-dimensional physical training environment and examines different generalization regimes. Its findings concern those data and models; they motivate distinguishing interpolation, parameter extrapolation, and new combinations rather than treating more data as a single variable.

9. Conclusion

Spatial intelligence requires models to obtain, organize, retain, and use information about geometry, state, and change. MLLMs provide an important perception and language-understanding foundation, while spatial learning, memory, world modeling, and action learning extend the problems that models can address. Their combinations are best understood through the information and mechanisms involved.

We reviewed representative benchmarks through inputs and interfaces, spatial capabilities, evaluation protocols, and data sources. Across these perspectives, the evidence shows task-dependent progress and limits on transfer. Evaluating how an input, representation, or training change affects a concrete spatial task provides a clearer account of progress than a single aggregate score. The resulting view connects perception and understanding with persistent state, simulation, and action while preserving what each experiment actually establishes.

References