NeurIPS 2026

SGIP: Learning Scene-Grounded Interaction Priorsfor Scene-Aware Human Motion Prediction

Donghyeon Soon1*, Jiil Yoon1*, Hyeondong Kim1, Daehee Park1,2†
1DGIST 2UNIVA
* Equal contribution.† Corresponding author.
SGIP teaser: previous gaze- and distance-based methods leave future grounding ambiguous; SGIP predicts an explicit interaction prior over objects and open space to ground future motion without extra sensing at inference

Figure 1: Overview of the proposed method. Scene-aware human motion prediction (HMP) requires resolving future-motion ambiguity. (a) Previous methods handle this ambiguity implicitly, relying on motion–scene distance or gaze without an explicit interaction target; geometry alone is ambiguous when several objects lie at similar distances, and gaze requires extra sensing. (b) Given the same observed motion, a person may approach a bag, sit on the sofa or chair, or freely navigate the room. SGIP resolves this by predicting an explicit scene-grounded interaction prior, a distribution over candidate scene tokens covering both object instances and free space, from the observed motion and 3D scene alone. Distilled at training time from an intent-aware teacher that combines semantic and geometric scores, the prior places the highest probability on the sofa token, which conditions both trajectory and full-body pose prediction to correctly anticipate walking toward and sitting on the sofa, with no extra sensing at inference.

Abstract

Human motion prediction (HMP) aims to forecast future movement from an observed motion sequence, and recent methods have incorporated 3D scene geometry and object semantics to improve long-term plausibility. However, the destination of future motion remains ambiguous in cluttered scenes: the same observed motion can lead to different interaction targets or navigation modes. Existing methods either handle this ambiguity implicitly or rely on auxiliary signals like gaze that are hard to obtain in practice.

To address this, we propose SGIP, which explicitly predicts where in the scene future motion will be grounded across both objects and open navigation regions, and uses it to condition future trajectory and full-body pose prediction. This scene-grounded interaction prior is learned via intent distillation: at training time only, a teacher model scores candidate scene regions by combining an LLM prompted with a textual description of the person's intent with geometric motion-scene features. A student model learns to reproduce these scores from only the observed motion and 3D scene, making intent and LLM unnecessary at test time.

Across multiple benchmark datasets, SGIP outperforms prior scene-aware baselines on both trajectory and pose accuracy in a realistic deployment setting where neither gaze nor intent is available at inference, demonstrating effectiveness of the proposed framework.

SGIP Overview

Scene-grounded human motion prediction. SGIP learns an interaction prior over objects and open space to predict future trajectories and full-body poses from observed motion and the 3D scene.

Why Future Motion Is Ambiguous

A short motion history does not uniquely determine a person's destination. Scene-aware prediction must distinguish among multiple interaction targets, reason about free navigation, and work with the observations available at deployment.

Problem 1
Multiple Possible Futures

The same motion prefix can precede approaching a bag, sitting on a chair, or walking through the room.

Problem 2
Implicit Scene Interaction

Scene features alone leave the interaction target implicit, without a directly supervised grounding decision.

Problem 3
Unavailable Auxiliary Cues

Gaze and textual intent can disambiguate motion, but they are difficult to obtain in practical deployment.

How SGIP Works

SGIP makes scene interaction an explicit, supervisable prediction target. An intent-guided teacher learns a distribution over candidate scene tokens during training. Through prior-level and motion-level distillation, an observation-only student learns to select a grounding token and predict the corresponding future motion.

  1. Observe Motion + 3D Scene History and environment
  2. Ground Interaction Prior Objects and open space
  3. Predict Trajectory + Pose Conditioned on the selected token
Overall Framework of SGIP
SGIP teacher-student framework with training-only intent guidance, prior and motion distillation, and a motion predictor for trajectory-aware full-body pose prediction.

Figure 2. Overall framework of SGIP. Candidate tokens represent object instances and a scene-global navigation region. The training-only teacher combines LLM-based intent scoring with geometric compatibility. The student learns from the teacher through prior and motion distillation, then conditions trajectory and full-body pose prediction on a selected scene token using only observed motion and the 3D scene at inference.

Step 1
Build Scene Tokens

Represent individual object instances and a scene-global navigation region as candidate grounding tokens.

Step 2
Learn an Intent-Guided Prior

At training time, combine LLM intent scores with motion-scene geometry to rank candidate tokens.

Step 3
Distill Prior and Motion

Transfer the teacher’s token distribution and future-motion predictions to an observation-only student.

Step 4
Ground and Predict Motion

Select the top-ranked scene token and predict future trajectory and full-body poses without auxiliary cues.

Alternative Futures from Scene Grounding
Two motion futures from a fixed SGIP student conditioned on different scene tokens.

Figure 3. Token-conditioned alternative futures. With the student, observed motion, and scene held fixed, changing only the grounding token produces different trajectories and final interaction poses. The panels compare the ground-truth interaction object with another high-probability token. Standard benchmark evaluation uses hard top-1 selection; these examples illustrate alternative token-conditioned futures.

Benchmark Results

On GIMO and PROX, SGIP improves trajectory and full-body pose prediction in the observation-only setting. Models that use additional or privileged cues are reported separately. Lower is better for all four metrics; best and second-best results are highlighted within each table.

Table 1. Observation-Only Human Motion Prediction on GIMO
All methods use observed motion and the 3D scene without auxiliary cues. * denotes gaze-free re-evaluation. SGIP reports mean ± standard deviation over five seeds.
MethodTraj-path ↓Traj-dest ↓MPJPE-path ↓MPJPE-dest ↓
LTD673890156.4203.0
SPGSN700893157.0202.2
AuxFormer732902165.1202.4
SIF3D*700823164.2199.6
GAP3DS*642701170.1206.4
SGIP (Ours)549 ± 5.0681 ± 7.6153.1 ± 1.8195.1 ± 2.6
Table 2. Observation-Only Human Motion Prediction on PROX
All methods use observed motion and the 3D scene without auxiliary cues. * denotes gaze-free re-evaluation.
MethodTraj-path ↓Traj-dest ↓MPJPE-path ↓MPJPE-dest ↓
LTD677753223.4226.6
SPGSN595663286.3271.3
AuxFormer553727199.2240.8
SIF3D*488659228.5217.4
SGIP (Ours)470576194.6197.7
Table 3. Models with Additional or Privileged Cues on GIMO
Auxiliary cues are inputs beyond observed motion and the scene. † denotes diagnostic privileged SGIP branches, which are not deployed. These results use different input conditions from the observation-only comparisons above.
MethodAuxiliary cueTraj-path ↓Traj-dest ↓MPJPE-path ↓MPJPE-dest ↓
LTD w/ gazeGaze655801155.6202.0
SPGSN w/ gazeGaze702982156.0202.0
AuxFormer w/ gazeGaze622746161.5201.1
BiFuGaze661727167.8205.0
SIF3DGaze590666156.6195.7
GAP3DSGaze575623141.2171.4
SGIPpriv†Gaze528635149.9190.6
SGIPpriv†Textual intent485556151.2187.8

Qualitative Results

Watch SGIP (right) and the gaze-free baseline SIF3D* (left) predict future motion in the kitchen and seminar-room scenes from GIMO. Both methods use observed motion and the 3D scene. The figures below include additional comparisons with the privileged teacher.

Observed motion Ground-truth future Predicted future

Kitchen

SIF3D* (no gaze) vs. SGIP

Top-down comparison of future full-body motion in a cluttered kitchen.

Seminar Room

SIF3D* (no gaze) vs. SGIP

Top-down comparison of future full-body motion around the seminar-room table.

GIMO: Trajectories and Full-Body Motion
SGIP and gaze-free SIF3D predictions in a kitchen, living room, and seminar room.

Figure 4. Qualitative comparison across three GIMO scenes. The top row shows SGIP and the bottom row shows the gaze-free baseline SIF3D*. Each scene includes a top-down trajectory, the final interaction frame, and a full-body pose sequence. Yellow: observed motion; red: ground-truth future; blue: predicted future.

Observation-Only Prediction and Privileged Guidance
The SGIP teacher and student in the top row, compared with SIF3D with and without gaze in the bottom row.

Figure 5. Comparing baselines and SGIP variants. Given the same observed prefix, the top row compares the privileged SGIP teacher and the deployed SGIP student; the bottom row shows SIF3D with and without gaze. Each panel includes a top-down trajectory, a final interaction frame, and a pose sequence. The student uses neither gaze nor textual intent at inference.

Ablation Study

Prior distillation provides explicit supervision for scene grounding, while motion distillation transfers the teacher's future-motion predictions. Combining both achieves the lowest trajectory and destination-pose errors in this ablation; prior-only distillation has the lowest MPJPE-path.

Table 4. Distillation Components on GIMO
All variants are observation-only at inference. The full model combines prior-level and motion-level distillation. Lower is better for every metric.
VariantTraj-path ↓Traj-dest ↓MPJPE-path ↓MPJPE-dest ↓
Prior + motion (full)549681153.1195.1
No distillation740838169.1204.8
Motion only856896164.1198.9
Prior only574688151.8198.6

BibTeX

@inproceedings{soon2026sgip,
  title     = {Learning Scene-Grounded Interaction Priors for Scene-Aware Human Motion Prediction},
  author    = {Soon, Donghyeon and Yoon, Jiil and Kim, Hyeondong and Park, Daehee},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}