wing / index.html
Jam1e3's picture
Mirror WING project page
9396f67 verified
Raw History Blame Contribute Delete
46.7 kB
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="WING transfers interaction knowledge from egocentric human videos to robot policies through interaction-centric spectral latent guidance.">
<meta name="theme-color" content="#182723">
<title>WING — World Action Learning</title>
<link rel="icon" type="image/svg+xml" sizes="any" href="https://mikuz12.github.io/wing/assets/favicon.svg?v=2">
<link rel="stylesheet" href="https://mikuz12.github.io/wing/assets/fonts/source-serif-4.css">
<link rel="stylesheet" href="styles.css?v=compact-notes-2">
<script src="experiments.js?v=compact-notes-15" defer></script>
<script src="script.js?v=scroll-reveal-1" defer></script>
</head>
<body>
<a class="skip-link" href="#abstract">Skip to abstract</a>
<header class="header" id="header">
<nav class="nav wrap" aria-label="Main navigation">
<a class="wordmark" href="#home">WING</a>
<details class="section-menu">
<summary><span id="current-section">Overview</span><span class="menu-chevron" aria-hidden="true">⌄</span></summary>
<div class="section-menu-links"><a href="#home" aria-current="location">Overview</a><a href="#overview">At a Glance</a><a href="#method">Methodology</a><a href="#experiments">What WING does</a><a href="#abstract">Abstract</a><a href="#bibtex">BibTeX</a></div>
</details>
<div class="nav-resources">
<a href="https://arxiv.org/abs/2610.03607" target="_blank" rel="noopener">Paper <span class="nav-badge">ARXIV</span></a>
<span class="nav-unavailable" role="link" aria-disabled="true" title="Code repository coming soon">Code <span class="nav-badge">COMING SOON</span></span>
<a href="#bibtex">BibTeX</a>
</div>
</nav>
</header>
<main>
<section class="hero" id="home" aria-labelledby="project-title">
<div class="hero-media"><canvas class="video-edge-extension" width="1120" height="540" aria-hidden="true"></canvas><video class="hero-video" id="hero-video" autoplay muted loop playsinline preload="metadata" poster="https://mikuz12.github.io/wing/assets/wing-cover-poster.jpg" aria-hidden="true"><source src="https://mikuz12.github.io/wing/assets/wing-cover-4k.mp4" type="video/mp4"></video>
<div class="hero-shade"></div></div>
<div class="hero-content wrap">
<h1 id="project-title"><span class="title-line"><img class="title-logo" src="https://mikuz12.github.io/wing/wing_logo.png" alt=""><span class="hero-wing">WING</span>: <em class="hero-title-italic">World Action Learning</em> via</span><span class="title-line">INteraction-Centric <em class="hero-title-italic">Spectral Latent Guidance</em></span></h1>
<p class="tagline">From human experience to robot intelligence.</p>
<div class="authors" aria-label="Authors"><span>Zhiming Liu<sup>†</sup></span><span>Yikun Miao<sup>†</sup></span><span>Ying Chen<sup>†</sup></span><span>Hongrui Yin</span><span>Fangqi Zhu</span><span>Xiaoyi Pang</span><span>Quanxin Shou</span><span class="author-row-break" aria-hidden="true"></span><span>Zhengyang Yan</span><span>Haodong Wang</span><span>Song Guo<sup class="corresponding-star">*</sup></span></div>
<div class="affiliations"><span>The Hong Kong University of Science and Technology, Hong Kong SAR, China</span></div>
<p class="author-notes"><span><sup>†</sup> Equal contribution</span><span><sup class="corresponding-star">*</sup> Corresponding author</span></p>
</div>
<a class="scroll-cue" href="#overview">SCROLL TO EXPLORE<span aria-hidden="true">↓</span></a>
<button class="video-toggle" id="video-toggle" type="button" aria-label="Pause background video">Ⅱ <span>Pause video</span></button>
</section>
<div class="reading-layout">
<aside class="page-sidebar" aria-label="Page contents">
<details class="contents-menu" open>
<summary>On this page</summary>
<nav class="sidebar-nav" aria-label="Section navigation">
<ul>
<li><a href="#overview">At a Glance</a></li>
<li><a href="#method">Methodology</a>
<ul><li><a href="#motivation-one">Motivation 1</a></li><li><a href="#wing-lam">WING-LAM</a></li><li><a href="#motivation-two">Motivation 2</a></li><li><a href="#spectral-guidance">Spectral guidance</a></li></ul>
</li>
<li><a href="#experiments">What WING does</a><ul><li><a href="#transfer-structure">Transferable structure</a></li><li><a href="#robot-performance">Robot performance</a></li><li><a href="#component-analysis">Core components</a></li><li><a href="#ego-pretraining">Ego pretraining</a></li><li><a href="#additional-experiments">Additional experiments</a></li><li><a href="#real-generalization">Generalization</a></li></ul></li>
<li><a href="#abstract">Abstract</a></li>
<li><a href="#bibtex">BibTeX</a></li>
</ul>
</nav>
</details>
</aside>
<div class="reading-content">
<section class="section wrap" id="overview">
<p class="eyebrow">HUMAN EXPERIENCE. ROBOT ACTION.</p>
<h2>At a Glance</h2>
<figure class="teaser-frame"><a href="https://mikuz12.github.io/wing/teaser.pdf" target="_blank" rel="noopener" aria-label="Open WING overview figure as PDF"><img src="https://mikuz12.github.io/wing/assets/teaser.png" width="3200" height="685" loading="lazy" alt="WING overview: transferring interaction knowledge from egocentric videos to robot actions through interaction-centric spectral latent guidance, with results on LIBERO, RoboTwin 2.0, RoboCasa and real-world tasks."></a></figure>
<p class="section-intro">Learning what transfers across embodiments.</p>
<div class="principles"><article><span class="number">01</span><h3>Focus on interaction</h3><p>Decouple interaction-relevant signals from task-irrelevant motion in egocentric videos, and distill them into latent actions.</p></article><article><span class="number">02</span><h3>Find shared dynamics</h3><p>Identify slowly varying temporal structures shared by human interactions and robot behaviors in the spectral domain.</p></article><article><span class="number">03</span><h3>Guide robot actions</h3><p>Use the shared spectral latent action to guide robot action generation at inference time.</p></article></div>
<div class="metrics" aria-label="Average benchmark success rates reported in the paper"><div><strong>99.20<span>%</span></strong><span>LIBERO</span></div><div><strong>93.80<span>%</span></strong><span>RoboTwin 2.0</span></div><div><strong>57.7<span>%</span></strong><span>RoboCasa–GR1</span></div></div>
<p class="metric-note">Average success rates reported in the paper.</p>
<section id="simulation" class="benchmark-comparison" aria-labelledby="comparison-title">
<div class="comparison-heading"><h3 id="comparison-title">WING simulation performance</h3><span class="comparison-unit">Success rate (%) · 0–100 scale</span></div>
<div class="benchmark-grid">
<section class="benchmark-column" aria-labelledby="libero-title"><h4 id="libero-title">LIBERO</h4><p class="benchmark-subtitle">Four-suite average</p><div class="benchmark-rows">
<button type="button" class="benchmark-row is-wing" data-benchmark-detail="{&quot;method&quot;: &quot;WING&quot;, &quot;benchmark&quot;: &quot;LIBERO&quot;, &quot;value&quot;: &quot;99.20&quot;, &quot;details&quot;: [[&quot;Spatial&quot;, &quot;99.00&quot;], [&quot;Object&quot;, &quot;100.00&quot;], [&quot;Goal&quot;, &quot;98.80&quot;], [&quot;Long&quot;, &quot;99.00&quot;]], &quot;source&quot;: &quot;Paper \u00b7 Table 2(a)&quot;, &quot;note&quot;: &quot;&quot;, &quot;comparison&quot;: &quot;+0.70 pp over the strongest shown baseline&quot;}" aria-label="WING, LIBERO, success rate 99.20 percent. Show details."><span class="benchmark-method">WING</span><span class="benchmark-track" aria-hidden="true"><span class="benchmark-fill" style="width:99.2%"></span></span><span class="benchmark-value">99.20<small>%</small></span></button>
<button type="button" class="benchmark-row" data-benchmark-detail="{&quot;method&quot;: &quot;Fast-WAM&quot;, &quot;benchmark&quot;: &quot;LIBERO&quot;, &quot;value&quot;: &quot;97.60&quot;, &quot;details&quot;: [[&quot;Spatial&quot;, &quot;98.20&quot;], [&quot;Object&quot;, &quot;100.00&quot;], [&quot;Goal&quot;, &quot;97.00&quot;], [&quot;Long&quot;, &quot;95.20&quot;]], &quot;source&quot;: &quot;Paper \u00b7 Table 2(a)&quot;, &quot;note&quot;: &quot;&quot;, &quot;comparison&quot;: &quot;WING +1.60 percentage points&quot;}" aria-label="Fast-WAM, LIBERO, success rate 97.60 percent. Show details."><span class="benchmark-method">Fast-WAM</span><span class="benchmark-track" aria-hidden="true"><span class="benchmark-fill" style="width:97.6%"></span></span><span class="benchmark-value">97.60<small>%</small></span></button>
<button type="button" class="benchmark-row" data-benchmark-detail="{&quot;method&quot;: &quot;\u03c0\u2080.\u2085&quot;, &quot;benchmark&quot;: &quot;LIBERO&quot;, &quot;value&quot;: &quot;96.90&quot;, &quot;details&quot;: [[&quot;Spatial&quot;, &quot;98.80&quot;], [&quot;Object&quot;, &quot;98.20&quot;], [&quot;Goal&quot;, &quot;98.00&quot;], [&quot;Long&quot;, &quot;92.40&quot;]], &quot;source&quot;: &quot;Paper \u00b7 Table 2(a)&quot;, &quot;note&quot;: &quot;&quot;, &quot;comparison&quot;: &quot;WING +2.30 percentage points&quot;}" aria-label="π₀.₅, LIBERO, success rate 96.90 percent. Show details."><span class="benchmark-method">π₀.₅</span><span class="benchmark-track" aria-hidden="true"><span class="benchmark-fill" style="width:96.9%"></span></span><span class="benchmark-value">96.90<small>%</small></span></button>
<button type="button" class="benchmark-row" data-benchmark-detail="{&quot;method&quot;: &quot;LingBot-VA&quot;, &quot;benchmark&quot;: &quot;LIBERO&quot;, &quot;value&quot;: &quot;98.50&quot;, &quot;details&quot;: [[&quot;Spatial&quot;, &quot;98.50&quot;], [&quot;Object&quot;, &quot;99.60&quot;], [&quot;Goal&quot;, &quot;97.20&quot;], [&quot;Long&quot;, &quot;98.50&quot;]], &quot;source&quot;: &quot;Paper \u00b7 Table 2(a)&quot;, &quot;note&quot;: &quot;&quot;, &quot;comparison&quot;: &quot;WING +0.70 percentage points&quot;}" aria-label="LingBot-VA, LIBERO, success rate 98.50 percent. Show details."><span class="benchmark-method">LingBot-VA</span><span class="benchmark-track" aria-hidden="true"><span class="benchmark-fill" style="width:98.5%"></span></span><span class="benchmark-value">98.50<small>%</small></span></button>
</div><p class="benchmark-gain"><strong>+0.70 pp</strong> over the strongest shown baseline</p></section>
<section class="benchmark-column" aria-labelledby="robotwin-title"><h4 id="robotwin-title">RoboTwin 2.0</h4><p class="benchmark-subtitle">Clean + randomized average</p><div class="benchmark-rows">
<button type="button" class="benchmark-row is-wing" data-benchmark-detail="{&quot;method&quot;: &quot;WING&quot;, &quot;benchmark&quot;: &quot;RoboTwin 2.0&quot;, &quot;value&quot;: &quot;93.80&quot;, &quot;details&quot;: [[&quot;Clean&quot;, &quot;94.56&quot;], [&quot;Randomized&quot;, &quot;93.04&quot;]], &quot;source&quot;: &quot;Paper \u00b7 Table 2(b)&quot;, &quot;note&quot;: &quot;&quot;, &quot;comparison&quot;: &quot;+0.20 pp over the strongest shown baseline&quot;}" aria-label="WING, RoboTwin 2.0, success rate 93.80 percent. Show details."><span class="benchmark-method">WING</span><span class="benchmark-track" aria-hidden="true"><span class="benchmark-fill" style="width:93.8%"></span></span><span class="benchmark-value">93.80<small>%</small></span></button>
<button type="button" class="benchmark-row" data-benchmark-detail="{&quot;method&quot;: &quot;Fast-WAM&quot;, &quot;benchmark&quot;: &quot;RoboTwin 2.0&quot;, &quot;value&quot;: &quot;91.83&quot;, &quot;details&quot;: [[&quot;Clean&quot;, &quot;91.88&quot;], [&quot;Randomized&quot;, &quot;91.78&quot;]], &quot;source&quot;: &quot;Paper \u00b7 Table 2(b)&quot;, &quot;note&quot;: &quot;&quot;, &quot;comparison&quot;: &quot;WING +1.97 percentage points&quot;}" aria-label="Fast-WAM, RoboTwin 2.0, success rate 91.83 percent. Show details."><span class="benchmark-method">Fast-WAM</span><span class="benchmark-track" aria-hidden="true"><span class="benchmark-fill" style="width:91.83%"></span></span><span class="benchmark-value">91.83<small>%</small></span></button>
<button type="button" class="benchmark-row" data-benchmark-detail="{&quot;method&quot;: &quot;\u03c0\u2080.\u2085&quot;, &quot;benchmark&quot;: &quot;RoboTwin 2.0&quot;, &quot;value&quot;: &quot;79.75&quot;, &quot;details&quot;: [[&quot;Clean&quot;, &quot;82.74&quot;], [&quot;Randomized&quot;, &quot;76.76&quot;]], &quot;source&quot;: &quot;Paper \u00b7 Table 2(b)&quot;, &quot;note&quot;: &quot;&quot;, &quot;comparison&quot;: &quot;WING +14.05 percentage points&quot;}" aria-label="π₀.₅, RoboTwin 2.0, success rate 79.75 percent. Show details."><span class="benchmark-method">π₀.₅</span><span class="benchmark-track" aria-hidden="true"><span class="benchmark-fill" style="width:79.75%"></span></span><span class="benchmark-value">79.75<small>%</small></span></button>
<button type="button" class="benchmark-row" data-benchmark-detail="{&quot;method&quot;: &quot;LingBot-VA 2.0&quot;, &quot;benchmark&quot;: &quot;RoboTwin 2.0&quot;, &quot;value&quot;: &quot;93.60&quot;, &quot;details&quot;: [[&quot;Clean&quot;, &quot;93.80&quot;], [&quot;Randomized&quot;, &quot;93.40&quot;]], &quot;source&quot;: &quot;Reported LingBot-VA 2.0 results&quot;, &quot;note&quot;: &quot;&quot;, &quot;comparison&quot;: &quot;WING +0.20 percentage points&quot;}" aria-label="LingBot-VA 2.0, RoboTwin 2.0, success rate 93.60 percent. Show details."><span class="benchmark-method">LingBot-VA 2.0</span><span class="benchmark-track" aria-hidden="true"><span class="benchmark-fill" style="width:93.6%"></span></span><span class="benchmark-value">93.60<small>%</small></span></button>
</div><p class="benchmark-gain"><strong>+0.20 pp</strong> over the strongest shown baseline</p></section>
<section class="benchmark-column" aria-labelledby="robocasa-title"><h4 id="robocasa-title">RoboCasa–GR1</h4><p class="benchmark-subtitle">24-task tabletop average</p><div class="benchmark-rows">
<button type="button" class="benchmark-row is-wing" data-benchmark-detail="{&quot;method&quot;: &quot;WING&quot;, &quot;benchmark&quot;: &quot;RoboCasa\u2013GR1&quot;, &quot;value&quot;: &quot;57.7&quot;, &quot;details&quot;: [], &quot;source&quot;: &quot;Paper \u00b7 Table 1&quot;, &quot;note&quot;: &quot;Average over 24 tabletop tasks.&quot;, &quot;comparison&quot;: &quot;+2.30 pp over the strongest shown baseline&quot;}" aria-label="WING, RoboCasa–GR1, success rate 57.7 percent. Show details."><span class="benchmark-method">WING</span><span class="benchmark-track" aria-hidden="true"><span class="benchmark-fill" style="width:57.7%"></span></span><span class="benchmark-value">57.7<small>%</small></span></button>
<button type="button" class="benchmark-row" data-benchmark-detail="{&quot;method&quot;: &quot;GR00T-N1.6&quot;, &quot;benchmark&quot;: &quot;RoboCasa\u2013GR1&quot;, &quot;value&quot;: &quot;47.6&quot;, &quot;details&quot;: [], &quot;source&quot;: &quot;Paper \u00b7 Table 1&quot;, &quot;note&quot;: &quot;Average over 24 tabletop tasks.&quot;, &quot;comparison&quot;: &quot;WING +10.10 percentage points&quot;}" aria-label="GR00T-N1.6, RoboCasa–GR1, success rate 47.6 percent. Show details."><span class="benchmark-method">GR00T-N1.6</span><span class="benchmark-track" aria-hidden="true"><span class="benchmark-fill" style="width:47.6%"></span></span><span class="benchmark-value">47.6<small>%</small></span></button>
<button type="button" class="benchmark-row" data-benchmark-detail="{&quot;method&quot;: &quot;Fast-WAM\u2020&quot;, &quot;benchmark&quot;: &quot;RoboCasa\u2013GR1&quot;, &quot;value&quot;: &quot;51.9&quot;, &quot;details&quot;: [], &quot;source&quot;: &quot;Paper \u00b7 Table 1&quot;, &quot;note&quot;: &quot;Our reproduction, trained to convergence.&quot;, &quot;comparison&quot;: &quot;WING +5.80 percentage points&quot;}" aria-label="Fast-WAM†, RoboCasa–GR1, success rate 51.9 percent. Show details."><span class="benchmark-method">Fast-WAM†</span><span class="benchmark-track" aria-hidden="true"><span class="benchmark-fill" style="width:51.9%"></span></span><span class="benchmark-value">51.9<small>%</small></span></button>
<button type="button" class="benchmark-row" data-benchmark-detail="{&quot;method&quot;: &quot;LDA-1B&quot;, &quot;benchmark&quot;: &quot;RoboCasa\u2013GR1&quot;, &quot;value&quot;: &quot;55.4&quot;, &quot;details&quot;: [], &quot;source&quot;: &quot;Paper \u00b7 Table 1&quot;, &quot;note&quot;: &quot;Average over 24 tabletop tasks.&quot;, &quot;comparison&quot;: &quot;WING +2.30 percentage points&quot;}" aria-label="LDA-1B, RoboCasa–GR1, success rate 55.4 percent. Show details."><span class="benchmark-method">LDA-1B</span><span class="benchmark-track" aria-hidden="true"><span class="benchmark-fill" style="width:55.4%"></span></span><span class="benchmark-value">55.4<small>%</small></span></button>
</div><p class="benchmark-gain"><strong>+2.30 pp</strong> over the strongest shown baseline</p></section>
</div>
<p class="comparison-note">Hover, focus, or tap a bar for details. All bars use the same 0–100% scale.</p>
</section>
<div class="benchmark-tooltip" id="benchmark-tooltip" role="tooltip" hidden></div>
<div id="in-action" class="demo-gallery" aria-labelledby="gallery-title">
<div class="gallery-heading"><h3 id="gallery-title">WING in action</h3><div class="gallery-controls"><button id="preview-toggle" type="button">Pause previews</button><button type="button" class="gallery-prev" aria-label="Previous videos" aria-controls="demo-track">←</button><button type="button" class="gallery-next" aria-label="Next videos" aria-controls="demo-track">→</button></div></div>
<p class="gallery-instruction">Click on any video below to expand it.</p>
<ul class="demo-track" id="demo-track" aria-label="Real-world robot demonstrations">
<li class="demo-card"><button class="demo-open" type="button" data-video="https://mikuz12.github.io/wing/assets/demos/battery-insertion.mp4" data-title="Battery Insertion" aria-label="Expand Battery Insertion video">
<span class="demo-thumbnail"><video muted playsinline preload="none" poster="https://mikuz12.github.io/wing/assets/demos/battery-insertion.jpg" data-src="https://mikuz12.github.io/wing/assets/demos/battery-insertion.mp4" aria-hidden="true"></video><span class="demo-expand" aria-hidden="true">↗</span></span>
<span class="demo-title">Battery Insertion</span>
</button></li>
<li class="demo-card"><button class="demo-open" type="button" data-video="https://mikuz12.github.io/wing/assets/demos/battery-assembly.mp4" data-title="Battery Assembly" aria-label="Expand Battery Assembly video">
<span class="demo-thumbnail"><video muted playsinline preload="none" poster="https://mikuz12.github.io/wing/assets/demos/battery-assembly.jpg" data-src="https://mikuz12.github.io/wing/assets/demos/battery-assembly.mp4" aria-hidden="true"></video><span class="demo-expand" aria-hidden="true">↗</span></span>
<span class="demo-title">Battery Assembly</span>
</button></li>
<li class="demo-card"><button class="demo-open" type="button" data-video="https://mikuz12.github.io/wing/assets/demos/pack-objects.mp4" data-title="Pack Objects" aria-label="Expand Pack Objects video">
<span class="demo-thumbnail"><video muted playsinline preload="none" poster="https://mikuz12.github.io/wing/assets/demos/pack-objects.jpg" data-src="https://mikuz12.github.io/wing/assets/demos/pack-objects.mp4" aria-hidden="true"></video><span class="demo-expand" aria-hidden="true">↗</span></span>
<span class="demo-title">Pack Objects</span>
</button></li>
<li class="demo-card"><button class="demo-open" type="button" data-video="https://mikuz12.github.io/wing/assets/demos/stack-cups.mp4" data-title="Stack Cups" aria-label="Expand Stack Cups video">
<span class="demo-thumbnail"><video muted playsinline preload="none" poster="https://mikuz12.github.io/wing/assets/demos/stack-cups.jpg" data-src="https://mikuz12.github.io/wing/assets/demos/stack-cups.mp4" aria-hidden="true"></video><span class="demo-expand" aria-hidden="true">↗</span></span>
<span class="demo-title">Stack Cups</span>
</button></li>
</ul>
</div>
</section>
<section class="section wrap method-section" id="method" aria-labelledby="method-title">
<p class="eyebrow">HOW IT WORKS</p>
<h2 id="method-title">Methodology of WING</h2>
<p class="section-intro">Learn interactions from human videos. Turn shared dynamics into robot actions.</p>
<figure class="teaser-frame method-frame">
<video id="method-video" muted playsinline preload="auto" poster="https://mikuz12.github.io/wing/assets/method/wing-method-poster.png?v=20260930" aria-label="Animated overview of WING: interaction-centric latent learning and spectral guidance" aria-describedby="method-description">
<source src="https://mikuz12.github.io/wing/assets/method/wing-method.mp4?v=20260930" type="video/mp4">
<a href="https://mikuz12.github.io/wing/assets/method/wing-method.mp4?v=20260930">Watch the WING method animation.</a>
</video>
<figcaption class="method-playback"><button id="method-toggle" type="button" aria-controls="method-video">Play animation</button></figcaption>
</figure>
<div class="method-story" id="method-description">
<section class="method-chapter" id="motivation-one" aria-labelledby="motivation-one-title">
<div class="method-explanation">
<p class="eyebrow">MOTIVATION 1</p>
<h3 id="motivation-one-title">Head motion in egocentric videos</h3>
<p>Egocentric videos capture rich hand–object interactions, but the camera also moves with the wearer's head. Looking around or leaning forward shifts the entire scene, mixing observer motion with the motion of the hands and objects.</p>
</div>
<div class="ego-examples">
<figure class="ego-example">
<video muted playsinline preload="none" data-ego-src="https://mikuz12.github.io/wing/assets/method/ego-head-motion-1.mp4" poster="https://mikuz12.github.io/wing/assets/method/ego-head-motion-1.jpg" aria-label="First egocentric video illustrating head and camera motion"></video>
<figcaption><span>Egocentric view 01</span><button type="button" class="ego-toggle">Play video</button></figcaption>
</figure>
<figure class="ego-example">
<video muted playsinline preload="none" data-ego-src="https://mikuz12.github.io/wing/assets/ego_video2.mp4" poster="https://mikuz12.github.io/wing/assets/method/ego-head-motion-2.jpg" aria-label="Second egocentric video illustrating head and camera motion"></video>
<figcaption><span>Egocentric view 02</span><button type="button" class="ego-toggle">Play video</button></figcaption>
</figure>
</div>
<div class="method-explanation">
<p>A latent action model trained to reconstruct the next frame must explain both kinds of change. Its encoding can therefore absorb head motion alongside the interaction itself. This observer-specific motion does not describe the manipulation skill we want a robot to learn, and can obscure the action information that should transfer.</p>
<p class="method-insight"><strong>Our goal:</strong> preserve hand–object interactions while suppressing camera motion.</p>
</div>
</section>
<section class="method-chapter" id="wing-lam" aria-labelledby="wing-lam-title">
<div class="method-explanation">
<p class="eyebrow">OUR SOLUTION</p>
<h3 id="wing-lam-title">WING-LAM: interaction-centric latent actions</h3>
<p>WING-LAM explicitly separates camera motion from interaction motion, then distills the interaction representation into a latent action model that takes only RGB frame pairs.</p>
</div>
<figure class="teaser-frame wing-lam-frame">
<a href="https://mikuz12.github.io/wing/assets/wing-lam.pdf" target="_blank" rel="noopener" aria-label="Open the WING-LAM architecture figure as a PDF"><img src="https://mikuz12.github.io/wing/assets/method/wing-lam.png" width="2400" height="590" loading="lazy" alt="WING-LAM: geometric routing separates interaction and background tracks into interaction and camera latent branches; the interaction representation is distilled into a model trained on original and camera-warped RGB frame pairs."></a>
<figcaption>Geometric routing separates motion; distillation transfers interaction cues to an RGB-only model.</figcaption>
</figure>
<div class="method-explanation wing-lam-steps">
<p><strong>1. Separate the motion signals.</strong> Point tracks and hand–object region masks distinguish background motion from interaction motion. Background tracks estimate a global camera warp. Compensating for this warp reveals the residual motion associated with hand–object interactions.</p>
<p><strong>2. Learn two latent branches.</strong> A teacher model encodes camera motion and interaction motion separately. The camera branch predicts the global warp, while the interaction branch predicts the residual displacements. Together, the decoded motions reconstruct observed trajectories.</p>
<p><strong>3. Distill the interaction representation.</strong> An RGB-only student learns the teacher's interaction latent from frame pairs. We also apply camera-like warps that preserve the interaction target, encouraging the student to produce consistent latent actions despite changes in viewpoint. The resulting model extracts interaction-centric latent actions without requiring point tracks or region masks as inputs.</p>
<p class="method-insight">These latent actions preserve manipulation structure with less sensitivity to observer motion, providing the foundation for WING's spectral latent guidance.</p>
</div>
</section>
<section class="method-chapter" id="motivation-two" aria-labelledby="motivation-two-title">
<div class="method-explanation">
<p class="eyebrow">MOTIVATION 2</p>
<h3 id="motivation-two-title">Shared interactions, different execution rhythms</h3>
<p>Removing camera motion does not eliminate the gap between human and robot behavior. Even when both perform the same interaction, they can differ in speed, pauses, and the timing of individual movements. Interaction-centric latent actions still encode these temporal differences, making direct transfer of the full latent sequence challenging.</p>
<p>We therefore examine latent action sequences in the frequency domain. Our analysis finds stronger correspondence between human and robot interactions in their low-frequency components: the slowly varying temporal structure is more consistent across embodiments than the full trajectory.</p>
<p class="method-insight"><strong>Our goal:</strong> transfer the shared temporal structure while allowing the robot to generate fine-grained actions suited to its own embodiment.</p>
</div>
</section>
<section class="method-chapter" id="spectral-guidance" aria-labelledby="spectral-guidance-title">
<div class="method-explanation">
<p class="eyebrow">OUR SOLUTION</p>
<h3 id="spectral-guidance-title">Spectral latent action guidance</h3>
<p>WING transforms interaction-centric latent action sequences into the frequency domain and retains their lowest-frequency components as a structured guidance target for robot action generation.</p>
<div class="spectral-explorer" aria-label="Interactive spectral guidance illustration">
<div class="spectral-topline"><span>INTERACTION → SPECTRAL GUIDANCE</span><button type="button" class="spectral-play" aria-pressed="false">Play sequence</button></div>
<div class="spectral-stages" role="group" aria-label="Explore a stage">
<button type="button" class="spectral-stage" data-stage="0" aria-pressed="true"><span class="spectral-label">01 · TIME DOMAIN</span><strong>Latent action sequence</strong><svg viewBox="0 0 240 110" aria-hidden="true" class="spectral-signal"></svg><span>Interaction-centric motion from WING-LAM.</span></button>
<button type="button" class="spectral-stage" data-stage="1" aria-pressed="false"><span class="spectral-label">02 · FREQUENCY DOMAIN</span><strong>Temporal DCT</strong><svg viewBox="0 0 240 110" aria-hidden="true" class="spectral-spectrum"></svg><span>Resolve slow and rapid temporal variations.</span></button>
<button type="button" class="spectral-stage" data-stage="2" aria-pressed="false"><span class="spectral-label">03 · RETAIN LOW FREQUENCIES</span><strong>Spectral guidance</strong><svg viewBox="0 0 240 110" aria-hidden="true" class="spectral-retained"></svg><span>Keep the first K coefficients as guidance.</span></button>
</div>
<div class="spectral-detail"><div><h4 class="spectral-detail-title">Encode interactions over time</h4><p class="spectral-detail-copy">WING-LAM encodes successive frame pairs. The resulting sequence contains both slowly varying structure and rapid temporal changes.</p></div><label class="spectral-control" for="spectral-k">RETAINED COEFFICIENTS <output id="spectral-k-value" for="spectral-k">K = 4 / 16</output><input id="spectral-k" type="range" min="1" max="16" value="4"><span>Low → high frequency</span></label></div>
<p class="spectral-caption">Illustrative signal · Select a stage or adjust K to explore. At inference, WING predicts the retained coefficients to guide robot actions.</p>
</div>
<p><strong>1. Extract the shared temporal structure.</strong> We apply a discrete cosine transform (DCT) along the time axis of the latent sequence. Keeping the lowest K frequencies suppresses rapid variations and preserves shared human–robot dynamics.</p>
<p><strong>2. Predict guidance from the current context.</strong> The spectral target is constructed from future frame pairs during training. At inference time, a lightweight predictor estimates it from the current image, language instruction, and robot state, so future observations are not required.</p>
<p><strong>3. Guide robot action generation.</strong> The predicted frequency coefficients are projected into guidance tokens. The action model attends to these tokens through gated residual updates, using the shared spectral latent action to guide the generation of executable robot action chunks.</p>
<p class="method-insight">Low-frequency truncation is applied to the <strong>latent guidance</strong>. The robot's actions retain the fine-grained control needed to complete the task.</p>
</div>
</section>
</div>
</section>
<section class="section wrap experiments-section" id="experiments" aria-labelledby="experiments-title">
<p class="eyebrow">FROM REPRESENTATIONS TO ROBOT CONTROL</p>
<h2 id="experiments-title">What WING does</h2>
<p class="section-intro">Four questions about learning from human experience—and turning it into robot actions.</p>
<ol class="experiment-questions" aria-label="Four experimental questions">
<li><a href="#transfer-structure"><span>Q1</span>What structures in egocentric latent actions reveal transferable interaction knowledge?</a></li>
<li><a href="#robot-performance"><span>Q2</span>Can WING effectively transfer latent action knowledge to improve robot performance?</a></li>
<li><a href="#component-analysis"><span>Q3</span>How do interaction-centric latent actions and spectral latent action guidance contribute?</a></li>
<li><a href="#ego-pretraining"><span>Q4</span>How effective is WING’s pretraining strategy for downstream robot policy learning?</a></li>
</ol>
<section class="experiment-chapter" id="transfer-structure" aria-labelledby="q1-title">
<div class="experiment-copy"><p class="eyebrow">QUESTION 01 · TRANSFERABLE STRUCTURE</p><h3 id="q1-title">What is worth transferring?</h3>
<p>Useful latent actions should describe the interaction, rather than the observer’s movement. WING-LAM preserves action semantics while reducing camera-motion information and sensitivity to camera perturbations. Its frozen representations achieve the highest average action-classification accuracy in this comparison.</p></div>
<div class="experiment-copy"><p>Across human–robot pairs, low-frequency latent components separate matching interactions more clearly from mismatched ones. The shared slow structure provides a stronger transfer signal than the full trajectory or its high-frequency components.</p></div>
<div class="result-pair"><div class="result-chart diagnostic-chart" id="motion-decodability"></div><div class="result-chart diagnostic-chart" id="camera-sensitivity"></div><div class="result-chart" data-chart="semantics"></div><div class="result-chart" data-chart="similarity"></div></div>
</section>
<section class="experiment-chapter" id="robot-performance" aria-labelledby="q2-title">
<div class="experiment-copy"><p class="eyebrow">QUESTION 02 · ROBOT PERFORMANCE</p><h3 id="q2-title">Does human experience improve robot control?</h3>
<p>WING achieves the highest average success rate among the methods reported in the paper on LIBERO, RoboTwin 2.0, and RoboCasa–GR1. The tables below compare every method and evaluation setting reported in the main simulation results.</p></div>
<div class="simulation-tables" id="simulation-tables" aria-label="Simulation benchmark results"></div>
<div class="experiment-copy"><h4>From simulation to four real-world tasks</h4><p>On the dual-arm robot, WING reaches 75.0% average success in the standard setting. Under generalization, it remains competitive with π₀.₅ in average success and attains a slightly higher average progress score. Success measures completion; progress credits partial execution.</p></div>
<div class="real-world-panel" id="real-world-panel" aria-label="Real-world performance across four tasks"></div>
</section>
<section class="experiment-chapter" id="component-analysis" aria-labelledby="q3-title">
<div class="experiment-copy"><p class="eyebrow">QUESTION 03 · CORE COMPONENTS</p><h3 id="q3-title">Why do both components matter?</h3>
<p>Interaction-centric encoding and spectral guidance make complementary contributions. These controlled ablations exclude egocentric WAM pretraining. Without WING-LAM, we use DreamDojo’s latent encoder; without DCT, we guide the policy with the full time-domain latent trajectory.</p></div>
<div class="component-table-panel" id="component-table-panel"></div>
<div class="experiment-copy"><h4>Enough temporal structure, without the full spectrum</h4><p>Keeping four DCT coefficients performs best across all four LIBERO suites. Two coefficients lose useful dynamics; all eight add high-frequency variation without gains.</p></div>
<div class="bandwidth-overview" aria-label="DCT bandwidth ablation on four LIBERO suites">
<div class="bandwidth-grid">
<div class="result-chart" data-chart="bandwidth" data-view-index="0"></div>
<div class="result-chart" data-chart="bandwidth" data-view-index="1"></div>
<div class="result-chart" data-chart="bandwidth" data-view-index="2"></div>
<div class="result-chart" data-chart="bandwidth" data-view-index="3"></div>
</div>
<p class="bandwidth-caption">Success rate (%) · K = 8 retains the full spectrum. <strong>K = 4</strong> performs best in every suite. <a href="https://mikuz12.github.io/wing/main.pdf#page=9" target="_blank" rel="noopener">Figure 5(a) ↗</a></p>
</div>
</section>
<section class="experiment-chapter" id="ego-pretraining" aria-labelledby="q4-title">
<div class="experiment-copy"><p class="eyebrow">QUESTION 04 · EGO PRETRAINING</p><h3 id="q4-title">How should the model learn from egocentric video?</h3>
<p>All strategies use the same 200-hour egocentric pretraining budget. Predicting spectral guidance works better than video-only learning or treating latent actions as direct action targets. Removing interaction-centric debiasing also reduces performance in most settings.</p></div>
<div class="pretraining-comparison" id="pretraining-comparison" aria-label="Egocentric pretraining strategy comparison"></div>
<div class="experiment-copy"><p class="method-insight">Human latent actions are most useful as <strong>guidance for action generation</strong>, while the robot policy retains its own action-prediction objective.</p></div>
</section>
<section class="experiment-chapter" id="additional-experiments" aria-labelledby="additional-title">
<div class="experiment-copy"><p class="eyebrow">QUALITATIVE RESULTS</p><h3 id="additional-title">Additional experiments</h3><p>What does WING-LAM encode? Across different scenes, decoded motion concentrates on hands and their interactions with objects.</p></div>
<div class="interaction-gallery" aria-label="Interaction motion examples">
<div class="interaction-controls"><div class="interaction-stages" role="group" aria-label="Displayed frame"><button type="button" aria-pressed="true">01 · First frame</button><button type="button" aria-pressed="false">02 · Second frame</button><button type="button" aria-pressed="false">03 · Interaction motion</button></div><button class="interaction-play" type="button" aria-pressed="true">Pause sequence</button></div>
<div class="interaction-grid"><figure class="interaction-example"><div class="interaction-images"><img src="https://mikuz12.github.io/wing/assets/interaction-motion/example-1-frame-1.jpg" width="640" height="480" alt="Example 1: first frame"><img src="https://mikuz12.github.io/wing/assets/interaction-motion/example-1-frame-2.jpg" width="640" height="480" alt="Example 1: second frame" hidden><img src="https://mikuz12.github.io/wing/assets/interaction-motion/example-1-motion.jpg" width="640" height="480" alt="Example 1: decoded interaction motion with orange trajectories" hidden></div><figcaption>Example 01</figcaption></figure>
<figure class="interaction-example"><div class="interaction-images"><img src="https://mikuz12.github.io/wing/assets/interaction-motion/example-2-frame-1.jpg" width="640" height="480" alt="Example 2: first frame"><img src="https://mikuz12.github.io/wing/assets/interaction-motion/example-2-frame-2.jpg" width="640" height="480" alt="Example 2: second frame" hidden><img src="https://mikuz12.github.io/wing/assets/interaction-motion/example-2-motion.jpg" width="640" height="480" alt="Example 2: decoded interaction motion with orange trajectories" hidden></div><figcaption>Example 02</figcaption></figure>
<figure class="interaction-example"><div class="interaction-images"><img src="https://mikuz12.github.io/wing/assets/interaction-motion/example-3-frame-1.jpg" width="640" height="480" alt="Example 3: first frame"><img src="https://mikuz12.github.io/wing/assets/interaction-motion/example-3-frame-2.jpg" width="640" height="480" alt="Example 3: second frame" hidden><img src="https://mikuz12.github.io/wing/assets/interaction-motion/example-3-motion.jpg" width="640" height="480" alt="Example 3: decoded interaction motion with orange trajectories" hidden></div><figcaption>Example 03</figcaption></figure>
<figure class="interaction-example"><div class="interaction-images"><img src="https://mikuz12.github.io/wing/assets/interaction-motion/example-4-frame-1.jpg" width="640" height="480" alt="Example 4: first frame"><img src="https://mikuz12.github.io/wing/assets/interaction-motion/example-4-frame-2.jpg" width="640" height="480" alt="Example 4: second frame" hidden><img src="https://mikuz12.github.io/wing/assets/interaction-motion/example-4-motion.jpg" width="640" height="480" alt="Example 4: decoded interaction motion with orange trajectories" hidden></div><figcaption>Example 04</figcaption></figure>
</div>
<p class="interaction-caption">WING-LAM latent actions are passed to the teacher decoder to reconstruct motion trajectories. Trajectories from three consecutive clips are concatenated to reveal longer-term interaction motion. <a href="https://mikuz12.github.io/wing/assets/int_vis.pdf" target="_blank" rel="noopener">Figure 8 ↗</a></p>
</div>
</section>
<section class="experiment-chapter" id="real-generalization" aria-labelledby="generalization-title">
<div class="experiment-copy"><p class="eyebrow">BEYOND THE TRAINING SETTING</p><h3 id="generalization-title">Real-world generalization</h3></div>
<div class="generalization-gallery">
<div class="gen-toolbar"><span class="gen-label">SELECT TASK</span><div class="gen-tasks" role="group" aria-label="Generalization task"><button type="button" data-task="pack3" aria-pressed="true">Pack Objects</button><button type="button" data-task="stack" aria-pressed="false">Stack Cups</button><button type="button" data-task="insertion" aria-pressed="false">Battery Insertion</button><button type="button" data-task="assembly" aria-pressed="false">Battery Assembly</button></div></div>
<div class="gen-layout"><div class="gen-previews" role="group" aria-label="Generalization videos"></div><div class="gen-main"><video class="gen-video" controls muted playsinline loop preload="none" aria-label="Selected generalization video"></video><div class="gen-description" aria-live="polite"><h4 class="gen-title"></h4><p class="gen-type"></p></div><p class="gen-error" hidden>Video could not load. <a href="#">Open the video directly ↗</a></p></div></div>
</div>
</section>
</section>
<section class="section wrap abstract-section" id="abstract"><p class="eyebrow">THE RESEARCH</p><h2>Abstract</h2><div class="abstract-copy" id="abstract-copy"><p>Learning general-purpose robot policies requires large-scale real-world interaction data, yet collecting robot demonstrations through teleoperation remains expensive and difficult to scale. Egocentric videos provide a rich source of human interaction experience that shares task-relevant semantics with robotic manipulation, creating an opportunity to align human and robot actions in a shared latent action space for cross-embodiment knowledge transfer. However, existing latent action approaches typically infer actions through reconstruction between consecutive frames, which are not inherently interaction-centric and can be dominated by nuisance variations, such as ego-camera motion. Moreover, although human interactions and robot actions share interaction semantics, they often exhibit substantially different temporal dynamics, making direct transfer to robot policies challenging. To address these challenges, we propose WING (World Action Learning via INteraction-Centric Spectral Latent Guidance), a framework for transferring interaction knowledge from egocentric videos to robot policies. First, we introduce an interaction–motion decoupling mechanism that separates interaction-relevant signals from task-irrelevant motion and selectively distills the interaction-centric components into latent actions. Second, motivated by the observation that cross-embodiment task semantics are primarily encoded in slowly varying temporal structures, we identify shared components between egocentric latent actions and robot behaviors in the spectral domain and use them as guidance for action generation at inference time. With these designs, WING achieves average success rates of 99.20% on LIBERO, 93.80% on RoboTwin 2.0, and 57.7% on RoboCasa–GR1, while also demonstrating strong performance across four real-world tasks under diverse generalization settings. These results demonstrate that WING can effectively distill embodied interaction knowledge from large-scale egocentric videos and transfer it to robot control, providing a scalable pathway for acquiring physical interaction <span class="keep-phrase">knowledge from human experience.</span></p></div><a class="paper-link" href="https://arxiv.org/abs/2610.03607" target="_blank" rel="noopener">Read the full paper <span aria-hidden="true">↗</span></a></section>
<section class="section wrap abstract-section" id="bibtex">
<p class="eyebrow">CITE THIS WORK</p><div class="citation-heading"><h2>BibTeX</h2><button type="button" id="copy-citation">Copy</button></div>
<p class="citation-note">arXiv:2610.03607 · 2026</p>
<pre class="citation"><code id="citation-text">@misc{liu2026worldactionlearninginteractioncentric,
title={World Action Learning via Interaction-Centric Spectral Latent Guidance},
author={Zhiming Liu and Yikun Miao and Ying Chen and Hongrui Yin and Fangqi Zhu and Xiaoyi Pang and Quanxin Shou and Zhengyang Yan and Haodong Wang and Song Guo},
year={2026},
eprint={2610.03607},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2610.03607},
}</code></pre><p class="copy-status" id="copy-status" role="status"></p>
</section>
</div>
</div>
</main>
<dialog class="demo-dialog" aria-labelledby="demo-dialog-title">
<div class="demo-dialog-heading"><h2 id="demo-dialog-title"></h2><button class="demo-close" type="button" aria-label="Close video" autofocus>×</button></div>
<video id="expanded-demo" controls playsinline preload="none" aria-labelledby="demo-dialog-title"></video>
<p class="demo-playback-status" role="status"></p>
</dialog>
<footer class="wrap"><a class="wordmark" href="#home">WING.</a><p>World Action Learning via INteraction-Centric Spectral Latent Guidance</p><a href="https://arxiv.org/abs/2610.03607" target="_blank" rel="noopener">Paper ↗</a></footer>
</body>
</html>