Refine Actions If Needed

Adaptive Action Chunk Refinement
for Robot Manipulation

Robot manipulation does not require the same amount of action refinement at every decision. Direct policies produce executable actions quickly, while iterative generative policies spend more computation to correct them. A fixed refinement budget can provide too little correction in one state and more than is needed in another.

Selected real-robot rollouts · 3× source playback · silent loops

We propose Refine Actions If Needed (RAIN), which learns whether an action needs refinement and how many steps to use. ACT first proposes an action chunk. RAIN predicts how accurate the action would be after different numbers of refinement steps, then weighs the improvement against computation cost. It runs only the chosen steps; choosing zero executes ACT’s proposal directly.

RAIN achieves 66.6% success across eight simulation tasks and 72.5% across four real-robot tasks. Compared with three fixed refinements, it uses 26.6% fewer flow-matching evaluations and 15.7% less policy compute, with a similar success rate.

From Prediction to Selective Refinement

RAIN starts with an action that the robot can already execute. Before running the refinement model, it uses the observation, proposed action and prediction uncertainty to choose how much correction to apply.

From direct prediction to optional correction. RAIN preserves the executable proposal and invokes flow matching only when selected.

Robot manipulation

Real-Robot Results

Across four real-robot tasks, RAIN achieves 72.5% success, compared with 55.0% for ACT and 61.25% for DP. Experiments use an ARX arm with two RGB views and proprioception.

Overall success rate

ACT55.0%
DP61.25%
RAIN72.5%

Completion time · successful trials

ACT17.17 s
DP21.49 s
RAIN17.57 s

50 demonstrations and 20 evaluation trials per method per task.

RAIN · selected successful trial

Success rate

View complete real-robot results
Success rate (%) by task and average completion time (s).
MethodStackRingSortPourOverallTime (s)

Completion time includes observation processing, RTX 4080 inference and physical execution. We average successful trials within each task, then average across the four tasks.

ManiSkill evaluation

Simulation Results

We evaluate eight ManiSkill tasks with 100 trials per task. RAIN achieves a 66.6% average success rate with 0.322 ms of policy compute per executed action.

Success rate versus policy compute time.

0.322 ms / action

Average policy compute per executed action.

5.71× lower policy latency than DP-5, with 66.6% versus 67.9% success.

NVIDIA A800 · FP32 · CUDA Graph.

View all simulation results and settings

Each task has 500 demonstrations. RAIN uses 400 for training, 50 for calibration and 50 held out; baselines train on all 500. All methods use the same 100 evaluation seeds per task. Timing measures policy computation on an A800, excluding environment execution and input transfer, and averages it over eight executed actions.

Success rate (%) and policy compute time (ms/action), 800 trials per method.
MethodPickPushStackPegPlugPullPushTPyramidOverallms/action

Method suffixes give the number of model evaluations. ProbeFlow uses a frozen FM model, Gaussian initialization, probe time 0.5, threshold 0.008 and 2/4/6/8/10-step schedules; its timing includes both probes.

Method

ACT proposes an action chunk. RAIN estimates how much refinement will improve it, chooses the number of steps, and runs those steps before executing the action.

  1. 01

    Propose an action

    ACT predicts an action chunk from the observation history, together with an estimate of its uncertainty.

  2. 02

    Choose how much to refine

    Using the observation and proposal, RAIN predicts the action error after different numbers of refinement steps and weighs it against their compute cost.

  3. 03

    Refine and execute

    Run the chosen number of refinement steps and execute the action chunk. With zero steps, use ACT’s proposal directly.

Training compares actions at different refinement levels with expert demonstrations. At deployment, RAIN predicts these errors and runs the chosen number of steps.
How is the refinement choice learned?

We jointly train ACT and the refinement model, then freeze them. Actions produced with different numbers of refinement steps are compared with expert demonstrations. A group of predictors learns the resulting action errors and uncertainty. Calibration sets the balance between accuracy, uncertainty and computation, while recent refinement choices help keep decisions consistent over time.

Simulation uses a 400/50/50 training, calibration and held-out split per task. On the real robot, fitting and calibration use the same 50 demonstrations per task.

Refinement During Manipulation

Watch how RAIN changes the number of refinement steps as the robot completes each task. Switch tasks to compare different manipulation examples.

Refinement steps

—steps for this
action chunk

0 · Direct123

Click a segment below to jump to that part of the video.

The number of refinement steps varies across tasks.

Ablation Results

The following controls share proposal and refiner weights, evaluation seeds, preprocessing and eight-action execution blocks.

26.6%fewer refinement-model evaluations

15.7%lower policy latency

Compared with always using three refinement steps, RAIN saves computation with similar success rates: 66.6% for RAIN and 66.4% for Fixed 3.
Matched routing variants; simulation timing follows the isolated A800 protocol.
VariantSuccess (%)ms/actionFM evals/action
Refinement beyond flow matching

A separate ACT–diffusion fixed-refinement experiment improves success from 52.9% for the unrefined proposal to 63.6% at its best evaluated fixed setting. This supports the proposal-refinement principle; the dynamic routing experiments shown here use the flow-matching refiner.

Video Overview

Method, manipulation tasks and refinement allocation.

English narration · no background music · playback speeds are indicated within the video.

Research Resources

Paper, implementation and pretrained models.

Code and pretrained models are available under MIT. Read the arXiv preprint and find its BibTeX citation below.

Questions & Answers

How does RAIN choose the number of refinement steps?

RAIN predicts the action error and uncertainty for each possible number of steps. It combines these estimates with compute cost and recent refinement choices, then runs the selected number of steps.

What happens when RAIN chooses zero steps?

The robot executes ACT’s proposed action chunk directly.

How does the refinement amount vary across tasks?

Some tasks often use direct actions or one refinement step; others use more correction. For example, PushT and StackPyramid use three steps for 99.6% and 97.4% of decisions. The task-level chart above shows the full distribution.

Citation

arXiv:2610.19765 [cs.RO] · 2026

BibTeXAvailable
@misc{liu2026refineactionsneeded,
  title={Refine Actions If Needed: Adaptive Action Chunk Refinement for Robot Manipulation},
  author={Hanlin Liu and Zichen Zhou and Jiacheng Zhang and Liuyuan Na and Jin Jin and Guangyan Chen and Yufeng Yue and Gen Li and Fangqiang Ding},
  year={2026},
  eprint={2610.19765},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2610.19765}
}