章节总览
参考资料与复现实验 / 42 · 参考资料
原目录

InstructPart 动作监督实验的问题、协议、结果和仍待复核的证据限制。

InstructPart action-supervision experiment

Status and conclusion

This quantitative experiment is included in the shared experiment registry. Its pseudo ground truth still has review_status=pending, so results remain provisional until the generated review panels have been inspected and approved.

At layer 18, MolmoAct2 improved depth-direction accuracy over Molmo2-ER from 63.07% to 69.02% (+5.96 percentage points; image-clustered 95% bootstrap CI +0.89 to +10.93). MolmoAct2-Pretrain reached 67.11% (+4.04 points; CI -1.07 to +9.16). This supports a depth-specific MolmoAct2 advantage in this setting; it does not support a general overall-accuracy advantage, and layer 18 belongs to a diagnostic test sweep rather than an independently selected layer.

Research question

For an image containing an object and an annotated object part, does the S-Space displacement between the part token and whole-object token point in the same H/V/D direction as the annotated part relative to the whole object? The comparison follows the training progression from Molmo2-ER through MolmoAct2-Pretrain to MolmoAct2.

This is an explicit named-part localization test. The prompt tells the model the part name. It does not test whether a model can infer an unnamed affordance region.

Data and pseudo ground truth

The source is IffYuan/InstructPart at revision bcd06969e32582ceeba3e1841183106027b379fb. The deterministic selection first takes 360 rows by round-robin sampling over (object, part, affordance) groups with seed 42. Grounding DINO Tiny detects the whole object and retains rows only when the detected box covers at least 80% of the supplied part mask. The historical filter retains 321 candidates. SlimSAM receives the detected box and the first 300 valid rows become the experiment set.

For a binary mask M, its 2-D coordinate is the arithmetic mean of its pixel coordinates. The whole-object coordinate uses the SlimSAM object mask, which is unioned with the supplied part mask. Horizontal and vertical ground truth are:

dx = (part_centroid_x - object_centroid_x) / object_mask_width
dy = (part_centroid_y - object_centroid_y) / object_mask_height

Depth Anything V2 Small predicts relative inverse depth for the original image and a horizontal flip. The flipped prediction is restored and the two maps are averaged. The whole-object depth reference excludes the annotated part:

body_mask = object_mask AND NOT part_mask
dz = (mean_depth(part_mask) - mean_depth(body_mask)) / (q95(depth) - q05(depth))

Larger inverse depth means closer. A direction is ambiguous when abs(dx), abs(dy), or abs(dz) is below 0.05. Depth is also ambiguous when the original and flipped predictions disagree in sign. The resulting valid counts are 178 horizontal, 238 vertical, and 225 depth images.

Model input and projection

Every image is forwarded twice per template: once with the whole-object name as the target and once with the dataset-provided part name. The action and object fields come from InstructPart. For the five action-context templates, code renders those fields into a fixed grammatical action phrase. The measured state is the last subtoken overlapping the exact target character span.

ID Group Exact template
T01 context-free Point to {target}.
T02 context-free Find the {target}.
T03 context-free Locate the {target}.
T04 context-free Look for {target} in the image and show me where they are.
T05 context-free Please find {target} and show me where they are.
T06 action context Given the instruction "{instruction}", locate the {target}.
T07 action context The robot needs to {instruction}. Find the {target}.
T08 action context The robot has been asked to {instruction}. Point to the {target}.
T09 action context The robot's task is to {instruction}. Help me find the {target}.
T10 action context To carry out the instruction "{instruction}", find the {target}.

Example for action open, object bottle, and part cap under T06:

Given the instruction "open the bottle", locate the bottle.
Given the instruction "open the bottle", locate the cap.

At every exported post-block layer, the experiment computes:

object_coordinate = axes @ object_target_state
part_coordinate   = axes @ part_target_state
margin            = part_coordinate - object_coordinate

The fixed positive directions are right, below, and close. A prediction is correct when the margin and GT have the same strict sign. An exact zero margin is incorrect.

Evaluation protocol

The report covers layers 17--22. Each of the ten templates receives equal weight. Overall accuracy pools every valid template/image/axis decision. For each model and layer, across_prompt_summary reports the mean accuracy and 95% Student-t interval across the ten prompts: mean +/- t(0.975, 9) * std(ddof=1) / sqrt(10). Values use accuracy units (0--1); intervals are not clipped. The depth entries at layer 18 reproduce the historical blog means and error bars from the saved per-template scores. This formula matches the published bars numerically; the original plotting script was not recovered.

Paired comparisons keep all ten templates for an image together and bootstrap images 10,000 times with seed 42. This avoids treating ten prompt variants of one image as ten independent samples. These intervals describe model differences, not the blog's per-model error bars.

Historical layer-18 results:

Model Horizontal Vertical Depth Overall
Molmo2-ER 82.75% 88.57% 63.07% 78.00%
MolmoAct2-Pretrain 78.71% 86.89% 67.11% 77.68%
MolmoAct2 78.26% 84.92% 69.02% 77.49%

MolmoAct2 depth accuracy is higher than Molmo2-ER for every individual layer-18 template (67.56%--70.67% versus 60.89%--65.78%). The aggregate horizontal and vertical scores are lower, which is why the overall score does not improve.

Reproduction

From the repository root, install the standard model environments, prepare the three pseudo-GT models, and run the experiment:

./scripts/prepare_models.sh action_supervision
./scripts/run_experiment.sh --experiment action_supervision

Review panels are written below .cache/assets/datasets/instructpart_affordance_parts/pseudo_ground_truth/review_panels/. Do not change review_status to passed before completing that review.

For a bounded chain that cannot overwrite the full output:

./scripts/run_experiment.sh --workflow action_supervision --limit 10

The full result is outputs/action_supervision/instructpart_prompt_ensemble/results.json. The bounded result is written under outputs/small/action_supervision/instructpart_prompt_ensemble/limit_10/.

The single source of truth is configs/experiments/action_supervision/instructpart_prompt_ensemble.json. Source images, masks, model weights, generated masks, depth maps, and run outputs remain outside Git.

Historical regression identity

The migration is bound to the following historical files:

Artifact SHA-256
360-row source selection cb6d668645f7e5bbf40bb34f0ee00af95940141c89fbfee6e9dacf9c13d08c71
321-row filtered selection 0a81e740ae35caee96d8f5ee7ca3eba34495a8f93fdc2f71c569bbe7c4e579ac
Grounding DINO boxes 0741621b7de2eaf10a639ab1198cb66baaa1eaf8d4d6623387c3389102c638f2
300-row retained selection 75c592e590d5b37f55d0e38915b851e4d774f0b28f9157f5e72769cd10ded174
Portable pseudo-GT values c0168d376730e167ac629dbddc686adc3f09e3c9f15db7ac94b98eba517eb82e
Historical full score report 0e0e7094a2fd5de6e6ca392bc0a648f87e68732b607daea540b0ead446bc8757

The portable pseudo-GT digest excludes only depth_map paths and review status. Paths are now relative, and a later human approval may legitimately change review status without changing numerical GT.