- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces FigmaTrace, a dataset of approximately 200 hours of expert Figma workflows covering 126 tasks across eight categories and ten high-level skills. The authors convert recordings and OS-level interaction logs into 3,469 trajectories segmented into eleven design phases, then fine-tune four vision-language models and evaluate them on held-out Figma data and four external GUI benchmarks. Results show improvements in action prediction and grounding, benefits of phase-based segmentation, and a preference for fine-tuned outputs in blind expert evaluations. The key contributions are the dataset, its skill- and phase-based organization, and evidence that expert workflows can improve GUI interaction performance.
Critical strengths
-
FigmaTrace captures intermediate states and actions throughout professional design workflows, providing process-level supervision beyond completed design artifacts. Its 200 hours of demonstrations and structured skill taxonomy offer a potentially valuable resource for training design agents.
-
The evaluation spans four model families and multiple GUI benchmarks, showing improvements in both in-domain and external settings. The analysis also examines regressions and failure patterns, offering useful insights into the effects of workflow-based training.
Critical weaknesses
-
Insufficient evidence for creative design capabilities and preference alignment. Most evaluations focus on action accuracy and coordinate grounding, which do not directly measure aesthetic quality, design judgment, or end-to-end task success. The human study is limited to 30 paired outputs and two experts, with insufficient detail on task selection and score aggregation. Despite a 72% preference rate for fine-tuned outputs, the pass rate on key design criteria is only 5%, compared with 0% for the base model. Skewed skill coverage and regression on VideoGUI-undirected further limit the generalizability of the findings. Overall, the evidence supports improved GUI operation more strongly than enhanced creative design capabilities.
-
Insufficient ablations and methodological validation. Table 3 does not establish whether phase-based and length-based training are matched for instruction content, retained actions, token exposure, and optimization budget. The effects of action correction, generated thoughts, and coordinate-format adaptation are not isolated, and a matched baseline using generic GUI supervision is absent. Moreover, the segmentation advantage varies across benchmarks, while the data-splitting strategy, annotation validation, external evaluation sample sizes, and scoring procedures are insufficiently documented. Agreement on phase labels for 20 sampled shards does not validate the generated thoughts or corrected actions. These limitations hinder reproducibility and weaken the claim that the gains specifically result from semantic phase segmentation and expert design supervision.
Questions for authors
-
What is the exact unit of train–evaluation separation, and do related recordings, tasks, or source designs overlap across splits? How were thought annotations and corrected action labels generated and independently validated? Did their construction involve access to future frames or actions?
-
Were phase-based and length-based training matched for retained actions, instruction content, token exposure, and optimization budget? How do the experiments disentangle the effects of phase segmentation from instruction construction, action correction, generated thoughts, and generic GUI fine-tuning?
-
Could the authors specify the evaluation splits, sample sizes, input histories, scoring criteria, and coordinate-handling rules? How much of the reported gains remains after controlling for coordinate-format errors? Are uncertainty estimates provided that account for dependencies among steps within the same task?
-
Could the authors clarify the task-selection procedure, execution budgets, raw preference counts, and aggregation rules for the human study? What evidence demonstrates improvements in absolute design quality and end-to-end task success, beyond relative preferences and longer action sequences?
3 comments
Part 2 of Rebuttal to Reviewer 3m2t
-
We ensure that instructions, actions, token exposure, tools, compaction specifics and optimization budgets all remain consistent across these experiments.
-
In lines 297-309, we discuss in detail how we human validate the effectiveness of our action correction and also describe that the reason for doing this is video capturing tool inconsistencies (which are dominant across all capture toolings since SMEs do not have access to the strongest of machines that enable highest recording resolution and framerates). We assure the reviewer that based on the reported findings and our internal evaluations, the action corrections are accurate and high fidelity which is seen through our downstream results on the internal and external OOD sets.
-
Distilling reasoning chains from DeepSeek-v4-Flash and Qwen-Flash-Next gave inconsistent gains, as these models also struggled at the task. To avoid distillation noise, we compute loss only on tool calls and the final completion, letting the model reason freely which gives the best results. Reasoning lengths during evaluation ranged from 4,700 to 20,000 tokens, spent mainly on understanding context and images. The trained model, unlike the base model, naturally learns to plan several steps ahead upfront and then act with minimal reasoning, which makes it more confident as well.
-
The coordinates are kept consistent with the base model across the paper. We use the norm-1000 for all models that are consistent with the base model (for example: Qwen uses [x1, y1, x2, y2] but Gemma uses [y1, x1, y2, x2] and Muse uses its custom implementation of multi-cropping and bounding to norm-1000)
-
Data splitting is done on a recording basis to avoid contamination across eval and training sets.
-
Annotation validation: Action-frame alignment is checked deterministically over the full corpus (Section 3.1), and phase labels are validated against SMEs (88% agreement on 20 shards; 78% of boundaries match the action log vs. 31% for random placement). However, the impact of this is limited since no loss is computed on the thoughts and all segmentation experiments use the same corrected labels and thought chains.
-
The generated thoughts are not produced synthetically but are self-improved during training using selective loss calculation. This means that we never need off-policy traces for distillation and these do not need separate validation other than downstream performance.
For the weaknesses in the "Questions for Authors" that aren’t addressed above but are present in the questions, we address them below:
Q1.
Did their construction involve access to future frames or actions?
No they did not since these are the model’s own thoughts and not distilled from another model.
Q2. The harness remains the same across experiments and we use each benchmark’s exact setup and instructions to rollout. Hence, the results are already separated perfectly.
Q3. Since we use the complete sets of publicly available evaluation splits for all datasets listed in the paper, these results are consistent across the board and externally too. We use the percentage step accuracy for the results reported in the paper because it provides the most fine grained understanding of action-closeness to human actions. For the rest of the setup, we ensure that the harness is consistently setup with the benchmark. For uncertainty, we report 95% intervals below:
| Comparison | Reference | FigmaTrace SFT | Difference [95% CI] |
|---|---|---|---|
| Qwen3.6-35B-A3B: base to SFT | 32.0 | 58.3 | +26.3 [+23.9, +28.7] |
| Muse-Glimmer-30B: base to SFT | 52.0 | 64.0 | +12.0 [+9.8, +14.2] |
| Gemma-4-31B: base to SFT | 58.3 | 61.1 | +2.9 [+0.7, +4.9] |
| Qwen3.8-27B: base to SFT | 58.8 | 63.8 | +5.0 [+3.0, +7.0] |
| Qwen3.8-27B: length-based to phase-based | 56.3 | 63.8 | +7.5 [+5.2, +9.8] |
| Muse-Glimmer-30B: length-based to phase-based | 58.1 | 64.0 | +5.8 [+4.2, +7.8] |
Training used three seeds (42, 800, 9000) and picked best result. Hyperparameters in Table 6 are selected based on a validation split held out from the FigmaTrace training split.
Q4. For the creation of FigmaTrace, we ensure that each task is produced in hindsight with the targeted task taxonomy in Table 1 which in turn maps specifically to our skills taxonomy. Since this task taxonomy is SME advised and we record SMEs solving the task to end to end, we have guarantees of solvability for each task. FigmaTrace does not include tasks that are unsolvable since these are derived from real world, scoped tasks that experts perform on a daily basis.
Follow-up on the rebuttal
As the deadline for the rebuttal phase approaches, we would be happy to address any additional concerns about the paper. We look forward to your engagement and thank you for your time.
Part 1 of Rebuttal to Reviewer 3m2t
We thank the reviewer for their feedback on our work:
CANVAS:
DesignBench
Generation split (430 samples: write the page's code from a screenshot)
Edit split (359 samples: apply an instruction to existing code and its screenshot)
Repair (111 samples: find and fix bugs from buggy code plus a screenshot)
Since DesignBench doesn’t utilize tools and the CANVAS figma set utilizes MCP tools for creation, deletion, layout adjustment, styling, and content modification (and not playwright tools), we more strongly show that the performance does not depend on tools or a certain visual structure but instead training on FigmaTrace teaches models to real front end and design skills involving generating, editing and repairing broken components.
Raw counts for Qwen3.8-27B. SMEs prefer the SFT output in 43 of 60 ratings (72%), the base output in 15 (25%), with 2 ties (3%). The two SMEs agree on 25 of 30 pairs (83.3%).
The skill distribution best represents real world SME skill distributions when solving tasks and is not a limitation. To make models more human-like in terms of design preferences means that they should closely follow the phases that humans show when solving tasks. Regarding VideoGUI-undirected performance, this is not universal and Qwen3.6-35B-A3B and Muse-Glimmer-30B show improvement (10.1 to 15.9 and 28.7 to 32.8) which shows that these base models are better at following undirected instructions (for example, this is studied in detail in the Muse report through the custom harnesses like GAIA-2 where an event daemon adds new things to the environment while the agent continues to work which makes it better at undirected search and navigation of containers).