acceptodds
Under review as a conference paper at ICLR 2027

FigmaTrace: Capturing Creative Nuances in Human Figma Design Workflows

Abstract

Vision Language Models have recently shown improvements in several objective and verifiable domains such as object detection but continue to underperform on subjective and creative design tasks. A major contributor to this performance gap is the lack of high quality human workflow data that captures a diverse set of preferences and decisions that make human experts good at design tasks. In this work, we first define a unique, expert curated taxonomy of design skills and best practices which we further expand into a set of 126 open ended, subjective, long horizon tasks. Built on top of this and expert solutions, our dataset, FigmaTrace, contains over 200 hours of human captured video data converted into 3469 design trajectories using a novel design phase-based method. We use our dataset to train four models and show that training on FigmaTrace leads to a performance improvement comparable to frontier closed models such as Claude-Opus-5 and GPT-5.6-Sol on four out of distribution agentic GUI environments. We further perform a useful ablation to attribute these performance improvements to a design phase-based video to trajectory conversion which outperforms prior length-based conversion approaches. Finally, we perform a qualitative analysis on the overall best performing Qwen3.8-27B outputs to better correlate performance improvements to FigmaTrace's trends. We open source our dataset and the best Qwen3.8-27B model for the community.

Primary area:
datasets and benchmarks
Keywords:
computer use agents, GUI agents, vision language models, design workflows, figma

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3m2tOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FigmaTrace, a dataset of approximately 200 hours of expert Figma workflows covering 126 tasks across eight categories and ten high-level skills. The authors convert recordings and OS-level interaction logs into 3,469 trajectories segmented into eleven design phases, then fine-tune four vision-language models and evaluate them on held-out Figma data and four external GUI benchmarks. Results show improvements in action prediction and grounding, benefits of phase-based segmentation, and a preference for fine-tuned outputs in blind expert evaluations. The key contributions are the dataset, its skill- and phase-based organization, and evidence that expert workflows can improve GUI interaction performance.

Critical strengths
  1. FigmaTrace captures intermediate states and actions throughout professional design workflows, providing process-level supervision beyond completed design artifacts. Its 200 hours of demonstrations and structured skill taxonomy offer a potentially valuable resource for training design agents.

  2. The evaluation spans four model families and multiple GUI benchmarks, showing improvements in both in-domain and external settings. The analysis also examines regressions and failure patterns, offering useful insights into the effects of workflow-based training.

Critical weaknesses
  1. Insufficient evidence for creative design capabilities and preference alignment. Most evaluations focus on action accuracy and coordinate grounding, which do not directly measure aesthetic quality, design judgment, or end-to-end task success. The human study is limited to 30 paired outputs and two experts, with insufficient detail on task selection and score aggregation. Despite a 72% preference rate for fine-tuned outputs, the pass rate on key design criteria is only 5%, compared with 0% for the base model. Skewed skill coverage and regression on VideoGUI-undirected further limit the generalizability of the findings. Overall, the evidence supports improved GUI operation more strongly than enhanced creative design capabilities.

  2. Insufficient ablations and methodological validation. Table 3 does not establish whether phase-based and length-based training are matched for instruction content, retained actions, token exposure, and optimization budget. The effects of action correction, generated thoughts, and coordinate-format adaptation are not isolated, and a matched baseline using generic GUI supervision is absent. Moreover, the segmentation advantage varies across benchmarks, while the data-splitting strategy, annotation validation, external evaluation sample sizes, and scoring procedures are insufficiently documented. Agreement on phase labels for 20 sampled shards does not validate the generated thoughts or corrected actions. These limitations hinder reproducibility and weaken the claim that the gains specifically result from semantic phase segmentation and expert design supervision.

Questions for authors
  1. What is the exact unit of train–evaluation separation, and do related recordings, tasks, or source designs overlap across splits? How were thought annotations and corrected action labels generated and independently validated? Did their construction involve access to future frames or actions?

  2. Were phase-based and length-based training matched for retained actions, instruction content, token exposure, and optimization budget? How do the experiments disentangle the effects of phase segmentation from instruction construction, action correction, generated thoughts, and generic GUI fine-tuning?

  3. Could the authors specify the evaluation splits, sample sizes, input histories, scoring criteria, and coordinate-handling rules? How much of the reported gains remains after controlling for coordinate-format errors? Are uncertainty estimates provided that account for dependencies among steps within the same task?

  4. Could the authors clarify the task-selection procedure, execution budgets, raw preference counts, and aggregation rules for the human study? What evidence demonstrates improvements in absolute design quality and end-to-end task success, beyond relative preferences and longer action sequences?

3 comments
AuthorsOpenReview ↗

Part 1 of Rebuttal to Reviewer 3m2t

We thank the reviewer for their feedback on our work:

  • To clearly separate the design preference learning from interface learning, we use two new datasets: CANVAS[1] (for its non-interface Figma design tests) and DesignBench (to test frontend design abilities and their transfer from Figma without tools). Each evaluation below uses the dataset specific metrics and temp=0 for minimal variance:

CANVAS:

ModelSSIM (higher better)LPIPS (lower better)Position similarityComponent similarity
Qwen3.8-27B Base0.8450.1720.9150.774
Qwen3.8-27B SFT0.8850.1310.9450.794
Gemma-4-31B-it0.7770.3830.6450.674
Gemma-4-31B-it SFT0.8010.1990.7550.701
Muse-Glimmer-30B0.7310.4510.6350.639
Muse-Glimmer-30B SFT0.8100.1820.8030.698

DesignBench

Generation split (430 samples: write the page's code from a screenshot)

ModelCLIP (higher better)SSIM (higher better)Compile success rate
Qwen3.8-27B Base0.7810.67788.4%
Qwen3.8-27B SFT0.8020.71790.6%
Muse-Glimmer-30B0.8050.67992.3%
Muse-Glimmer-30B SFT0.8240.68992.1%
Gemma-4-31B0.7310.64086.3%
Gemma-4-31B SFT0.7210.65988.1%

Edit split (359 samples: apply an instruction to existing code and its screenshot)

ModelJudge score (0–10)CMLS (code match)CMCS (code match)Compile success rate
Qwen3.8-27B Base8.890.6200.53798.1%
Qwen3.8-27B SFT9.100.6400.54899.1%
Muse-Glimmer-30B8.840.5930.50598.3%
Muse-Glimmer-30B SFT8.910.6040.55198.0%
Gemma-4-31B8.780.5920.50696.4%
Gemma-4-31B SFT8.890.6010.52197.0%

Repair (111 samples: find and fix bugs from buggy code plus a screenshot)

ModelJudge score (0–10)CMLS (code match)CMCS (code match)Issue accuracy
Qwen3.8-27B Base8.210.6160.5440.552
Qwen3.8-27B SFT8.350.6190.5560.581
Muse-Glimmer-30B8.050.5490.4650.438
Muse-Glimmer-30B SFT8.40.5770.5100.511
Gemma-4-31B8.050.5120.4430.573
Gemma-4-31B SFT8.240.5350.5100.610

Since DesignBench doesn’t utilize tools and the CANVAS figma set utilizes MCP tools for creation, deletion, layout adjustment, styling, and content modification (and not playwright tools), we more strongly show that the performance does not depend on tools or a certain visual structure but instead training on FigmaTrace teaches models to real front end and design skills involving generating, editing and repairing broken components.

  • The study uses 30 tasks, each completed end to end by the base and the fine-tuned model in the same sandboxed Playwright harness, with a budget of 1000 max steps per task. These tasks are separate from the training or evaluation sets. To reduce grading subjectivity, 20 tasks come from verifiable categories (8 replication, 5 flaw repair, 4 edge-content, 3 a11y remediation) and 10 from non-verifiable ones (3 responsive adaptation, 3 theming, 2 sketch-to-Figma, 2 prototype wiring). Two SMEs independently score each final file against the readiness checklist (Table 5) and give a blind pairwise preference. This gives 60 ratings per model, and we report all percentages over ratings without further aggregation.

Raw counts for Qwen3.8-27B. SMEs prefer the SFT output in 43 of 60 ratings (72%), the base output in 15 (25%), with 2 ties (3%). The two SMEs agree on 25 of 30 pairs (83.3%).

Skewed skill coverage and regression on VideoGUI-undirected further limit the generalizability of the findings

The skill distribution best represents real world SME skill distributions when solving tasks and is not a limitation. To make models more human-like in terms of design preferences means that they should closely follow the phases that humans show when solving tasks. Regarding VideoGUI-undirected performance, this is not universal and Qwen3.6-35B-A3B and Muse-Glimmer-30B show improvement (10.1 to 15.9 and 28.7 to 32.8) which shows that these base models are better at following undirected instructions (for example, this is studied in detail in the Muse report through the custom harnesses like GAIA-2 where an event daemon adds new things to the environment while the agent continues to work which makes it better at undirected search and navigation of containers).

AuthorsOpenReview ↗

Part 2 of Rebuttal to Reviewer 3m2t

  • We ensure that instructions, actions, token exposure, tools, compaction specifics and optimization budgets all remain consistent across these experiments.

  • In lines 297-309, we discuss in detail how we human validate the effectiveness of our action correction and also describe that the reason for doing this is video capturing tool inconsistencies (which are dominant across all capture toolings since SMEs do not have access to the strongest of machines that enable highest recording resolution and framerates). We assure the reviewer that based on the reported findings and our internal evaluations, the action corrections are accurate and high fidelity which is seen through our downstream results on the internal and external OOD sets.

  • Distilling reasoning chains from DeepSeek-v4-Flash and Qwen-Flash-Next gave inconsistent gains, as these models also struggled at the task. To avoid distillation noise, we compute loss only on tool calls and the final completion, letting the model reason freely which gives the best results. Reasoning lengths during evaluation ranged from 4,700 to 20,000 tokens, spent mainly on understanding context and images. The trained model, unlike the base model, naturally learns to plan several steps ahead upfront and then act with minimal reasoning, which makes it more confident as well.

  • The coordinates are kept consistent with the base model across the paper. We use the norm-1000 for all models that are consistent with the base model (for example: Qwen uses [x1, y1, x2, y2] but Gemma uses [y1, x1, y2, x2] and Muse uses its custom implementation of multi-cropping and bounding to norm-1000)

  • Data splitting is done on a recording basis to avoid contamination across eval and training sets.

  • Annotation validation: Action-frame alignment is checked deterministically over the full corpus (Section 3.1), and phase labels are validated against SMEs (88% agreement on 20 shards; 78% of boundaries match the action log vs. 31% for random placement). However, the impact of this is limited since no loss is computed on the thoughts and all segmentation experiments use the same corrected labels and thought chains.

  • The generated thoughts are not produced synthetically but are self-improved during training using selective loss calculation. This means that we never need off-policy traces for distillation and these do not need separate validation other than downstream performance.

For the weaknesses in the "Questions for Authors" that aren’t addressed above but are present in the questions, we address them below:

Q1.

Did their construction involve access to future frames or actions?

No they did not since these are the model’s own thoughts and not distilled from another model.

Q2. The harness remains the same across experiments and we use each benchmark’s exact setup and instructions to rollout. Hence, the results are already separated perfectly.

Q3. Since we use the complete sets of publicly available evaluation splits for all datasets listed in the paper, these results are consistent across the board and externally too. We use the percentage step accuracy for the results reported in the paper because it provides the most fine grained understanding of action-closeness to human actions. For the rest of the setup, we ensure that the harness is consistently setup with the benchmark. For uncertainty, we report 95% intervals below:

ComparisonReferenceFigmaTrace SFTDifference [95% CI]
Qwen3.6-35B-A3B: base to SFT32.058.3+26.3 [+23.9, +28.7]
Muse-Glimmer-30B: base to SFT52.064.0+12.0 [+9.8, +14.2]
Gemma-4-31B: base to SFT58.361.1+2.9 [+0.7, +4.9]
Qwen3.8-27B: base to SFT58.863.8+5.0 [+3.0, +7.0]
Qwen3.8-27B: length-based to phase-based56.363.8+7.5 [+5.2, +9.8]
Muse-Glimmer-30B: length-based to phase-based58.164.0+5.8 [+4.2, +7.8]

Training used three seeds (42, 800, 9000) and picked best result. Hyperparameters in Table 6 are selected based on a validation split held out from the FigmaTrace training split.

Q4. For the creation of FigmaTrace, we ensure that each task is produced in hindsight with the targeted task taxonomy in Table 1 which in turn maps specifically to our skills taxonomy. Since this task taxonomy is SME advised and we record SMEs solving the task to end to end, we have guarantees of solvability for each task. FigmaTrace does not include tasks that are unsolvable since these are derived from real world, scoped tasks that experts perform on a daily basis.

AuthorsOpenReview ↗

Follow-up on the rebuttal

As the deadline for the rebuttal phase approaches, we would be happy to address any additional concerns about the paper. We look forward to your engagement and thank you for your time.

Reviewer_hrifOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces FigmaTrace, a dataset of about 200 hours of screen recordings of Figma experts. It covers 126 tasks in 8 categories and is based on an expert-made taxonomy of 10 design skills. Raw OS events are mapped to Playwright-MCP actions and filtered with pixel-change rules. Gemini-3.6-Flash then splits them into 11 intent-based phases, giving 3,469 trajectories (2,883 train / 586 eval).

Critical strengths

1.Expert process data from professional creative software is rare. Recordings up to 4.5 hours long that keep intermediate states would be a useful resource if released. 2.The preprocessing pipeline is described concretely and can be reused: two-pass frame extraction, screen-settle detection, change-fraction thresholds, and consensus boundaries across 3/6/12 shardings. 3.Phase segmentation is checked beyond LLM-vs-human agreement. 78% of video boundaries fall within ±1 s of an action-log event, compared with 31% for random placement. 4.RQ4 openly reports regressions the fine-tuning introduced: repeated clicks and a bias toward the screen center.

Critical weaknesses

W1. The evaluation doesn’t isolate design ability, and the OOD gains may mostly come from aligning output format.

RQ1 is framed around design tasks, but all four OOD benchmarks are mobile or web navigation. RQ4 credits the gains to correct element selection, to the base model outputting raw pixels instead of norm-1000 coordinates, and to the base model giving no coordinates at all. Some base scores are very low. Qwen3.6-35B-A3B scores 36.8 on AndroidControl and 83.2 after SFT, which is where the “up to 46%” claim comes from. This pattern is what you would expect from any SFT that fixes the action format under the authors’ prompt. Two things are needed before the gains can be credited to FigmaTrace’s design content: (a) baselines evaluated with each model’s native prompt and coordinate convention, and (b) an SFT control at matched scale on generic GUI trajectories, or on FigmaTrace with phase structure removed.

W2. Several claims go beyond what the results show.

“Comparable to frontier closed models”: in Fig. 5, the closed models score higher on 4 of the 6 settings (Mind2Web vp800: 100.0 vs. 70.7; VideoGUI-undirected: 40.7 vs. 32.8). “Consistent improvements”: with phase SFT, Qwen3.8-27B drops on VideoGUI-undirected (26.0 → 19.3), and the Mind2Web gains are only 1–3 points. Claude-Opus-5 scoring 100.0 on Mind2Web vp800 also raises questions about the size of the evaluation subsets and whether they are saturated.

W3. The statistics are weak.

The OOD evaluation sets look small. RQ4 analyzes 150 GUI-Odyssey items, and many of the Mind2Web and VideoGUI scores are multiples of 1/150. How the subsets were sampled isn’t described. Table 3 and Fig. 5 are single runs with no confidence intervals. With n≈150, a 95% CI is roughly ±7 points, which is larger than most of the reported differences, including several in the ablation. Table 6 lists several training arms per model and calls the hyperparameters the “best”. It isn’t said which data was used to pick them.

Questions for authors

1.If the base models are evaluated with their native prompts and coordinate formats, or their outputs are rescored after converting coordinate formats, how much of the OOD gain remains? 2.Can you report end-to-end task success in Figma? 3.What are the size and sampling method of each OOD evaluation set? Please add seeds and CIs, and explain how hyperparameters were selected.

4 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer hrif

We thank the reviewer for the detailed comments.

W1.

All four OOD benchmarks are mobile or web navigation.

We add a transfer study on two design datasets: DesignBench [1] (front-end design) and CANVAS [2] (a limited set of skills we expect to transfer from FigmaTrace). We use each dataset's metrics and default prompts. All runs use temperature=0 to minimize variance. Cells show Base→SFT.

CANVAS

ModelSSIM↑LPIPS↓Position sim.↑Component sim.↑
Qwen3.8-27B0.845→0.8850.172→0.1310.915→0.9450.774→0.794
Gemma-4-31B-it0.777→0.8010.383→0.1990.645→0.7550.674→0.701
Muse-Glimmer-30B0.731→0.8100.451→0.1820.635→0.8030.639→0.698

DesignBench (CMLS/CMCS: code match)

Generation (430 samples: write the page's code from a screenshot)

ModelCLIP↑SSIM↑Compile success
Qwen3.8-27B0.781→0.8020.677→0.71788.4→90.6%
Muse-Glimmer-30B0.805→0.8240.679→0.68992.3→92.1%
Gemma-4-31B0.731→0.7210.640→0.65986.3→88.1%

Edit (359 samples: apply an instruction to existing code and its screenshot)

ModelJudge (0–10)CMLSCMCSCompile success
Qwen3.8-27B8.89→9.100.620→0.6400.537→0.54898.1→99.1%
Muse-Glimmer-30B8.84→8.910.593→0.6040.505→0.55198.3→98.0%
Gemma-4-31B8.78→8.890.592→0.6010.506→0.52196.4→97.0%

Repair (111 samples: find and fix bugs from buggy code plus a screenshot)

ModelJudge (0–10)CMLSCMCSIssue acc.
Qwen3.8-27B8.21→8.350.616→0.6190.544→0.5560.552→0.581
Muse-Glimmer-30B8.05→8.40.549→0.5770.465→0.5100.438→0.511
Gemma-4-31B8.05→8.240.512→0.5350.443→0.5100.573→0.610

DesignBench uses no tools. CANVAS (Figma) uses MCP tools for creation, deletion, layout adjustment, styling and content modification instead of Playwright. Therefore, these gains do not depend on tools or on a particular visual structure and training on FigmaTrace teaches real front-end and design skills including generating, editing and repairing broken components.

Any SFT that fixes the action format..

To show that this is not the case, we first create action-log shards according to the SME action log (tool switch, selection change or idle gap as mentioned in the paper) and run three SFT runs with consistent hyperparameters and training budget, and differ only in how recordings are sharded:

Qwen3.8-27B step acc. (%)BaseLengthAction-logPhase (ours)
GUI-Odyssey44.040.045.153.7
AndroidControl82.767.380.999.1
Mind2Web vp80069.369.370.070.7
Mind2Web vp100065.370.070.268.7
VideoGUI-undirected26.021.320.219.3
VideoGUI-directed65.370.068.871.3
Mean58.856.359.263.8

If format alignment explained the gains, all three runs would improve similarly, but they do not: Length is below Base on average, while Phase is 7.5 points above Length.

  • Second, our harness follows each base model's official norm-1000 coordinate normalization: Qwen [x1,y1,x2,y2] and Gemma [y1,x1,y2,x2]. Muse uses custom multi-cropping and bounding to norm-1000, as in its official paper.

  • Third, we use the original prompts and evaluation setups, ensuring reproducibility.

FigmaTrace with phase structure removed.

The Length and Action-log runs above are this control.

W2. We evaluate on the complete public sets wherever reported, ensuring fairness. Because these sets are public, closed models may have been trained on them (e.g., Claude's 100%). This is not the central point of the paper. We use these sets to show that FigmaTrace also improves out-of-domain, non-design tasks, lifting general navigation and agentic skills to the level of much larger models.

W3.

ComparisonReferenceFigmaTrace SFTDiff. [95% CI]
Qwen3.6-35B-A3B base→SFT32.058.3+26.3 [+23.9, +28.7]
Muse-Glimmer-30B base→SFT52.064.0+12.0 [+9.8, +14.2]
Gemma-4-31B base→SFT58.361.1+2.9 [+0.7, +4.9]
Qwen3.8-27B base→SFT58.863.8+5.0 [+3.0, +7.0]
Qwen3.8-27B length→phase56.363.8+7.5 [+5.2, +9.8]
Muse-Glimmer-30B length→phase58.164.0+5.8 [+4.2, +7.8]

We selected hyperparameters (Table 6) on a validation split held out from FigmaTrace training data, and trained with three seeds (42, 800, 9000).

Q1. See W1.

Q2. We evaluate end-to-end Figma tasks with a blind two-expert study scored across 30 paired final-state files against the readiness checklist (Table 5).

Passing ratingsBaseSFT
Qwen3.8-27B0 (0%)3 (5.0%)
Qwen-Flash-Next-LoRA8 (13.3%)12 (20.0%)
GLM-5.3-LoRA10 (16.7%)16 (26.7%)

Absolute pass rates are low because the tasks are long-horizon and the checklist is strict. Still, SFT improves all three models, and the gain grows with parameters.

Q3. See W3 (set sizes, sampling, CIs, seeds, hyperparameter selection).

[1] Xiao et al. "DesignBench: A comprehensive benchmark for MLLM-based front-end code generation." arXiv:2506.06251, 2025.

[2] Jeong et al. "CANVAS: A Benchmark for Vision-Language Models on Tool-Based User Interface Design." AAAI 2026.

AuthorsOpenReview ↗

Follow-up on the rebuttal

As the deadline for the rebuttal phase approaches, we would be happy to address any additional concerns about the paper. We look forward to your engagement and thank you for your time.

Thank you for the detailed rebuttal and new experiments.

W1. Partly addressed.

  • The segmentation ablation is useful: Length-based SFT falls below base, so the gains are not just "any SFT fixes format."
  • There is still no generic-GUI SFT control, so the contribution of FigmaTrace's design content remains unisolated.
  • I agree the harness uses native norm-1000, and I framed this imprecisely in my review. However, RQ4 attributes part of the gain to the base model not complying with that format: raw pixels on 10/150 GUI-Odyssey items, and no coordinates at all on some AndroidControl items. This is format alignment rather than design ability. Please report results after rescoring base outputs with pixel →\rightarrow norm-1000 conversion, with no-coordinate items reported separately.

W3 / Seeds.

  • Reporting the best of three seeds biases the gains upward, and the intervals exclude seed variance. Please report mean ±\pm std across seeds.
  • Please give the size and sampling method of each OOD set. "Complete public sets" does not match "150 analyzed GUI-Odyssey items" in RQ4.

The dataset is valuable, but the core concerns remain and no revised PDF has been uploaded. I keep my score for now

AuthorsOpenReview ↗

Response to Reviewer hrif

W1.

  • No-generic GUI SFT:

As a quick supplementary comparison, we evaluate WebCoder-27B [1]. It shares our Qwen3.8-27B base. We use the same prompts and temperature 0 as for the other models and evaluate on the same design evaluations below:

Qwen3.8-27BBaseWebCoder-27BFigmaTrace SFT
CANVAS (component sim.)0.7740.7060.794
DesignBench (final judge score)8.898.029.10
In-domain held-out (action acc.)46.348.453.9

We consistently observe that FigmaTrace training consistently gets the best scores out of the set as compared to the generic GUI trained WebCoder model which is fine tuned from Qwen3.8-27B base.

  • The norm-1000 is simply an observational weakness of the base models and their behaviors. More concretely, the reported results are clamped at 1000 as is standard practice. The 150 samples discussed above are sampled from model failures and not from the overall set. The overall GUI-Odyssey contains 1,666 episodes (averaging at 15.3 steps), out of which we observed 304 overall steps to produce out of bound pixel values. For AndroidControl, out of the 8444 steps across 1543 episodes, we observed that the model produces only-text instead of coordinates at least once in 23 episodes. These occurrences did not have a fixed pattern in our manual review.

W3.

  • All SFT results are the mean over three training seeds (42, 800, 9000). Table 6 lists the best hyperparameter configuration selected for each model on the FigmaTrace validation split. Below is the 95% CIs across seeds for the in-domain set:
ModelBase (± 95% CI)SFT, mean of 3 seeds (± 95% CI)
Qwen3.6-35B-A3B40.6 ± 4.148.6 ± 4.5
Muse-Glimmer-30B44.9 ± 5.061.7 ± 4.2
Gemma-4-31B45.0 ± 4.548.0 ± 4.1
Qwen3.8-27B46.3 ± 4.253.9 ± 4.9

For OOD set, we have previously included the CI:

ComparisonReferenceFigmaTrace SFTDiff. [95% CI]
Qwen3.6-35B-A3B base → SFT32.058.3+26.3 [+23.9, +28.7]
Muse-Glimmer-30B base → SFT52.064.0+12.0 [+9.8, +14.2]
Gemma-4-31B base → SFT58.361.1+2.9 [+0.7, +4.9]
Qwen3.8-27B base → SFT58.863.8+5.0 [+3.0, +7.0]
Qwen3.8-27B length → phase56.363.8+7.5 [+5.2, +9.8]
Muse-Glimmer-30B length → phase58.164.0+5.8 [+4.2, +7.8]

Find full dataset statistics below:

  • GUI-Odyssey: 1,666 episodes
  • AndroidControl: 1,543 episodes
  • Mind2Web (test_task split): 1,339 actions from 177 tasks
  • VideoGUI: 178 tasks covering 2712 atomic actions

We did not revise the paper on openreview since this is not a strict policy during the rebuttal phase according to ICLR (here). To avoid repeated work across reviewers (since a majority of reviewer complaints overlap), we will make a singular, atomic edit and include the DesignBench and CANVAS results in the main paper upon acceptance, and clarify the above details and include the other experiments such as the parameter size vs performance table in the appendix. We are happy to address any outstanding concerns.

[1] Dou, Zijun, et al. "WebCoder: Scaling General Frontend Development with Artifact-Grounded Agentic Reinforcement Learning." arXiv preprint arXiv:2610.23104 (2026).

Reviewer_bJjVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FigmaTrace collects expert Figma workflows and converts them into training trajectories using design-phase segmentation. The authors fine-tune four models and evaluate action prediction, transfer to other GUI tasks, and expert preferences over generated design files.

Critical strengths
  1. Expert workflows contain useful information that finished screenshots leave out, including how designers build components, revise layouts, and check their work. Collecting these intermediate decisions is a worthwhile data contribution.

  2. Phase-based segmentation improves average performance over length-based segmentation for both tested models. The reported 72% expert preference also provides some evidence that the improvements extend to the resulting files.

Critical weaknesses
  1. The central question is whether the models learn better design judgment or mainly become better at operating the interface. Most results measure action prediction and grounding. The expert study is encouraging, but the reported component/layout checklist pass rate is still only 5%, versus 0% for the base model. Relative preference does not yet establish that the models make substantially better design decisions.
  2. The phase-segmentation result has another plausible explanation: keeping a task fragment coherent helps training. Comparing against maximum-length cuts does not establish that the design-specific phase taxonomy is important. A comparison with coherent subtask segmentation would help distinguish learning design structure from simply avoiding broken contexts.
Questions for authors
  1. Which results show better design decisions after separating out failures to select or manipulate the intended element?
  2. Does design-phase segmentation still help compared with generic, coherent subtask segmentation?
2 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer bJjV

  • Succeeding at end-to-end figma design tasks requires a good understanding of both interface and design judgment and our dataset teaches both of these. The agent’s harness is a general computer-use one and hence gives it the freedom to explore not just the Figma interface (which is different on web and desktop app along with difference in keyboard shortcuts, etc for both) but also perform websearch for assets and references, similar to how humans do in the video captures. We faithfully capture these through FigmaTrace.

  • The 0% to 5% jump is due to the base model’s lack of design abilities.

Model NamePass Rate (across two SME validations)
Qwen3.8-27B Base0%
Qwen-Flash-Next Base13.3%
GLM-5.3 Base16.7%
Qwen3.8-27B SFT5%
Qwen-Flash-Next SFT20.0%
GLM-5.3 SFT26.7%

Our dataset doesn’t overindex on the GUI since annotator SMEs use visually distinct versions of web and app versions of Figma interchangeably. To show the gains persist on benchmarks with no GUI interaction, we run a transfer study on two distinct design datasets: DesignBench [1] (which tests performance on front end designing) and CANVAS [2] (which contains a limited set of skills that we expect to transfer from FigmaTrace). Below are the results for these experiments (dataset specific metrics used for each and all runs are executed with temp=0 to ensure minimal variance):

CANVAS:

ModelSSIM ↑LPIPS ↓Position similarityComponent similarity
Qwen3.8-27B Base0.8450.1720.9150.774
Qwen3.8-27B SFT0.8850.1310.9450.794
Gemma-4-31B-it0.7770.3830.6450.674
Gemma-4-31B-it SFT0.8010.1990.7550.701
Muse-Glimmer-30B0.7310.4510.6350.639
Muse-Glimmer-30B SFT0.8100.1820.8030.698

DesignBench

Generation split (430 samples)

ModelCLIP ↑SSIM ↑Compile success rate
Qwen3.8-27B Base0.7810.67788.4%
Qwen3.8-27B SFT0.8020.71790.6%
Muse-Glimmer-30B0.8050.67992.3%
Muse-Glimmer-30B SFT0.8240.68992.1%
Gemma-4-31B0.7310.64086.3%
Gemma-4-31B SFT0.7210.65988.1%

Edit split (359 samples)

ModelJudge score (0–10)CMLSCMCSCompile success rate
Qwen3.8-27B Base8.890.6200.53798.1%
Qwen3.8-27B SFT9.100.6400.54899.1%
Muse-Glimmer-30B8.840.5930.50598.3%
Muse-Glimmer-30B SFT8.910.6040.55198.0%
Gemma-4-31B8.780.5920.50696.4%
Gemma-4-31B SFT8.890.6010.52197.0%

Repair split (111 samples)

ModelJudge score (0–10)CMLSCMCSIssue accuracy
Qwen3.8-27B Base8.210.6160.5440.552
Qwen3.8-27B SFT8.350.6190.5560.581
Muse-Glimmer-30B8.050.5490.4650.438
Muse-Glimmer-30B SFT8.40.5770.5100.511
Gemma-4-31B8.050.5120.4430.573
Gemma-4-31B SFT8.240.5350.5100.610

We use the default prompts from each of these datasets for evaluation.

  1. To address this, we present an ablation using the design agnostic action boundaries from SME action logs. Please find the Qwen3.8-27B, step accuracy results below (evals run with temp=0 to minimize variance):
BenchmarkBaseLengthAction-logPhase (ours)
GUI-Odyssey44.040.045.153.7
AndroidControl82.767.380.999.1
Mind2Web vp80069.369.370.070.7
Mind2Web vp100065.370.070.268.7
VideoGUI-undirected26.021.320.219.3
VideoGUI-directed65.370.068.871.3
Mean58.856.359.263.8

All three experiments use the same recordings, hyperparameters and training budget to keep results consistent.

Q1. The cleanest separation comes from the two additional benchmarks reported under W1, since neither involves selecting an on-screen element. DesignBench has no GUI interaction at all and the model writes, edits, or repairs frontend code. CANVAS builds designs through structured MCP tool calls and not through coordinate-based clicks. Any gain on these benchmarks therefore cannot come from better element selection or manipulation.

Q2. As addressed in W2, the action-log segmentation still falls short in terms of absolute performance on OOD sets. When compared against length based segmentation, action log based segmentation improves performance from 56.3 to 59.2%, which confirms the reviewer’s point that coherence alone helps. We observe an overall difference in mean of 4.6 points above action-log when using the phase based segmentation, proving that it is the most effective method for FigmaTrace.

We are happy to answer further questions and will incorporate these additional results in the camera ready version of the paper.

[1] Xiao, Jingyu, et al. "Designbench: A comprehensive benchmark for mllm-based front-end code generation." arXiv preprint arXiv:2506.06251 (2025). [2] Jeong, Daeheon, et al. "CANVAS: A Benchmark for Vision-Language Models on Tool-Based User Interface Design." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 40. No. 26. 2026.

AuthorsOpenReview ↗

Follow-up on the rebuttal

As the deadline for the rebuttal phase approaches, we would be happy to address any additional concerns about the paper. We look forward to your engagement and thank you for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.