Large language models (LLMs) are used to generate user interface (UI) code, yet the accessibility of this code is difficult to evaluate in a controlled way. Existing evaluations typically examine full-generated webpages or loosely specified tasks, which makes it difficult to determine where accessibility outcomes come from. We introduce PACE for “Prompt Accessibility Controlled Evaluation,” a benchmarking testbed for controlled evaluation of accessibility in LLM-generated HTML form components. Using PACE, we evaluated the accessibility of 58 form components generated by five developer-centric LLMs with variations across three system prompts with increasing accessibility guidance, and three request variants containing increasing detail about each form components. We collected 10 trials per configuration for a total of n = 26,100 outputs. Our findings show that accessibility in LLM-generated code moved with prompt construction and with the circumstance around it (e.g., model, form component, and measure used to assess the output). This pattern calls for evaluation designs that can better separate sources of variation in the accessibility of LLM-generated code.

Yumeng Ma, Jacob O. Wobbrock, Joe Devon, Aaron Gustafson, Michael Fairchild, Jennifer Mankoff, and Venkatesh Potluri. 2026. Prompt and Circumstance: Evaluating Accessibility in LLM-Generated HTML Form Components. In Proceedings of the 28th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’26). ACM. (pdf)

Open source version here: https://a11yhood.org/product/pace