Tuesday 12pm, 25 September 2018


Reverse Engineering User Interface Structure to Enable Access and Design Reuse

Amanda Swearngin

PhD Student - University of Washington


In this talk, I will present two projects. Genie is a system that reverse engineers a web application to enable it to be used through additional input techniques. For example, you can use your voice as an input method to a web page that was only implemented for mouse input. The system uses program analysis techniques to reverse engineer an abstract command model of an interface and provides a template to create alternate command interfaces. I will also talk about a second project called Rewire that enables interface designers more seamlessly adapt screenshots into their design mockups. Interface designers use screenshots in their design, but they are unstructured and hard to edit. Designers recreate vectorized versions of them by hand to reuse or edit parts of the screenshot. Rewire infers a vectorized design document from an example screenshot where each UI component is an object with editable shape and style properties. Rewire provides three design assistance modes that help designers reuse or redraw components of the screenshot design.


Amanda is a 4th year PhD student at the University of Washington working with Andy Ko and James Fogarty. Her research has focused on interface reverse engineering for accessibility and user interface design, and is recent interested in using program synthesis to help designers explore variations. She has interned at Adobe Research and Google and was previously a Software Engineer at Microsoft.