911������

Two new research papers examine what happens when human behaviors are linked up with the capabilities of AI models. The Chartist and Typoist models interact with, respectively, data visualizations and touchscreen keyboards, in the same way a human user would, giving a clearer ‘glass-box’ view into how and why these models operate the way they do. The Chartist model can read and understand charts like a human would, which could provide instant feedback for creating more understandable designs. The human-like typing errors made by Typoist, created with the Google Gboard team, can help make text entry on mobiles more accurate. Both models are being presented in late April at CHI, the premier conference for research in human-computer interaction.

If you ask people to look at the same statistical chart and answer questions based on the data, their eye movements will vary a lot, as shown in the example image. “How people move their eyes around a chart and how they then use that to comprehend what the chart is about, these two things are interrelated,” says professor Antti Oulasvirta from the Finnish Center for Artificial Intelligence FCAI and 911������. Chartist can replicate the diversity of human eye movements when reading charts. Combined with the capability of a large language model, Chartist can then integrate what was viewed at each spot on a chart. “This connection has not been made before,” continues Oulasvirta. “We can ask the model to gather information from a chart to answer a particular question, and it can make sense of the chart as a whole, in ways that really simulate human behavior.” 

Typoist is the latest state-of-the-art model of typing behavior that Oulasvirta and colleagues have been working on for a number of years. “Nearly everyone has a mobile device, but we know that for some, like aging adults, text entry can be very hard,” says Oulasvirta. “If we want a society where everyone can participate, we need to know why people make errors when they’re typing.” Errors might arise from being in a rush while texting to forgetfulness, and researchers classify them in three ways: slips of the finger; lapses or forgetting to give a command to fingers; and mistakes, where what is seen onscreen is wrong and a typo-in-progress is missed.

But how does having a computer model that makes these same errors help make typing easier for people? Simulating how people type can lead to the design of better keyboards, without the need to test them with lots of users. When a keyboard app like Gboard has a billion users, even a small change to it can have a huge societal impact, Shi points out, so it’s crucial to make sure it works optimally for users. “A simulated typist, like Typoist, that behaves like a human, can produce lots of data to guide the development of better text entry systems for real human users,” says Shi.

Both Chartist and Typoist, models that can accurately simulate how people read charts or type on touchscreen keyboards, offer routes to quicker and cheaper user research through synthetic data, the production of large volumes of human-like measurements. Moreover, the new way to create models using simulations and machine learning offers a path to increased transparency and explainability in AI systems for human-computer interaction. The Typoist paper has been recognized with an honorable mention, given to the top 5% of papers at the CHI conference. 

Read more in the papers: