Summary: How do non-expert users exploit simultaneous
inputs in multimodal interaction?
Knut Kvale, John Rugelbak and Ingunn Amdal1
firstname.lastname@example.org , ,email@example.com firstname.lastname@example.org
This paper evaluates scenario based user tests of speech-centric multimodal interaction on a
small mobile terminal. Non-expert users solved tasks in the tourist guide domain using a
functional multimodal PDA-based application. The tasks required pen and speech input, but
the users were free to choose either sequential or simultaneous pen and speech input at each
step in the dialogue.
Multimodal interfaces are still a novelty to most users so we had to explain this functionality
to the test users. The format of the introduction had a noticeable effect on user behaviour.
Users who had seen a video demonstration used simultaneous pen and speech input more
often than the users who had had a text only introduction even if the same information was
present in both formats. 9 of 14 subjects who had seen the video demo, applied simultaneous
pen and speech input instantly. We therefore claim that people will use simultaneous
multimodal input when they have been properly introduced to this functionality. However,
simultaneous use of pen and speech may impose an extra cognitive load, at least until people