A main focus of this work package is social interaction by means of multi modal signals, including recognition, interpretation, and generation of speech and language as well as various non-verbal signals. The WP5 work package is meant to create a demonstrator where many of these techniques can be shown in an integrated way. A second major objective is the transfer of knowledge between industrial partners and academia, by creating products mainly by our industrial partners.
Requirements capturing is done by the creation of use case scenarios, mostly by KLPD, in cooperation with other partners. On the basis of this, a serious game will be developed by T-Xchange, partly based on research done by T-Xchange in work package WP4. An implementation and realization of the use cases will be done by Re-lion. Finally, an extensive form of evaluation will be done by regular user testing and evaluation by all partners.
The content of the use case is intended to focus on aspects that are relevant in police training. Mental wellness, social engagement and security awareness are part of the training activity of social workers, police and others, often operating in small teams. An intelligent virtual environment will be created in which policing activities can be prepared and where scenarios can be tested and evaluated. The focus lies on the intelligent security of neighborhoods, and includes training situations for police teams to handle challenging social situations. Current simulators as for instance the Virtual Infantry Trainer, under development by Re-Lion provide a good starting point, and offer a full fledged VR environment with a fair amount of visual richness and realism.
Based upon this existing expertise, a representative environment will be created that incorporates smart sensors like cameras, heat sensors, microphones, tactile sensors, position and (neuro)physiological sensors. Automatic detection of relevant or strange behavior and activities will be supported, including scenario generation where such activities can be specified by human instructors. High quality interaction between trainees, human instructors, and virtual characters is one of the goals of the training aspect of the simulator. Smart sensing will be used to interpret behavioral aspects, like stress detection, back channeling frustration from the human part, or for signaling agreement or disagreement in dialogue. Among the techniques to be used we count interpretation of hand gestures or pose recognition. The response of the system, towards the humans, will consist of verbal as well as nonverbal behavior, including character animation for sending social cues.