Entrepreneurial Research Initiative: INSIDE Research Progress on Human-Robot Interaction to Help Children with Impaired Development
 |
How can robots plan their course of action to coordinate with and accommodate for the actions of their human teammates? How can task, context and environment information collected from a set of networked sensors be exploited to create more natural and engaging interactions between humans and robots involved in a joint cooperative activity in a physical environment? – These were the main questions in the beginning of the Entrepreneurial Research Initiative (ERI) INSIDE. |
After one year of the INSIDE ERI and more than 20 researchers involved, the results can already be seen and some questions can already be answered. – Some steps towards getting the technologies closer to the end-user, studies made with children with autism in a Portuguese Hospital, some preliminary publications, the conclusion of two Master thesis on the context of the project and a lot more.
This project is led by Francisco Melo, from INESC-ID and Instituto Superior Técnico (Universidade de Lisboa), and Manuela Veloso, faculty member at the School of Computer Science at Carnegie Mellon University (CMU). To know more about the ERI INSIDE that is carried out in the scope of the CMU Portugal Program, funded by Fundação para a Ciência e a Tecnologia (FCT), read the following interview with both PIs.
 |
 |
|
Portuguese PI
Francisco Melo (INESC-ID/ IST-UL) |
CMU PI
Manuela Veloso (CS) |
CMU Portugal Program: How do you comment on this first year of activity of the ERI?
INSIDE PIs : The first year was very active at several levels.
In a broad, project-wide level, we had extensive meetings between the technical partners and the medical team. In light of the multidisciplinary nature of the team, these meetings were important learning opportunities where we were able to reach a “common language” and solidify our vision of the project. Additionally, these meetings were decisive in designing the scenarios in which we will test the technology developed in the context of INSIDE with end-users.
In terms of research, we investigated symbiotic human-robot interaction, a central topic to the project, where both humans and robot interact in a mutually beneficial way: robots assist humans in their activity but also rely on human assistance to complete their task.
We also took important steps in terms of bringing our work closer to end-users. We ran several studies in Hospital Garcia de Orta where robots and children with Autism Spectrum Disorders collaborated during therapy sessions. These studies, which relied on a Wizard-of-Oz framework (where the robot is tele-operated without the user realizing) allowed us to evaluate the response of the children to the robot while, at the same time, collect extensive data (audio, video and sensor data from the robot) to be used for further development.
“We also took important steps in terms of bringing our work closer to end-users”
 |
 |
CMU Portugal Program: What outcomes can be communicated so far?
INSIDE PIs : Although this was only the first year of the INSIDE initiative, we are very excited about the pace at which our work is progressing in all three axis around which INSIDE is structured (research, innovation and technology and education).
In terms of research, our work on human-robot interaction already led to some preliminary publications, both in international conferences and journals, on the topics of ad hoc teamwork, human-agent interaction in tutoring scenarios, multimodal human-robot interaction (for example using lights and speech for task-specific interaction).
Additionally, we had two MSc students concluding their thesis in the context of the project, with two other on their way to conclusion. We have an active dual-PhD student working in human-robot collaboration and are currently in the process of recruiting additional PhD students. The integration of both MSc and PhD students in the project is, in our view, an important outcome that meets the educative goals of INSIDE.
Still in terms of education, we hired two excellent post-doctoral researchers from Portugal and Spain who have been important additions to the team. We also integrated several younger researchers in the team who have helped us in the significant effort of deploying the robot (and associated infrastructure) to the hospital.
In terms of technology and innovation, we emphasize the preliminary studies involving end-users in the hospital—which were, perhaps, the most significant outcome of this first year. The preliminary studies were essential at several levels. First, the studies allowed us to successively improve the layout of the interaction scenario, the design of the tasks that the robot and the children jointly face and how the interaction between the robot and the children takes place.
Second, they allowed us to collect extensive data (audio, video and sensor data from the robot), which is now being cleaned and annotated and from which the robot controllers and behaviors will be constructed.
Thirdly, they allowed us to gain important insights into the children’s response to the robot, both to the robots’ requests and to several important situations created around the robot during the therapy session—situations which have an important role in better understanding the potential impact of our intervention in therapy.
Overall, this has been a very productive first year and we have great expectations of the times ahead.
“(…) we are very excited about the pace at which our work is progressing in all three axis around which INSIDE is structured (research, innovation and technology and education).”
 |
 |
CMU Portugal Program: How is the research work with Hospital Garcia de Orta going on?
INSIDE PIs : The interaction with the medical team from Hospital Garcia de Orta has been, in our opinion, both fundamental and very enriching. On one hand, it provides a much needed, humane vision of our research, contrasting with the more technology-oriented perspective of the technical partners.
On the other hand, the medical team played a central role in the studies conducted so far. Their intervention took place at several levels (logistic, technical and scientific).
The studies were conducted in the facilities of the hospital, with the HGO team providing the technical partners extended access to the facilities during preparation. The medical team was responsible for selecting the children that participated in the studies, while guaranteeing strict adherence to established ethical practices involved in studies with children.
Additionally, the design of the tasks in the therapy sessions was done in close collaboration with the medical team, and the robot operation was closely monitored/oriented by elements of the medical team standing by the wizard. Currently, the medical team is finalizing the analysis of the video data from the studies.
“The interaction with the medical team from Hospital Garcia de Orta has been, in our opinion, both fundamental and very enriching.”
CMU Portugal Program: What has been the role of the partner companies involved so far?
INSIDE PIs : INSIDE includes three companies in its consortium: PLUX, IDMind and VoiceInteraction. They have participated in all meetings, bringing important insights from the industry regarding the role of the different technologies in INSIDE. The companies have also played an important role in all the activity developed during this first year:
– PLUX, whose activity is centered around biometric sensing and processing of biometric signals, developed a solution allowing hear-beat monitors to be incorporated in the clothes to be worn by the children participating in the studies, often unwilling to wear unknown devices. This device was already tested in one of the studies.
– IDMind has played a central role in all matters concerning the robot platform used in the studies. They designed and built the robot used so far, and have included a number of adaptations to meet the research goals of INSIDE. Additionally, they are in the process of concluding the design of a robotic platform especially conceived for the INSIDE project.
– Finally, in one of the studies the voice of the robot was synthesized (instead of pre-recorded) using VoiceInteraction’s speech synthesis module, DIXI.
CMU Portugal Program: What are the major activities scheduled for the next year?
INSIDE PIs : We are currently exploring several work avenues in parallel. On a more research-oriented level, we are looking into the idea of symbiotic human-robot interaction from different perspectives. In particular, we have investigated human-robot ad hoc teams and collaborative manipulation as implicit forms of symbiotic autonomy. Additionally, we have also been working on identifying and synthesizing emotions in speech, and on personalizing the appearance of the robot using expressions and lighting to complement speech during task execution.
In terms of technology, we are currently cleaning and annotating the data collected in the studies while designing a semi-autonomous control architecture for the robot. We expect to run a new study in early Spring where the robot will interact with the children autonomously, and the human operator will only be monitoring the robot behavior, correcting it when necessary. This is an important step towards having a fully autonomous robot interact with children in a therapy session.
Finally, in terms of education, we are very excited with the possibility of bringing in one or even two additional PhD students in the project.
“We expect to run a new study in early Spring where the robot will interact with the children autonomously, and the human operator will only be monitoring the robot behavior, correcting it when necessary.”
 |
 |
CMU Portugal Program: Now that a year has passed, what do you feel that are the main differences between an ERI and other research projects?
INSIDE PIs : Our perception is that an ERI has a broader scope than a “standard” research project. Our work does not (and should not) focus exclusively on research or research-driven technology, but should also look beyond research into the potential application of this research. In a sense, the broader scope of INSIDE pushes our work out of the lab, allowing a much clearer and more immediate understanding of the implications that our research and the technology we developed may have in society.
Additionally, in INSIDE we have been particularly driven by the benefits that the integration of young researchers in the project may have in their education and career.
“(…) an ERI has a broader scope than a ‘standard’ research project.”
_______________________________________
Entrepreneurial Research Initiatives (ERIs) are projects in science, engineering, management and policy that link both fundamental and applied research to technological innovation and economic development. This bridging is pursued by explicitly focusing on important real world problems entailing significant scientific challenges – more
Photo Credits: Exame Informática and INSIDE website
