A research team led by Shinji Watanabe, Associate Professor at the Language Technologies Institute (CMU), visited Lisbon from June 15 to 18 under the scope of the CMU Portugal supported project, “ACE-FM: Adaptive Conversational Engine powered by Foundation Models”. The project aims to enhance the capability of speech foundation models to understand long-form, multi-party conversations toward conversational AI, supporting the collaboration between CMU and PRR Mobilizing Agenda Accelerat.AI, led at INESC ID by Alberto Abad, Associate Professor at Instituto Superior Técnico (Técnico) and INESC ID researcher.
The visit agenda featured an intensive two-day workshop hosted by INESC-ID, designed to exchange knowledge and officially launch the collaboration.
“As part of the CMU Portugal initiative, my visit to Portugal focused on launching a new research collaboration with Prof. Alberto Abad, centered around the development of Adaptive Conversational Engines. We aim to leverage several foundation models developed at CMU as pre-trained components for this work. During the visit, we provided an overview of these models through lectures and conducted a hands-on tutorial to support researchers in the Portugal team in using them effectively. This visit served as a productive starting point for joint research and knowledge exchange.”, shared Shinji Watanabe.
On Monday, June 16, the teams introduced the ACE-FM project and both held presentations with Shinji Watanabe delivering a keynote titled “Open Whisper-Style Speech Models: Transparency, Scalability, and Advancing Explainability.”
In the afternoon, hands-on tutorials were conducted by CMU students Masao Someki, MLT student, and William Chen, PhD student, who joined Shinji Watanabe on this visit to Lisbon. The tutorials focused on fine-tuning techniques and model training. The second day was fully dedicated to practical sessions, including model fine-tuning with custom data and an introduction to ESPnet3, CMU’s next-generation speech modeling platform.
According to Alberto Abad “The team led by Prof. Shinji Watanabe is pioneering in the development of large-scale speech foundation models. Their contributions to the field are truly remarkable, and Prof. Watanabe’s research principles are both genuine and ambitious: advancing open-source speech technology with a strong emphasis on benchmarking and reproducibility, aiming to match — and even surpass — the performance of large proprietary models developed by major industrial players. The CMU program initiative offers a valuable opportunity to learn from their unique expertise. It will support our progress in developing more adaptive conversational speech foundation models — both tailored to specific domains and to the particularities of European Portuguese.”
Both teams anticipate that this collaboration will lay the foundation for future projects and long-term academic exchanges.
“Through the visit, we gained a deeper understanding of each other’s expertise, research styles, and focus areas. Prof. Abad also introduced us to several colleagues working in language technologies, and both he and Prof. Trancoso shared insights into their team structure and successful past collaborations within the CMU Portugal program. We also explored the potential of leveraging the CMU-Portugal dual degree program to further strengthen our collaborative efforts.”, highlighted the CMU PI.
“The event marked the kick-off of the activities of this CMU Portugal collaboration. It helped lay the groundwork that will, by the end of this initiative, advance the state of the art in the adaptation of speech foundation models. Moreover, the outcomes of this collaboration will benefit not only the Accelerat.ai project, but also other PRR-supported initiatives, including the development of a national Portuguese large language model,” shares the Portuguese PI.
The 2024 CMU Portugal PRR Call for Collaborative Research Grants aims to strengthen collaboration between CMU and Portuguese research institutions aligned with the Portuguese Recovery and Resilience Plan (PRR) Mobilizing Agendas. The call funded 12 projects, representing a significant step in fostering impactful research between Portugal and CMU.
Learn more about the 12 CMU Portugal PRR projects here.
