A paper co-authored by CMU Portugal Dual Ph.D. students in Software Engineering, Catarina Gamboa, Cláudia Mamede, Daniel Ramos, and Paulo Canelas, received the Best Paper Award at the 2nd International Workshop on Large Language Models for Code (LLM4Code). The paper was also co-authored by Kush Jain, PhD student at CMU and CMU Portugal faculty member Claire Le Goues, who supervises the research work of Cláudia Mamede and Daniel Ramos. The workshop, co-located with the International Conference on Software Engineering (ICSE 2025), took place on May 3rd in Ottawa, Canada.
The winning paper, “Are Large Language Models memorizing bug benchmarks?” evaluates popular Large Language Models for data leakage susceptibility. Older and smaller models show significant evidence of memorization in widely used benchmarks, while recent models, which are trained on larger datasets, exhibit limited signs of leakage. These findings emphasize the need for careful benchmark selection and robust metrics to accurately assess model capabilities.
Cláudia Mamede presented the paper at the Workshop, stating that she had a great time at ICSE: “We got a lot of thoughtful questions and had great follow-up conversations with people interested in LLMs, software engineering… and bugs! 🐞”