Valerio Terragni | publications

2024

ESE
Semantic matching in GUI test reuse Farideh Khalili, Leonardo Mariani, Ali Mohebbi, Mauro Pezzè, Valerio Terragni Empirical Software Engineering , pages 70 , 2024 [BIBTEX] [Abstract] [PDF] [DOI]
Reusing test cases across apps that share similar functionalities reduces both the effort required to produce useful test cases and the time to offer reliable apps to the market. The main approaches to reuse test cases across apps combine different semantic matching and test generation algorithms to migrate test cases across Android apps. In this paper we define a general framework to evaluate the impact and effectiveness of different choices of semantic matching with Test Reuse approaches on migrating test cases across Android apps. We offer a thorough comparative evaluation of the many possible choices for the components of test migration processes. We propose an approach that combines the most effective choices for each component of the test migration process to obtain an effective approach. We report the results of an experimental evaluation on 8,099 GUI events from 337 test configurations. The results attest the prominent impact of semantic matching on test reuse. They indicate that sentence level perform better than word level embedding techniques. They surprisingly suggest a negligible impact of the corpus of documents used for building the word embedding model for the Semantic Matching Algorithm. They provide evidence that semantic matching of events of selected types perform better than semantic matching of events of all types. They show that the effectiveness of overall Test Reuse approach depends on the characteristics of the test suites and apps. The replication package that we make publicly available online (https://star.inf.usi.ch/#/software-data/11) allows researchers and practitioners to refine the results with additional experiments and evaluate other choices for test reuse components.

@article{khaliliESE2024, author = {Khalili, Farideh and Mariani, Leonardo and Mohebbi, Ali and Pezz{\`{e}}, Mauro and Terragni, Valerio}, title = {{Semantic matching in GUI test reuse}}, journal = {Empirical Software Engineering}, volume = {29}, number = {3}, pages = {70}, year = {2024}, doi = {10.1007/S10664-023-10406-8} }
ASE 2024
MR-Adopt: Automatic Deduction of Input Transformation Function for Metamorphic Testing Congying Xu, Songqiang Chen, Jiarong Wu, Shing-Chi Cheung, Valerio Terragni, Hengcheng Zhu, Jialun Cao In Proceedings of the IEEE/ACM International Conference on Automated Software Engineering , 2024 [BIBTEX] [PDF]
@inproceedings{xu2024mrAdopt, author = {Xu, Congying and Chen, Songqiang and Wu, Jiarong and Cheung, Shing-Chi and Terragni, Valerio and Zhu, Hengcheng and Cao, Jialun}, journal = {arXiv preprint arXiv:2408.15815}, title = {MR-Adopt: Automatic Deduction of Input Transformation Function for Metamorphic Testing}, booktitle = {IEEE/ACM International Conference on Automated Software Engineering}, year = {2024} }
2030 SE
The Future of Software Engineering in an AI-Driven World Valerio Terragni, Partha Roop, Kelly Blincoe In Proceedings of the Workshop 2030 Software Engineering co-located with FSE 2024 , 2024 [BIBTEX] [Abstract] [PDF]
A paradigm shift is underway in Software Engineering, with AI systems such as LLMs gaining increasing importance for improving software development productivity. This trend is anticipated to persist. In the next five years, we will likely see an increasing symbiotic partnership between human developers and AI. The Software Engineering research community cannot afford to overlook this trend; we must address the key research challenges posed by the integration of AI into the software development process. In this paper, we present our vision of the future of software development in an AI-Driven world and explore the key challenges that our research community should address to realize this vision.

@inproceedings{terragni2024future, title = {{The Future of Software Engineering in an AI-Driven World}}, author = {Terragni, Valerio and Roop, Partha and Blincoe, Kelly}, year = {2024}, url = {https://arxiv.org/abs/2406.07737}, booktitle = {Workshop 2030 Software Engineering co-located with FSE 2024} }
arXiv 2024
Automated Trustworthiness Testing for Machine Learning Classifiers Steven Cho, Seaton Cousins-Baxter, Stefano Ruberto, Valerio Terragni arXiv , 2024 [BIBTEX] [Abstract] [PDF]
Machine Learning (ML) has become an integral part of our society, commonly used in critical domains such as finance, healthcare, and transportation. Therefore, it is crucial to evaluate not only whether ML models make correct predictions but also whether they do so for the correct reasons, ensuring our trust that will perform well on unseen data. This concept is known as trustworthiness in ML. Recently, explainable techniques (e.g., LIME, SHAP) have been developed to interpret the decision-making processes of ML models, providing explanations for their predictions (e.g., words in the input that influenced the prediction the most). Assessing the plausibility of these explanations can enhance our confidence in the models’ trustworthiness. However, current approaches typically rely on human judgment to determine the plausibility of these explanations. This paper proposes TOWER, the first technique to automatically create trustworthiness oracles that determine whether text classifier predictions are trustworthy. It leverages word embeddings to automatically evaluate the trustworthiness of a model-agnostic text classifiers based on the outputs of explanatory techniques. Our hypothesis is that a prediction is trustworthy if the words in its explanation are semantically related to the predicted class. We perform unsupervised learning with untrustworthy models obtained from noisy data to find the optimal configuration of TOWER. We then evaluated TOWER on a human-labeled trustworthiness dataset that we created. The results show that TOWER can detect a decrease in trustworthiness as noise increases, but is not effective when evaluated against the human-labeled dataset. Our initial experiments suggest that our hypothesis is valid and promising, but further research is needed to better understand the relationship between explanations and trustworthiness issues.

@article{cho2024automated, title = {Automated Trustworthiness Testing for Machine Learning Classifiers}, author = {Cho, Steven and Cousins-Baxter, Seaton and Ruberto, Stefano and Terragni, Valerio}, year = {2024}, url = {https://arxiv.org/abs/2406.05251}, journal = {arXiv}, eprint = {2406.05251}, archiveprefix = {arXiv} }
ACM TOSEM
MR-Scout: Automated Synthesis of Metamorphic Relations from Existing Test Cases Congying Xu, Valerio Terragni, Hengcheng Zhu, Jiarong Wu, Shing-Chi Cheung ACM Transaction on Software Engineering and Methodology , 2024 [BIBTEX] [Abstract] [PDF] [DOI]
Metamorphic Testing (MT) alleviates the oracle problem by defining oracles based on metamorphic relations (MRs), that govern multiple related inputs and their outputs. However, designing MRs is challenging, as it requires domain-specific knowledge. This hinders the widespread adoption of MT. We observe that developer-written test cases can embed domain knowledge that encodes MRs. Such encoded MRs could be synthesized for testing not only their original programs but also other programs that share similar functionalities. In this paper, we propose MR-Scout to automatically synthesize MRs from test cases in open-source software (OSS) projects. MR-Scout first discovers MR-encoded test cases (MTCs), and then synthesizes the encoded MRs into parameterized methods (called codified MRs), and filters out MRs that demonstrate poor quality for new test case generation. MR-Scout discovered over 11,000 MTCs from 701 OSS projects. Experimental results show that over 97% of codified MRs are of high quality for automated test case generation, demonstrating the practical applicability of MR-Scout. Furthermore, codified-MRs-based tests effectively enhance the test adequacy of programs with developer-written tests, leading to 13.52% and 9.42% increases in line coverage and mutation score, respectively. Our qualitative study shows that 55.76% to 76.92% of codified MRs are easily comprehensible for developers.

@article{xu-tosem-2024, author = {Xu, Congying and Terragni, Valerio and Zhu, Hengcheng and Wu, Jiarong and Cheung, Shing-Chi}, title = {{MR-Scout: Automated Synthesis of Metamorphic Relations from Existing Test Cases}}, year = {2024}, publisher = {ACM}, doi = {10.1145/3656340}, journal = {ACM Transaction on Software Engineering and Methodology}, month = apr }
ACM TOSEM
StubCoder: Automated Generation and Repair of Stub Code for Mock Objects Hengcheng Zhu, Lili Wei, Valerio Terragni, Yepang Liu, Shing-Chi Cheung, Jiarong Wu, Qin Sheng, Bing Zhang, Lihong Song ACM Transaction on Software Engineering and Methodology , pages 16:1–16:31 , 2024 [BIBTEX] [Abstract] [PDF] [DOI]
Mocking is an essential unit testing technique for isolating the class under test from its dependencies. Developers often leverage mocking frameworks to develop stub code that specifies the behaviors of mock objects. However, developing and maintaining stub code is labor-intensive and error-prone. In this article, we present StubCoder to automatically generate and repair stub code for regression testing. StubCoder implements a novel evolutionary algorithm that synthesizes test-passing stub code guided by the runtime behavior of test cases. We evaluated our proposed approach on 59 test cases from 13 open source projects. Our evaluation results show that StubCoder can effectively generate stub code for incomplete test cases without stub code and repair obsolete test cases with broken stub code.

@article{zhu-tosem-2024, author = {Zhu, Hengcheng and Wei, Lili and Terragni, Valerio and Liu, Yepang and Cheung, Shing-Chi and Wu, Jiarong and Sheng, Qin and Zhang, Bing and Song, Lihong}, title = {{StubCoder: Automated Generation and Repair of Stub Code for Mock Objects}}, year = {2024}, volume = {33}, number = {1}, publisher = {ACM}, pages = {16:1--16:31}, doi = {10.1145/3617171}, journal = {ACM Transaction on Software Engineering and Methodology}, month = nov, articleno = {16}, numpages = {31} }
IEEE TSE
GenMorph: Automatically Generating Metamorphic Relations via Genetic Programming Jon Ayerdi, Valerio Terragni, Gunel Jahangirova, Aitor Arrieta, Paolo Tonella IEEE Transactions on Software Engineering , pages 1-12 , 2024 [BIBTEX] [Abstract] [PDF] [DOI]
Metamorphic testing is a popular approach that aims to alleviate the oracle problem in software testing. At the core of this approach are Metamorphic Relations (MRs), specifying properties that hold among multiple test inputs and corresponding outputs. Deriving MRs is mostly a manual activity, since their automated generation is a challenging and largely unexplored problem. This paper presents GenMorph, a technique to automatically generate MRs for Java methods that involve inputs and outputs that are boolean, numerical, or ordered sequences. GenMorph uses an evolutionary algorithm to search for effective test oracles, i.e., oracles that trigger no false alarms and expose software faults in the method under test. The proposed search algorithm is guided by two fitness functions that measure the number of false alarms and the number of missed faults for the generated MRs. Our results show that GenMorph generates effective MRs for 18 out of 23 methods (mutation score >= 20%). Furthermore, it can increase Randoop’s fault detection capability in 7 out of 23 methods, and Evosuite’s in 14 out of 23 methods. When compared with AUTOMR, a state-of-the-art MR generator, GenMorph also outperformed its fault detection capability in 9 out of 10 methods.

@article{ayerdi-genmorph-tse-2024, author = {Ayerdi, Jon and Terragni, Valerio and Jahangirova, Gunel and Arrieta, Aitor and Tonella, Paolo}, journal = {IEEE Transactions on Software Engineering}, title = {{GenMorph: Automatically Generating Metamorphic Relations via Genetic Programming}}, year = {2024}, volume = {}, number = {}, pages = {1-12}, doi = {10.1109/TSE.2024.3407840} }
FSE 2024
Understanding the Impact of APIs Behavioral Breaking Changes on Client Applications Dhanushka Jayasuriya, Valerio Terragni, Jens Dietrich, Kelly Blincoe Proceedings of the ACM on Software Engineering (PACMSE) , pages In press , 2024 [BIBTEX] [Abstract] [PDF]
Libraries play a significant role in software development as they provide reusable functionality, which helps expedite the development process. As libraries evolve, they release new versions with optimisations like new functionality, bug fixes, and patches for known security vulnerabilities. To obtain these optimisations, the client applications that depend on these libraries must update to use the latest version. However, this can cause software failures in the clients if the update includes breaking changes. These breaking changes can be divided into syntactic and semantic (behavioral) breaking changes. While there has been considerable research on syntactic breaking changes introduced between library updates and their impact on client projects, there is a notable lack of research regarding behavioral breaking changes introduced during library updates and their impacts on clients. We conducted an empirical analysis to identify the impact behavioral breaking changes have on clients by examining the impact of dependency updates on client test suites. We examined a set of java projects built using Maven, which included 30,548 dependencies under 8,086 Maven artifacts. We automatically updated out-of-date dependencies and ran the client test suites. We found that 2.30% of these updates had behavioral breaking changes that impacted client tests. Our results show that most breaking changes were introduced during a non-Major dependency update, violating the semantic versioning scheme. We further analyzed the effects these behavioral breaking changes have on client tests. We present a taxonomy of effects related to these changes, which we broadly categorize as Test Failures and Test Errors. Our results further indicate that the library developers did not adequately document the exceptions thrown due to precondition violations.

@article{jayasuriya2024behavioral, author = {Jayasuriya, Dhanushka and Terragni, Valerio and Dietrich, Jens and Blincoe, Kelly}, title = {{Understanding the Impact of APIs Behavioral Breaking Changes on Client Applications}}, journal = {Proceedings of the ACM on Software Engineering (PACMSE)}, pages = {In press}, year = {2024} }
ESE
An Extended Study of Syntactic Breaking Changes in the Wild Dhanushka Jayasuriya, Samuel Ou, Saakshi Hegde, Valerio Terragni, Jens Dietrich, Kelly Blincoe Empirical Software Engineering , pages In press , 2024 [BIBTEX]
@article{jayasuriya2024emse, author = {Jayasuriya, Dhanushka and Ou, Samuel and Hegde, Saakshi and Terragni, Valerio and Dietrich, Jens and Blincoe, Kelly}, title = {An Extended Study of Syntactic Breaking Changes in the Wild}, journal = {Empirical Software Engineering}, year = {2024}, pages = {In press} }

2023

IEEE Software
Fostering Professionalism in Software Engineering: An Early-Exposure Approach Valerio Terragni, Catherine Watson, Rowe, Nasser Giacaman IEEE Software , pages 47-54 , 2023 [BIBTEX] [Abstract] [PDF] [DOI]
In this article, we advocate for an early-exposure approach to professionalism in software engineering. We believe that it is important to start preparing students for the “culture” of software professionalism from early on, which gives them a competitive edge in their careers.

@article{terragnifosteringapproach, author = {Terragni, Valerio and Watson, Catherine and Rowe and Giacaman, Nasser}, journal = {IEEE Software }, title = {{Fostering Professionalism in Software Engineering: An Early-Exposure Approach}}, year = {2023}, volume = {40}, number = {6}, pages = {47-54}, doi = {10.1109/ms.2023.3291711} }
ISSTA 2023
Understanding Breaking Changes in the Wild Dhanuska Jayasuriya, Valerio Terragni, Jens Dietrich, Sam Ou, Kelly Blincoe In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis , pages 1433–1444 , 2023 [BIBTEX] [Abstract] [PDF] [DOI]
Modern software applications rely heavily on the usage of libraries, which provide reusable functionality, to accelerate the development process. As libraries evolve and release new versions, the software systems that depend on those libraries (the clients) should update their dependencies to use these new versions as the new release could, for example, include critical fixes for security vulnerabilities. However, updating is not always a smooth process, as it can result in software failures in the clients if the new version includes breaking changes. Yet, there is little research on how these breaking changes impact the client projects in the wild. To identify if changes between two library versions cause breaking changes at the client end, we perform an empirical study on Java projects built using Maven. For the analysis, we used 18,415 Maven artifacts, which declared 142,355 direct dependencies, of which 71.60% were not up-to-date. We updated these dependencies and found that 11.58% of the dependency updates contain breaking changes that impact the client. We further analyzed these changes in the library which impact the client projects and examine if libraries have adhered to the semantic versioning scheme when introducing breaking changes in their releases. Our results show that changes in transitive dependencies were a major factor in introducing breaking changes during dependency updates and almost half of the detected client impacting breaking changes violate the semantic versioning scheme by introducing breaking changes in non-Major updates.

@inproceedings{jayasuriya2023understandingwild, author = {Jayasuriya, Dhanuska and Terragni, Valerio and Dietrich, Jens and Ou, Sam and Blincoe, Kelly}, pages = {1433--1444}, title = {Understanding Breaking Changes in the Wild}, year = {2023}, doi = {10.1145/3597926.3598147}, booktitle = {Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis} }
SBFT 2023
SBFT Tool Competition 2023 - Java Test Case Generation Track Gunel Jahangirova, Valerio Terragni In Proceedings of the , pages 61–64 , 2023 [BIBTEX] [DOI]
@inproceedings{jahangirova2023sbfttrack, author = {Jahangirova, Gunel and Terragni, Valerio}, pages = {61--64}, title = {SBFT Tool Competition 2023 - Java Test Case Generation Track}, year = {2023}, doi = {10.1109/sbft59156.2023.00025}, conference = {IEEE/ACM International Workshop on Search-Based and Fuzz Testing} }
ICSE 2023
GradeStyle: GitHub-Integrated and Automated Assessment of Java Code Style Callum Iddon, Nasser Giacaman, Valerio Terragni In Proceedings of the IEEE/ACM International Conference on Software Engineering, SEET track , pages 192–197 , 2023 [BIBTEX] [Abstract] [PDF] [Replication Package] [DOI]
Every programming language has its own style conventions and best practices, which help developers to write readable and maintainable code. Learning code style is an essential skill that every professional software engineer should master. As such, students should develop good habits for code style early on, when they start learning how to program. Unfortunately, manually assessing students’ code with timely and detailed feedback is often infeasible, and professional static analysis tools are unsuitable for educational contexts. This paper presents GRADESTYLE, a tool for automatically assessing the code style of Java assignments. GRADESTYLE automatically checks for violations of some of the most important Google Java Style conventions, and Java best practices. Students receive a report with a code style mark, a list of violations, and their source code locations. GRADESTYLE nicely integrates with GitHub and GitHub Classroom, and can be configured to provide continuous feedback every time a student pushes new code. We adopted our tool in a second-year software engineering programming course with 327 students and observed consistent improvements in the code style of their assignments.

@inproceedings{iddon-icse-2023, author = {Iddon, Callum and Giacaman, Nasser and Terragni, Valerio}, title = {{GradeStyle: GitHub-Integrated and Automated Assessment of Java Code Style}}, booktitle = {{IEEE/ACM} International Conference on Software Engineering, SEET track}, year = {2023}, doi = {10.1109/ICSE-SEET58685.2023.00024}, pages = {192--197} }
ACE 2023
Automated Assessment: Experiences From the Trenches Andrew Luxton-Reilly, Ewan Tempero, Nalin Arachchilage, Angela Chang, Paul Denny, Allan Fowler, Nasser Giacaman, Igor Kontorovich, Danielle Lottridge, Sathiamoorthy Manoharan, Shyamli Sindhwani, Paramvir Singh, Ulrich Speidel, Sudeep Stephen, Valerio Terragni, Jacqueline Whalley, Burkhard Wuensche, Xinfeng Ye In Proceedings of the Australasian Computing Education Conference , pages 1–-10 , 2023 [BIBTEX] [Abstract] [PDF] [DOI]
Automated assessment is commonly used across the spectrum of computing courses offered by Tertiary institutions. Such assessment is frequently intended to address the scalability of feedback that is essential for learning, and assessment for accreditation purposes. Although many reviews of automated assessment have been reported, the voices of teachers are not present. In this paper we present a variety of cases that illustrate some of the varied motivations and experiences of teaching using automated assessment.

@inproceedings{luxton-reilly-ace-2023, author = {Luxton-Reilly, Andrew and Tempero, Ewan and Arachchilage, Nalin and Chang, Angela and Denny, Paul and Fowler, Allan and Giacaman, Nasser and Kontorovich, Igor and Lottridge, Danielle and Manoharan, Sathiamoorthy and Sindhwani, Shyamli and Singh, Paramvir and Speidel, Ulrich and Stephen, Sudeep and Terragni, Valerio and Whalley, Jacqueline and Wuensche, Burkhard and Ye, Xinfeng}, title = {{Automated Assessment: Experiences From the Trenches}}, booktitle = {Proceedings of the Australasian Computing Education Conference}, year = {2023}, pages = {1–-10}, doi = {10.1145/3576123.3576124} }
SIGCSE 2023
Evolving a Programming CS2 Course: A Decade-Long Experience Report Nasser Giacaman, Partha Roop, Valerio Terragni In Proceedings of Technical Symposium on Computer Science Education , pages 507–513 , 2023 [BIBTEX] [Abstract] [PDF] [DOI]
Despite instructors’ best efforts in designing and delivering any given course, changes are likely required from time to time. This experience report presents the changes made in a second-year programming course for non-computing engineering majors over a decade’s worth of effort, and the reasons behind those changes. The changes were often reactive—in response to student feedback. However, many other changes were inspired by the desire to trial new interventions in the hope of strengthening the students’ positive experience. In addition to personnel and course content changes, the gradual evolvement included how labs, assignments, and activities were structured and executed. Teaching delivery evolved, along with a number of small-scale interventions that eventually became integral elements of the course. When COVID-19 demanded a sudden shift to online learning, the course was prepared to adapt quickly and successfully. The contributions here come in the form of lessons learned over the past decade: what worked, and what did not. We present the large range of changes and their rationales — that are particularly relevant and applicable to programming courses targeting engineering students where the luxury of pedagogically-friendlier programming languages is not possible

@inproceedings{giacaman-sigcse-2023, author = {Giacaman, Nasser and Roop, Partha and Terragni, Valerio}, title = {{Evolving a Programming CS2 Course: A Decade-Long Experience Report}}, booktitle = {Proceedings of Technical Symposium on Computer Science Education}, year = {2023}, pages = {507--513}, doi = {10.1145/3545945.3569831} }

2022

GECCO 2022
Evolutionary Generation of Metamorphic Relations for Cyber-Physical Systems Jon Ayerdi, Valerio Terragni, Aitor Arrieta, Paolo Tonella, Goiuria Sagardui, Maite Arratibel In Proceedings of the Genetic and Evolutionary Computation Conference , 2022 [BIBTEX] [Abstract] [PDF] [DOI]
A problem when testing Cyber-Physical Systems (CPS) is the difficulty of determining whether a particular system output or be- haviour is correct or not. Metamorphic testing alleviates such a problem by reasoning on the relations expected to hold among multiple executions of the system under test, which are known as Metamorphic Relations (MRs). However, the development of effective MRs is often challenging and requires the involvement of domain experts. This paper summarizes our recent publication: “Generating Metamorphic Relations for Cyber-Physical Systems with Genetic Programming: An Industrial Case Study”, presented at ESEC/FSE 2021. In that publication we presented GAssertMRs, the first technique to automatically generate MRs for CPS, leveraging GP to explore the space of candidate solutions. We evaluated GAssertMRs in an industrial case study, outperforming other baselines.

@inproceedings{ayerdigecco, author = {Ayerdi, Jon and Terragni, Valerio and Arrieta, Aitor and Tonella, Paolo and Sagardui, Goiuria and Arratibel, Maite}, title = {{Evolutionary Generation of Metamorphic Relations for Cyber-Physical Systems}}, booktitle = {Genetic and Evolutionary Computation Conference}, year = {2022}, doi = {10.1145/3520304. 3534077} }
ICPC 2022
The Ineffectiveness of Domain-Specific Word Embedding Models for GUI Test Reuse Farideh Khalili, Ali Mohebbi, Valerio Terragni, Mauro Pezzè, Leonardo Mariani, Abbas Heydarnoori In Proceedings of the 30th International Conference on Program Comprehension , 2022 [BIBTEX] [Abstract] [PDF] [DOI]
Reusing test cases across similar applications can significantly reduce testing effort. Some recent test reuse approaches successfully exploit word embedding models to semantically match GUI events across Android apps. It is a common understanding that word em- bedding models trained on domain-specific corpora perform better on specialized tasks. Our recent study confirms this understanding in the context of Android test reuse. It shows that word embedding models trained with a corpus of the English descriptions of apps in the Google Play Store lead to a better semantic matching of Android GUI events. Motivated by this result, we hypothesize that we can further increase the effectiveness of semantic matching by partitioning the corpus of app descriptions into domain-specific corpora. Our experiments do not confirm our hypothesis. This paper sheds light on this unexpected negative result that contradicts the common understanding.

@inproceedings{khalili-icpc-2022, author = {Khalili, Farideh and Mohebbi, Ali and Terragni, Valerio and Pezzè, Mauro and Mariani, Leonardo and Heydarnoori, Abbas}, title = {{The Ineffectiveness of Domain-Specific Word Embedding Models for GUI Test Reuse}}, booktitle = {30th International Conference on Program Comprehension}, year = {2022}, doi = {10.1145/3524610.3527873} }
SANER 2022
Detect, Fix, and Verify TensorFlow API Misuses Wilson Baker, Michael O’Connor, Seyed Reza Shahamiri, Valerio Terragni In Proceedings of the 29th IEEE International Conference on Software Analysis, Evolution and Reengineering SANER 2022 , 2022 [BIBTEX] [Abstract] [PDF] [Video]
The growing application of DL makes detecting and fixing defective DL programs of paramount importance. Recent studies on DL defects report that TensorFlow API misuses represent a common class of DL defects. However to effectively detect, fix, and verify them remains an understudied problem. This paper presents the TensorFlow API misuses Detector And Fixer (TADAF) technique, which relies on 11 common API misuses patterns and corresponding fixes that we extracted from StackOverflow. TADAF statically analyses a TensorFlow program for identifying matches of any of the 11 patterns. If it finds a match, it automatically generates a fixed version of the program. To verify that the misuse brings a tangible negative effect, TADAF reports functional, accuracy, or efficiency differences when training and testing (with the same data) the original and fixed versions of the program. Our preliminary evaluation on five GitHub projects shows that TADAF detected and fixed all the API misuses.

@inproceedings{baker-saner-2022, author = {Baker, Wilson and O’Connor, Michael and Shahamiri, Seyed Reza and Terragni, Valerio}, title = {Detect, Fix, and Verify TensorFlow API Misuses}, booktitle = {29th IEEE International Conference on Software Analysis, Evolution and Reengineering {SANER} 2022}, year = {2022} }
ACE 2022
GitHub in the Classroom: Lessons Learnt Yu-Cheng Tu, Valerio Terragni, Ewan Tempero, Asma Shakil, Andrew Meads, Nasser Giacaman, Allan Fowler, Kelly Blincoe In Proceedings of the Australasian Computing Education Conference , 2022 [BIBTEX] [Abstract] [PDF] [DOI]
The decision as to whether or not, and how, to use a Version Control System (VCS) in teaching is complex to make. There are a number of use cases for how a VCS can be used in teaching, there are several VCSs, each VCS has a variety of options for how to access them, each has a number of third-party support systems, and all combinations have different benefits, costs, and challenges. At University of Auckland, we have made significant use of Git and related systems (especially GitHub and GitHub Classroom). In this paper, we offer the lessons we have learned from our collective experience. While we by no means cover all of the possibilities, we hope that instructors considering the use of VCSs, in particular Git, will find the lessons we have learned helpful in making their decisions regarding how to use VCS in teaching.

@inproceedings{tu-ace-2022, author = {Tu, Yu-Cheng and Terragni, Valerio and Tempero, Ewan and Shakil, Asma and Meads, Andrew and Giacaman, Nasser and Fowler, Allan and Blincoe, Kelly}, title = {GitHub in the Classroom: Lessons Learnt}, booktitle = {Proceedings of the Australasian Computing Education Conference}, year = {2022}, doi = {10.1145/3511861.3511879} }

2021

GPEM 2021
A semantic genetic programming framework based on dynamic targets Stefano Ruberto, Valerio Terragni, Jason H. Moore Genet. Program. Evolvable Mach. , pages 463–493 , 2021 [BIBTEX] [Abstract] [PDF] [DOI]
Semantic GP is a promising branch of GP that introduces semantic awareness during genetic evolution to improve various aspects of GP. This paper presents a new Semantic GP approach based on Dynamic Target (SGP-DT) that divides the search problem into multiple GP runs. The evolution in each run is guided by a new (dynamic) target based on the residual errors of previous runs. To obtain the final solution, SGP-DT combines the solutions of each run using linear scaling. SGP-DT presents a new methodology to produce the offspring that does not rely on the classic crossover. The synergy between such a methodology and linear scaling yields final solutions with low approximation error and computational cost. We evaluate SGP-DT on eleven well-known data sets and compare with 𝜖-lexicase, a state-ofthe-art evolutionary technique, and seven Machine Learning techniques. SGP-DT achieves small RMSE values, on average 23.19% smaller than the one of e-lexicase. Tuning SGP-DT ’s configuration greatly reduces the computational cost while still obtaining competitive results.

@article{ruberto-gpem-2021, author = {Ruberto, Stefano and Terragni, Valerio and Moore, Jason H.}, title = {A semantic genetic programming framework based on dynamic targets}, journal = {Genet. Program. Evolvable Mach.}, volume = {22}, number = {4}, pages = {463--493}, year = {2021}, doi = {10.1007/s10710-021-09419-3} }
ASE 2021
APIzation: Generating Reusable APIs from StackOverflow Code Snippets Valerio Terragni, Pasquale Salza In Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering , pages 542–554 , 2021 [BIBTEX] [Abstract] [PDF] [Replication Package] [DOI]
Developer forums like StackOverflow have become essential resources to modern software development practices. However, many code snippets lack a well-defined method declaration, and thus they are often incomplete for immediate reuse. Developers must adapt the retrieved code snippets by parameterizing the variables involved and identifying the return value. This activity, which we call APIzation of a code snippet, can be tedious and time-consuming. In this paper, we present APIZATOR to perform APIzations of JAVA code snippets automatically. APIZATOR is grounded by four common patterns that we extracted by studying real APIzations in GitHub. APIZATOR presents a static analysis algorithm that automatically extracts the method parameters and return statements. We evaluated APIZATOR with a ground-truth of 200 APIzations collected from 20 developers. For 113 (56.50 %) and 115 (57.50%) APIzations, APIZATOR and the developers extracted identical parameters and return statements, respectively. For 163 (81.50 %) APIzations, either the parameters or the return statements were identical.

@inproceedings{terragni-ase-2021, author = {Terragni, Valerio and Salza, Pasquale}, title = {{APIzation: Generating Reusable APIs from StackOverflow Code Snippets}}, booktitle = {36th IEEE/ACM International Conference on Automated Software Engineering}, year = {2021}, pages = {542--554}, doi = {10.1109/ASE51524.2021.9678576} }
ESEC/ FSE 2021
Generating Metamorphic Relations for Cyber-Physical Systems with Genetic Programming: An Industrial Case Study Jon Ayerdi, Valerio Terragni, Aitor Arrieta, Paolo Tonella, Goiuria Sagardui, Maite Arratibel In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , pages 1264–1274 , 2021 [BIBTEX] [Abstract] [PDF] [DOI]
One of the major challenges in the verification of complex industrial Cyber-Physical Systems is the difficulty of determining whether a particular system output or behaviour is correct or not, the so- called test oracle problem. Metamorphic testing alleviates the oracle problem by reasoning on the relations that are expected to hold among multiple executions of the system under test, which are known as Metamorphic Relations (MRs). However, the development of effective MRs is often challenging and requires the involvement of domain experts. In this paper, we present a case study aiming at automating this process. To this end, we implemented GAssertMRs, a tool to automatically generate MRs with genetic programming. We assess the cost-effectiveness of this tool in the context of an industrial case study from the elevation domain. Our experimental results show that in most cases GAssertMRs outperforms the other baselines, including manually generated MRs developed with the help of domain experts. We then describe the lessons learned from our experiments and we outline the future work for the adoption of this technique by industrial practitioners.

@inproceedings{gassertmrs, author = {Ayerdi, Jon and Terragni, Valerio and Arrieta, Aitor and Tonella, Paolo and Sagardui, Goiuria and Arratibel, Maite}, title = {{Generating Metamorphic Relations for Cyber-Physical Systems with Genetic Programming: An Industrial Case Study}}, booktitle = {29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering}, year = {2021}, doi = {10.1145/3468264.3473920}, pages = {1264–1274} }
GECCO 2021
Towards Effective GP Multi-Class Classification Based on Dynamic Targets Stefano Ruberto, Valerio Terragni, Jason H. Moore In Proceedings of the Genetic and Evolutionary Computation Conference , pages 812–821 , 2021 Nominated for the Best Paper Award [BIBTEX] [Abstract] [PDF] [DOI]
In the multi-class classification problem GP plays an important role when combined with other non-GP classifiers. However, when GP performs the actual classification (without relying on other classifiers) its classification accuracy is low. This is especially true when the number of classes is high. In this paper, we present DTC, a GP classifier that leverages the effectiveness of the dynamic target approach to evolve a set of discriminant functions (one for each class). Notably, DTC is the first GP classifier that defines the fitness of individuals by using the synergistic combination of linear scaling and the hinge-loss function (commonly used by SVM). Differently, most previous GP classifiers use the number of correct classifications to drive the evolution. We compare DTC with eight state-of-art multi-class classification techniques (e.g., RF, RS, MLP, and SVM) on eight popular datasets. The results show that DTC achieves competitive classification accuracy even with 15 classes, without relying on other classifiers.

@inproceedings{ruberto-gecco-2021, author = {Ruberto, Stefano and Terragni, Valerio and Moore, Jason H.}, title = {Towards Effective GP Multi-Class Classification Based on Dynamic Targets}, doi = {10.1145/3449639.3459324}, year = {2021}, booktitle = {Genetic and Evolutionary Computation Conference}, pages = {812–821} }
GECCO 2021
Improving Assertion Oracles with Evolutionary Computation Valerio Terragni, Gunel Jahangirova, Mauro Pezzè, Paolo Tonella In Proceedings of the Genetic and Evolutionary Computation Conference Companion , pages 45–46 , 2021 [BIBTEX] [Abstract] [PDF] [DOI]
Assertion oracles are executable boolean expressions placed inside a software program that verify the correctness of test executions. A perfect assertion oracle passes (returns true) for all correct executions and fails (returns false) for all incorrect executions. Because designing perfect assertion oracles is difficult, assertions often fail to distinguish between correct and incorrect executions. In other words, they are prone to false positives and false negatives. GAssert is the first technique to automatically improve assertion oracles by reducing false positives and false negatives. Given an assertion oracle and a set of correct and incorrect program states, GAssert employs a novel co-evolutionary algorithm that explores the space of possible assertions to identify one with fewer false positives and false negatives. Our evaluation on 34 Java methods shows that GAssert effectively improves assertion oracles.

@inproceedings{terragni-gecco-2021, author = {Terragni, Valerio and Jahangirova, Gunel and Pezzè, Mauro and Tonella, Paolo}, title = {Improving Assertion Oracles with Evolutionary Computation}, booktitle = {Proceedings of the Genetic and Evolutionary Computation Conference Companion}, year = {2021}, pages = {45–46}, doi = {10.1145/3449726.3462722} }
ISSTA 2021
Semantic Matching of GUI Events for Test Reuse: Are We There Yet? Leonardo Mariani, Ali Mohebbi, Mauro Pezzè, Valerio Terragni (Authors are listed alphabetically) In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis , pages 177–190 , 2021 [BIBTEX] [Abstract] [PDF] [DOI]
GUI testing is an important but expensive activity. Recently, re- search on test reuse approaches for Android applications produced interesting results. Test reuse approaches automatically migrate human-designed GUI tests from a source app to a target app that shares similar functionalities. They achieve this by exploiting semantic similarity among textual information of GUI widgets. Semantic matching of GUI events plays a crucial role in these approaches. In this paper, we present the first empirical study on semantic matching of GUI events. Our study involves 253 configurations of the semantic matching, 337 unique queries, and 8,099 distinct GUI events. We report several key findings that indicate how to improve semantic matching of test reuse approaches, pro- pose SemFinder a novel semantic matching algorithm that outperforms existing solutions, and identify several interesting research directions.

@inproceedings{mariani-issta-2021, author = {Mariani, Leonardo and Mohebbi, Ali and Pezzè, Mauro and Terragni, Valerio}, title = {Semantic Matching of GUI Events for Test Reuse: Are We There Yet?}, booktitle = {Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis}, year = {2021}, pages = {177–190}, doi = {10.1145/3460319.3464827} }
STVR 2021
Statically driven generation of concurrent tests for thread-safe classes Valerio Terragni, Mauro Pezzè Software Testing, Verification and Reliability , pages e1774 , 2021 [BIBTEX] [Abstract] [PDF] [DOI]
Summary Concurrency testing is an important activity to expose concurrency faults in thread-safe classes. A concurrent test for a thread-safe class is a set of method call sequences that exercise the public interface of the class from multiple threads. Automatically generating fault-revealing concurrent tests within an affordable time budget is difficult due to the huge search space of possible concurrent tests. In this paper, we present DepCon+, a novel approach that reduces the search space of concurrent tests by leveraging statically computed dependencies among public methods. DepCon+ exploits the intuition that concurrent tests can expose thread-safety violations that manifest exceptions or deadlocks, only if they exercise some specific method dependencies. DepCon+ provides an efficient way to identify such dependencies by statically analysing the code and relies on the computed dependencies to steer the test generation towards those concurrent tests that exhibit the computed dependencies. We developed a prototype DepCon+ implementation for Java and evaluated the approach on 19 known concurrency faults of thread-safe classes that lead to thread-safety violations of either exception or deadlock type. The results presented in this paper show that DepCon+ is more effective than state-of-the-art approaches in exposing the concurrency faults. The search space pruning of DepCon+ dramatically reduces the search space of possible concurrent tests, without missing any thread-safety violations.

@article{terragni-stvr-2021, author = {Terragni, Valerio and Pezzè, Mauro}, title = {Statically driven generation of concurrent tests for thread-safe classes}, journal = {Software Testing, Verification and Reliability}, volume = {31}, number = {4}, pages = {e1774}, doi = {10.1002/stvr.1774}, year = {2021} }
ICSE 2021
GAssert: A Fully Automated Tool to Improve Assertion Oracles Valerio Terragni, Gunel Jahangirova, Paolo Tonella, Mauro Pezzè In Proceedings of the 43nd IEEE/ACM International Conference on Software Engineering Companion , pages 85-88 , 2021 [BIBTEX] [Abstract] [PDF] [DOI]
This demo presents the implementation and usage details of GASSERT, the first tool to automatically improve assertion oracles. Assertion oracles are executable boolean expressions placed inside the program that should pass (return true) for all correct executions and fail (return false) for all incorrect executions. Because designing perfect assertion oracles is difficult, assertions are prone to both false positives (the assertion fails but should pass) and false negatives (the assertion passes but should fail). Given a Java method containing an assertion oracle to improve, GASSERT returns an improved assertion with fewer false positives and false negatives than the initial assertion. Internally, GASSERT implements a novel co-evolutionary algorithm that explores the space of possible assertions guided by two fitness functions that reward assertions with fewer false positives, fewer false negatives, and smaller size.

@inproceedings{terragni-icse-2021, author = {Terragni, Valerio and Jahangirova, Gunel and Tonella, Paolo and Pezzè, Mauro}, title = {GAssert: A Fully Automated Tool to Improve Assertion Oracles}, booktitle = {43nd {IEEE/ACM} International Conference on Software Engineering Companion}, year = {2021}, pages = {85-88}, doi = {10.1109/ICSE-Companion52605.2021.00042} }
AST 2021
An Evolutionary Approach to Adapt Tests Across Mobile Apps Leonardo Mariani, Mauro Pezzè, Valerio Terragni, Daniele Zuddas (Authors are listed alphabetically) In Proceedings of the IEEE/ACM International Conference on Automation of Software Test (AST) , pages 70-79 , 2021 [BIBTEX] [Abstract] [PDF] [DOI]
Automatic generators of GUI tests often fail to generate semantically relevant test cases, and thus miss important test scenarios. To address this issue, test adaptation techniques can be used to automatically generate semantically meaningful GUI tests from test cases of applications with similar functionalities. In this paper, we present ADAPTDROID, a technique that approaches the test adaptation problem as a search-problem, and uses evolutionary testing to adapt GUI tests (including oracles) across similar Android apps. In our evaluation with 32 popular Android apps, ADAPTDROID successfully adapted semantically relevant test cases in 11 out of 20 cross-app adaptation scenarios.

@inproceedings{mariani-ast-2021, author = {Mariani, Leonardo and Pezzè, Mauro and Terragni, Valerio and Zuddas, Daniele}, title = {An Evolutionary Approach to Adapt Tests Across Mobile Apps}, booktitle = {IEEE/ACM International Conference on Automation of Software Test (AST) }, pages = {70-79}, doi = {10.1109/AST52587.2021.00016}, year = {2021} }

2020

ESEC/ FSE 2020
Evolutionary Improvement of Assertion Oracles Valerio Terragni, Gunel Jahangirova, Paolo Tonella, Mauro Pezzè In Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , pages 1178–1189 , 2020 [BIBTEX] [Abstract] [PDF] [DOI]
Assertion oracles are executable boolean expressions placed inside the program that should pass (return true) for all correct executions and fail (return false) for all incorrect executions. Because designing perfect assertion oracles is difficult, assertions often fail to distinguish between correct and incorrect executions. In other words, they are prone to false positives and false negatives. In this paper, we propose GAssert (Genetic ASSERTion improvement), the first technique to automatically improve assertion oracles. Given an assertion oracle and evidence of false positives and false negatives, GAssert implements a novel co-evolutionary algorithm that explores the space of possible assertions to identify one with fewer false positives and false negatives. Our empirical evaluation on 34 Java methods from 7 different Java code bases shows that GAssert effectively improves assertion oracles. GAssert outperforms two baselines (random and invariant- based oracle improvement), and is comparable with and in some cases even outperformed human-improved assertions.

@inproceedings{terragni-fse-2020, author = {Terragni, Valerio and Jahangirova, Gunel and Tonella, Paolo and Pezzè, Mauro}, title = {Evolutionary Improvement of Assertion Oracles}, booktitle = {28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering}, year = {2020}, pages = {1178–1189}, doi = {10.1145/3368089.3409758} }
ICSE 2020
A Container-Based Infrastructure for Fuzzy-Driven Root Causing of Flaky Tests Valerio Terragni, Pasquale Salza, Filomena Ferrucci In Proceedings of the 42nd IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) , pages 69-72 , 2020 [BIBTEX] [Abstract] [PDF] [DOI]
Intermittent test failures (test flakiness) is common during continuous integration as modern software systems have become inherently non-deterministic. Understanding the root cause of test flakiness is crucial as intermittent test failures might be the result of real non-deterministic defects in the production code, rather than mere errors in the test code. Given a flaky test, existing techniques for root causing test flakiness compare the runtime behavior of its passing and failing executions. They achieve this by repetitively executing the flaky test on an instrumented version of the system under test. This approach has two fundamental limitations: (i) code instrumentation might prevent the manifestation of test flakiness; (ii) when test flakiness is rare passively re-executing a test many times might be inadequate to trigger intermittent test outcomes. To address these limitations, we propose a new idea for root causing test flakiness that actively explores the non-deterministic space without instrumenting code. Our novel idea is to repetitively execute a flaky test, under different execution clusters. Each cluster explores a certain non-deterministic dimension (e.g., concurrency, I/O, and networking) with dedicated software containers and fuzzy- driven resource load generators. The execution cluster that manifests the most balanced (or unbalanced) sets of passing and failing executions is likely to explain the broad type of test flakiness.

@inproceedings{terragni-icse-2020, author = {Terragni, Valerio and Salza, Pasquale and Ferrucci, Filomena}, title = {{A Container-Based Infrastructure for Fuzzy-Driven Root Causing of Flaky Tests}}, booktitle = { 42nd IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER)}, pages = {69-72}, year = {2020}, doi = {10.1145/3377816.3381742} }
ICPC 2020
Measauring Software Testability Modulo Test Quality Valerio Terragni, Pasquale Salza, Mauro Pezzè In Proceedings of the 28th International Conference on Program Comprehension , pages 241-251 , 2020 [BIBTEX] [Abstract] [PDF] [Slides] [Replication Package] [DOI]
Comprehending the degree to which software components support testing is important to accurately schedule testing activities, train developers, and plan effective refactoring actions. Software testability estimates such property by relating code characteristics to the test effort. The main studies of testability reported in the literature investigate the relation between class metrics and test effort in terms of the size and complexity of the associated test suites. They report a moderate correlation of some class metrics to test-effort metrics, but suffer from two main limitations: (i) the results hardly generalize due to the small empirical evidence (datasets with no more than eight software projects); and (ii) mostly ignore the quality of the tests. However, considering the quality of the tests is important. Indeed, a class may have a low test effort because the associated tests are of poor quality, and not because the class is easier to test. In this paper, we propose an approach to measure testability that normalizes the test effort with respect to the test quality, which we quantify in terms of code coverage and mutation score. We present the results of a set of experiments on a dataset of 9,861 Java classes, belonging to 1,186 open source projects, with around 1.5 millions of lines of code overall. The results confirm that normalizing the test effort with respect to the test quality largely improves the correlation between class metrics and the test effort. Better correlations result in better prediction power, and thus better prediction of the test effort.

@inproceedings{terragni-icpc-2020, author = {Terragni, Valerio and Salza, Pasquale and Pezzè, Mauro}, title = {{Measauring Software Testability Modulo Test Quality}}, booktitle = {28th International Conference on Program Comprehension}, year = {2020}, pages = {241-251}, doi = {10.1145/3387904.3389273} }
PPSN 2020
Image Feature Learning with Genetic Programming Stefano Ruberto, Valerio Terragni, Jason H. Moore In Proceedings of the 16th International Conference on Parallel Problem Solving from Nature , pages 63–78 , 2020 [BIBTEX] [Abstract] [PDF] [DOI]
Learning features from raw data is an important topic in machine learning. This paper presents Genetic Program Feature Learner (GPFL), a novel generative GP feature learner for 2D images. GPFL executes multiple GP runs, each run generates a model that focuses on a particular high-level feature of the training images. Then, it combines the models generated by each run into a function that reconstructs the observed images. As a sanity check, we evaluated GPFL on the popular MNIST dataset of handwritten digits, and compared it with the convolutional neural network LeNet5. Our evaluation results show that when considering smaller training sets, GPFL achieves comparable/slightly- better classification accuracy than LeNet5. However, GPFL drastically outperforms LeNet5 when considering noisy images as test sets.

@inproceedings{ruberto-ppsn-2020, author = {Ruberto, Stefano and Terragni, Valerio and Moore, Jason H.}, title = {Image Feature Learning with Genetic Programming}, booktitle = {16th International Conference on Parallel Problem Solving from Nature}, year = {2020}, pages = {63--78}, doi = {10.1007/978-3-030-58115-2\_5} }
GECCO 2020
SGP-DT: Towards Effective Symbolic Regression with a Semantic GP Approach Based on Dynamic Targets Stefano Ruberto, Valerio Terragni, Jason H. Moore In Proceedings of the Genetic and Evolutionary Computation Conference Companion , pages 25-26 , 2020 [BIBTEX] [Abstract] [PDF] [DOI]
Semantic Genetic Programming (SGP) approaches demonstrated remarkable results in different domains. SGP-DT is one of the latest of such approaches. Notably, SGP-DT proposes a dynamic-target approach that combines multiple GP runs without relying on any form of crossover. On eight well-known datasets SGP-DT achieves small RMSE, on average 25% smaller than state-of-the-art approaches.

@inproceedings{ruberto-gecco-2020b, author = {Ruberto, Stefano and Terragni, Valerio and Moore, Jason H.}, title = {SGP-DT: Towards Effective Symbolic Regression with a Semantic GP Approach Based on Dynamic Targets}, booktitle = {Genetic and Evolutionary Computation Conference Companion}, year = {2020}, pages = {25-26}, doi = {10.1145/3377929.3397486} }
GECCO 2020
Image Feature Learning with a Genetic Programming Autoencoder Stefano Ruberto, Valerio Terragni, Jason H. Moore In Proceedings of the Genetic and Evolutionary Computation Conference Companion , pages 245-246 , 2020 [BIBTEX] [Abstract] [PDF] [DOI]
Learning features from raw data is an important topic in machine learning. This paper presents a novel GP approach to learn high- level features from 2D images. It is a generative approach that resembles the concept of an autoencoder. Our approach executes multiple GP runs, each run generates a (partial) model that focuses on a particular high-level feature of the training images. Then, it combines the models generated by each run into a parametric function that reconstructs the observed images. We evaluated our approach on the popular MNIST dataset of 2D images representing handwritten digits. Our evaluation results show that our parametric approach can precisely reconstruct the MNIST hand-written digits.

@inproceedings{ruberto-gecco-2020, author = {Ruberto, Stefano and Terragni, Valerio and Moore, Jason H.}, title = {Image Feature Learning with a Genetic Programming Autoencoder}, booktitle = {Genetic and Evolutionary Computation Conference Companion}, year = {2020}, doi = {10.1145/3377929.3389981}, pages = {245-246} }
EuroGP 2020
SGP-DT: Semantic Genetic Programming Based on Dynamic Targets Stefano Ruberto, Valerio Terragni, Jason H. Moore In Proceedings of the 23rd European Conference on Genetic Programming , pages 167-183 , 2020 Nominated for the Best Paper Award [BIBTEX] [Abstract] [PDF] [Slides] [DOI]
Semantic GP is a promising approach that introduces semantic awareness during genetic evolution. This paper presents a new Semantic GP approach based on Dynamic Target (SGP-DT) that divides the search problem into multiple GP runs. The evolution in each run is guided by a new (dynamic) target based on the residual errors. To obtain the final solution, SGP-DT combines the solutions of each run using linear scaling. SGP-DT presents a new methodology to produce the offspring that does not rely on the classic crossover. The synergy between such a methodology and linear scaling yields to final solutions with low approximation error and computational cost. We evaluate SGP-DT on eight well-known data sets and compare with epsilon-lexicase, a state-of-the-art evolutionary technique. SGP-DT achieves small RMSE values, on average 23.19% smaller than the one of epsilon-lexicase.

@inproceedings{ruberto-eurogp-2020, author = {Ruberto, Stefano and Terragni, Valerio and Moore, Jason H.}, title = {{SGP-DT:} Semantic Genetic Programming Based on Dynamic Targets}, booktitle = {23rd European Conference on Genetic Programming}, pages = {167-183}, year = {2020}, doi = {10.1007/978-3-030-44094-7\_11} }

2019

ICST 2019
Coverage-Driven Test Generation for Thread-Safe Classes via Parallel and Conflict Dependencies Valerio Terragni, Mauro Pezzè, Francesco Adalberto Bianchi In Proceedings of the 12th IEEE Conference on Software Testing, Validation and Verification , pages 264-275 , 2019 IEEE TCSE Distinguished Paper Award [BIBTEX] [Abstract] [PDF] [Slides] [DOI]
Thread-safe classes are common in concurrent object-oriented programs. Testing such classes is important to ensure the reliability of the concurrent programs that rely on them. Recently, researchers have proposed the automated generation of concurrent (multi-threaded) tests to expose concurrency faults in thread-safe classes (thread-safety violations). However, generating fault-revealing concurrent tests within an affordable time-budget is difficult due to the huge search space of possible concurrent tests. In this paper, we present DEPCON, an approach to effectively reduce the search space of concurrent tests by means of both parallel and conflict dependency analyses. DEPCON is based on the intuition that only methods that can both interleave (parallel dependent) and access the same shared memory locations (conflict dependent) can lead to thread-safety violations when concurrently executed. DEPCON implements an efficient static analysis to compute the parallel and conflict dependencies among the methods of a class and uses the computed dependencies to steer the generation of tests towards concurrent tests that exhibit the computed dependencies. We evaluated DEPCON by experimenting with a prototype implementation for Java programs on a set of thread-safe classes with known concurrency faults. The experimental results show that DEPCON is more effective in exposing concurrency faults than state-of-theart techniques.

@inproceedings{terragni-icst-2019, author = {Terragni, Valerio and Pezz{\`{e}}, Mauro and Bianchi, Francesco Adalberto}, title = {Coverage-Driven Test Generation for Thread-Safe Classes via Parallel and Conflict Dependencies}, booktitle = {12th {IEEE} Conference on Software Testing, Validation and Verification}, pages = {264-275}, year = {2019}, doi = {10.1109/ICST.2019.00034} }

2018

ASE 2018
Effectiveness and Challenges in Generating Concurrent Tests for Thread-Safe Classes Valerio Terragni, Mauro Pezzè In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering , pages 64-75 , 2018 [BIBTEX] [Abstract] [PDF] [Slides] [DOI]
Developing correct and efficient concurrent programs is difficult and error-prone, due to the complexity of thread synchronization. Often, developers alleviate such problem by relying on thread-safe classes, which encapsulate most synchronization-related challenges. Thus, testing such classes is crucial to ensure the reliability of the concurrency aspects of programs. Some recent techniques and corresponding tools tackle the problem of testing thread-safe classes by automatically generating concurrent tests. In this paper, we present a comprehensive study of the state-of-the-art techniques and an independent empirical evaluation of the publicly available tools. We conducted the study by executing all tools on the JaConTeBe benchmark that contains 47 well-documented concurrency faults. Our results show that 8 out of 47 faults (17%) were detected by at least one tool. By studying the issues of the tools and the generated tests, we derive insights to guide future research on improving the effectiveness of automated concurrent test generation.

@inproceedings{terragni-ase-2018, author = {Terragni, Valerio and Pezz{\`{e}}, Mauro}, title = {Effectiveness and Challenges in Generating Concurrent Tests for Thread-Safe Classes}, booktitle = {33rd {ACM/IEEE} International Conference on Automated Software Engineering}, pages = {64-75}, doi = {10.1145/3238147.3238224}, year = {2018} }

2017

ESEC/ FSE 2017
Reproducing Concurrency Failures from Crash Stacks Francesco A. Bianchi, Mauro Pezzè, Valerio Terragni (Authors are listed alphabetically) In Proceedings of the 11th Joint Meeting on Foundations of Software Engineering , pages 705-716 , 2017 [BIBTEX] [Abstract] [PDF] [Slides] [Replication Package] [DOI]
Reproducing field failures is the first essential step for understanding, localizing and removing faults. Reproducing concurrency field failures is hard due to the need of synthesizing a test code jointly with a thread interleaving that induce the failure in the presence of limited information from the field. Current techniques for reproducing concurrency failures focus on identifying failure-inducing interleavings, leaving largely open the problem of synthesizing the test code that manifests such interleavings. In this paper, we present ConCrash, a technique to automatically generate test codes that reproduce concurrency failures that violate thread-safety from crash stacks, which commonly summarize the conditions of field failures. ConCrash efficiently explores the huge space of possible test codes to identify a failure-inducing one by using a suitable set of search pruning strategies. Combined with existing techniques for exploring interleavings, ConCrash automatically reproduces a given concurrency failure that violates the thread-safety of a class by identifying both a failure-inducing test code and corresponding interleaving. In the paper, we define the ConCrash approach, present a prototype implementation of ConCrash, and discuss the experimental results that we obtained on a known set of ten field failures that witness the effectiveness of the approach.

@inproceedings{bianchi-fse-2017, author = {Bianchi, Francesco A. and Pezz{\`{e}}, Mauro and Terragni, Valerio}, title = {Reproducing Concurrency Failures from Crash Stacks}, booktitle = {11th Joint Meeting on Foundations of Software Engineering}, pages = {705-716}, year = {2017}, doi = {10.1145/3106237.3106292} }

2016

ICSE 2016
Coverage-driven Test Code Generation for Concurrent Classes Valerio Terragni, Shing-Chi Cheung In Proceedings of the 38th ACM/IEEE International Conference on Software Engineering , pages 1121-1132 , 2016 [BIBTEX] [Abstract] [PDF] [Slides] [DOI]
Previous techniques on concurrency testing have mainly focused on exploring the interleaving space of manually written test code to expose faulty interleavings of shared memory accesses. These techniques assume the availability of failure-inducing tests. In this paper, we present AutoConTest, a coverage-driven approach to generate effective concurrent test code that achieve high interleaving coverage. AutoConTest consists of three components. First, it computes the coverage requirements dynamically and iteratively during sequential test code generation, using a coverage metric that captures the execution context of shared memory accesses. Second, it smartly selects these sequential codes based on the computed result and assembles them for concurrent tests, achieving increased context-sensitive interleaving coverage. Third, it explores the newly covered interleavings. We have implemented AutoConTest as an automated tool and evaluated it using 6 real-world concurrent Java subjects. The results show that AutoConTest is able to generate effective concurrent tests that achieve high interleaving coverage and expose concurrency faults quickly. AutoConTest took less than 65 seconds (including program analysis, test generation and execution) to expose the faults in the program subjects.

@inproceedings{terragni-icse-2016, author = {Terragni, Valerio and Cheung, Shing{-}Chi}, title = {Coverage-driven Test Code Generation for Concurrent Classes}, booktitle = {38th {ACM/IEEE} International Conference on Software Engineering}, pages = {1121-1132}, year = {2016}, doi = {10.1145/2884781.2884876} }
ISSTA 2016
CSNIPPEX: Automated Synthesis of Compilable Code Snippets from Q&A Sites Valerio Terragni, Yepang Liu, Shing-Chi Cheung In Proceedings of the 25th International Symposium on Software Testing and Analysis , pages 118-129 , 2016 [BIBTEX] [Abstract] [PDF] [Slides] [Replication Package] [DOI]
Popular Q&A sites like StackOverflow have collected numerous code snippets. However, many of them do not have complete type information, making them uncompilable and inapplicable to various software engineering tasks. This paper analyzes this problem, and proposes a technique CSNIPPEX to automatically convert code snippets into compilable Java source code files by resolving external dependencies, generating import declarations, and fixing syntactic errors. We implemented CSNIPPEX as a plug-in for Eclipse and evaluated it with 242,175 StackOverflow posts that contain code snippets. CSNIPPEX successfully synthesized compilable Java files for 40,410 of them. It was also able to effectively recover import declarations for each post with a precision of 91.04% in a couple of seconds.

@inproceedings{terragni-issta-2016, author = {Terragni, Valerio and Liu, Yepang and Cheung, Shing{-}Chi}, title = {{CSNIPPEX:} Automated Synthesis of Compilable Code Snippets from Q{\&}A Sites}, booktitle = {25th International Symposium on Software Testing and Analysis}, pages = {118-129}, year = {2016}, doi = {10.1145/2931037.2931058} }
FSE 2016
Understanding and Detecting Wake Lock Misuses for Android Applications Yepang Liu, Chang Xu, Shing-Chi Cheung, Valerio Terragni In Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering , pages 396-409 , 2016 [BIBTEX] [Abstract] [PDF] [Slides] [Replication Package] [DOI]
Wake locks are widely used in Android apps to protect critical computations from being disrupted by device sleeping. Inappropriate use of wake locks often seriously impacts user experience. However, little is known on how wake locks are used in real-world Android apps and the impact of their misuses. To bridge the gap, we conducted a large-scale empirical study on 44,736 commercial and 31 open-source Android apps. By automated program analysis and manual investigation, we observed (1) common program points where wake locks are acquired and released, (2) 13 types of critical computational tasks that are often protected by wake locks, and (3) eight patterns of wake lock misuses that commonly cause functional and non-functional issues, only three of which had been studied by existing work. Based on our findings, we designed a static analysis technique, Elite, to detect two most common patterns of wake lock misuses. Our experiments on real-world subjects showed that Elite is effective and can outperform two state-of-the-art techniques.

@inproceedings{liu-fse-2016, author = {Liu, Yepang and Xu, Chang and Cheung, Shing{-}Chi and Terragni, Valerio}, title = {Understanding and Detecting Wake Lock Misuses for Android Applications}, booktitle = {24th {ACM} {SIGSOFT} International Symposium on Foundations of Software Engineering}, pages = {396-409}, year = {2016}, doi = {10.1145/2950290.2950297} }

2015

MOBILE Soft 2015
Hybrid Mobile Apps in the Google Play Store: An Exploratory Investigation Ivano Malavolta, Stefano Ruberto, Tommaso Soru, Valerio Terragni In Proceedings of the 2nd ACM International Conference on Mobile Software Engineering and Systems , pages 56-59 , 2015 [BIBTEX] [Abstract] [PDF] [DOI]
One of the most intriguing challenges in mobile apps development is its fragmentation with respect to mobile platforms (e.g., Android, Apple iOS, Windows Phone). Large companies like IBM and Adobe and a growing community of developers advocate hybrid mobile apps development as a possible solution to mobile platforms fragmentation. Hybrid mobile apps are consistent across platforms and built on web standards. How hybrid apps are performing in production settings is still highly debated, with limited objective evidence.In this paper, we present the first realistic investigation into mobile hybrid apps through a solid empirical strategy. Our goal is exploratory and we aim at identifying, analysing, and understanding the traits and distinctions of publicly available hybrid mobile apps within their real-life context. The study has been conducted by mining 11,917 free apps and 3,041,315 reviews from the Google Play Store, and analyzing them from both a technical and end users’ perception perspective. The results of this study build an objective and reproducible snapshot about how hybrid mobile development is performing "in the wild" in real projects.

@inproceedings{malavolta-mobilesoft-2015, author = {Malavolta, Ivano and Ruberto, Stefano and Soru, Tommaso and Terragni, Valerio}, title = {Hybrid Mobile Apps in the Google Play Store: An Exploratory Investigation}, booktitle = {2nd {ACM} International Conference on Mobile Software Engineering and Systems}, pages = {56-59}, year = {2015}, doi = {10.1109/MobileSoft.2015.15} }
ICSE 2015
RECONTEST: Effective Regression Testing of Concurrent Programs Valerio Terragni, Shing-Chi Cheung, Charles Zhang In Proceedings of the 37th IEEE/ACM International Conference on Software Engineering , pages 246–256 , 2015 [BIBTEX] [Abstract] [PDF] [Slides] [DOI]
Concurrent programs proliferate as multi-core technologies advance. As a result, the conventional approach that selects a sub-set of test cases for regression testing without considering interleavings is insufficient. In this paper we present RECONTEST to address the problem by selecting the new interleavings that arise due to code changes. These interleavings must be explored in order to uncover regression bugs. RECONTEST efficiently selects new interleavings by first identifying shared memory accesses that are affected by the changes, and then exploring only those problematic interleavings that contain at least one of these accesses. We have implemented RECONTEST as an automated tool and evaluated it using 13 real-world concurrent program subjects. Our results show that RECONTEST can significantly reduce the regression testing cost without missing any faulty interleavings induced by code changes.

@inproceedings{terragni-icse-2015, author = {Terragni, Valerio and Cheung, Shing{-}Chi and Zhang, Charles}, title = {{RECONTEST:} Effective Regression Testing of Concurrent Programs}, booktitle = {37th {IEEE/ACM} International Conference on Software Engineering}, pages = {246--256}, year = {2015}, doi = {10.1109/ICSE.2015.45} }
MS 2015
End Users’ Perception of Hybrid Mobile Apps in the Google Play Store Ivano Malavolta, Stefano Ruberto, Tommaso Soru, Valerio Terragni In Proceedings of the IEEE International Conference on Mobile Services , pages 25-32 , 2015 [BIBTEX] [Abstract] [PDF] [DOI]
Today millions of mobile apps are downloaded and used all over the world. Mobile apps are distributed via different app stores, such as the Google Play Store, the Apple App Store, the Windows Phone Store. One of the most intriguing challenges in mobile apps development is its fragmentation with respect to mobile platforms (e.g., Android, Apple iOS, Windows Phone). Recently, companies like IBM and Adobe and a growing community of developers advocate hybrid mobile apps development as a possible solution to mobile platforms fragmentation. Hybrid mobile apps are consistent across platforms and built on Web standards. In this paper, we present an empirical investigation into mobile hybrid apps. Our goal is to identify and analyse the traits and distinctions of publicly available hybrid mobile apps from end users’ perspective. The study has been conducted by mining 11,917 free apps and 3,041,315 reviews from the Google Play Store, and analyzing them from the end users’ perception perspective. The results of this study build an objective and reproducible snapshot about how hybrid mobile development is performing "in the wild" in real projects, thus establishing a base for future methods and techniques for developing hybrid mobile apps.

@inproceedings{malavolta-ms-2015, author = {Malavolta, Ivano and Ruberto, Stefano and Soru, Tommaso and Terragni, Valerio}, title = {End Users' Perception of Hybrid Mobile Apps in the Google Play Store}, booktitle = {{IEEE} International Conference on Mobile Services}, pages = {25-32}, year = {2015}, doi = {10.1109/MobServ.2015.14} }