TACT: Tools for Automated and Cost-effective Testing - overview

Our research in software testing encompasses a variety of topics, such as test automation, test-case generation, test-suite reduction, test repair, and regression testing.

Test Automation

In industrial practice, functional testing of enterprise applications starts with the identification of test scenarios from requirements and use cases. These scenarios are translated to manual test cases. A manual test case is a sequence of test steps written in natural language—each test step is an instruction for the tester to perform an action on the application user interface or verify some visible state of the user interface. Because tests are run repeatedly, they go through an automation process, in which they are converted to automated test scripts (or programs) that perform the test steps mechanically.

Conventional test-automation techniques, such as record-replay and keyword-driven automation, can be time-consuming, can require specialized skills, and can produce fragile scripts. To address these limitations, we have developed a technique and a tool, called ATA, for automating the test-automation task. The input to our technique is a sequence of test steps written in natural language, and the output is a test script that can drive the application without human intervention. Our key insight is to deal with the ambiguities in natural-language input by using exploration over a space of possible scripts to arrive at a desired one. This technology promises to bring tremendous productivity benefits in the area of testing of web applications; initial results have shown almost a two-fold gain in productivity.

More generally, our research on test automation investigates techniques for generating change-resilient test scripts, automatically patching test scripts across platforms, applications variants, and application versions, and improving the execution efficiency of GUI test suites.

Related Publications

  • Efficient and Flexible GUI Test Execution via Test Merging. P Devaki, S Thummalapenta, N Singhania, S Sinha. Proceedings of the International Symposium on Software Testing and Analysis (ISSTA), July 2013.
  • Efficient and Change-Resilient Test Automation: An Industrial Case Study. S Thummalapenta, P Devaki, S Sinha, S Chandra, S Gnanasundaram, D Nagaraj, S Sathishkumar. Proceedings of the 35th International Conference on Software Engineering (ICSE) Software Engineering in Practice Track, May 2013.
  • Automating Test Automation. S Thummalapenta, S Sinha, N Singhania, S Chandra. Proceedings of the 34th International Conference on Software Engineering (ICSE), May 2012.
  • Efficiently Scripting Change-Resilient Tests. N Singhania, P Devaki, S Sinha, S Chandra, A K Das and S Mangipudi. Proceedings of the 20th ACM SIGSOFT International Symposium on the Foundations of Software Engineering (FSE) Research Tool Demonstrations Track, November 2012.

Test-Case Generation

Functional specifications of enterprise applications are often given in the form of business rules. Business rules describe business logic, access control, or even navigational properties of an application's user interface. Therefore, business rules are a natural starting point for the design of functional tests.

The goal of our research is to generate executable tests automatically that drive an application along paths along which business rules apply (are "covered"), so that the application's conformance to relevant business rules can be validated. Test generation requires (1) the creation of adequate test data (to ensure that business rules are well-tested) and (2) the synthesis of flow through the application user interface to exercise a rule. To attain the goal, our research develops modeling techniques for capturing business rules in a formal representation that is amenable to automated analysis and that can drive the generation of adequate test data and test flows. We are also developing directed search techniques for synthesizing flows through the network of interconnected pages of web applications that cover business rules.

Related Publications

  • Guided Test Generation for Web Applications. S Thummalapenta, K V Lakshmi, S Sinha, N Sinha, S Chandra. Proceedings of the 35th International Conference on Software Engineering (ICSE), May 2013.

Test Repair

Test suites, once created, rarely remain static. Just like the application they are testing, they evolve throughout their lifetime. Test obsolescence is probably the most known reason for test-suite evolution---test cases cease to work because of changes in the code and must be suitably repaired. Repairing existing test cases manually, however, can be time consuming, especially for large test suites, which has motivated the recent development of automated test-repair techniques.

To develop effective repair techniques that are applicable in real-world scenarios, a fundamental prerequisite is a thorough understanding of how test cases evolve in practice. Without such knowledge, we risk to develop techniques that may not be widely applicable. In this research, we have developed a technique and a tool for studying test-suite evolution. Using the tool, we have conducted an extensive empirical study of how test suites evolve.

Related Publications

  • TestEvol: A Tool for Analyzing Test-Suite Evolution. L S Pinto, S Sinha, A Orso. Proceedings of the 35th International Conference on Software Engineering (ICSE), Tool Demonstrations Track, May 2013.
  • Understanding Myths and Realities of Test-Suite Evolution. L Sales Pinto, S Sinha, and A Orso. Proceedings of the 20th ACM SIGSOFT International Symposium on the Foundations of Software Engineering (FSE), November 2012.

Regression Testing

Modern software systems exhibit characteristics, such as heterogeneity and environment dependence, which complicate regression testing. As examples of heterogeneity, the systems can be implemented using multiple languages and technologies, and can be highly dynamic, involving runtime generation of code and other artifacts. As examples of environment dependence, the systems can interact with configuration files and databases. Existing regression-testing techniques are suitable for homogeneous and static applications, and also ignore changes in the environment that can affect application behavior. Our research develops effective regression test-selection techniques that can be applied to complex, heterogeneous, and environment-dependent software systems for which existing techniques are ineffective or completely inapplicable.

Related Publications

  • Regression Testing in the Presence of Non-code Changes. A Nanda, S Mani, S Sinha, M J Harrold, A Orso. Proceedings of the 4th International Conference on Software Testing, Verification and Validation (ICST), March 2011.