Invitation

Presentations are now available at the program page. Pictures are also available here.

The CT 2012 workshop is a one-day event at ICST 2012, focusing on combinatorial testing. The workshop welcomes academic research submissions, as well as industrial experience reports.

Combinatorial Testing (CT) is a widely applicable generic methodology and technology for software verification and validation. In a combinatorial test plan, all interactions between parameters up to a certain level are covered. For example, in pairwise testing, for every pair of parameters, every pair of values will appear at least once. Studies show that CT is more efficient and effective than random testing and expert test selection methods.

CT has gained significant interest in recent years, both in research and in practice. However, many issues still remain unresolved, and much research is still needed in the field. For example, while pairwise testing is a well recognized and popular test planning method, investigations of actual failures in a number of software and systems convincingly show that pairwise testing may not be sufficient so high strength CT (i.e., t-way for t>2) may be needed.

In addition, the combinatorial test suites need to exclude invalid combinations of test values, which limits the degrees of freedom the algorithms have, thus complicating the problem. Moreover, modeling languages and tools for easily capturing the input test space are also required for real-life applicability of CT. Other obstacles for wide acceptance of CT in industry are the gap between the generated test plans and executable tests, and the difficulty in determining expected results for the generated tests. Finally, empirical studies on CT, as well as thorough comparison with other methods are also required.

In this workshop, we plan to bring together researchers actively working on combinatorial testing, and create a productive and creative environment for sharing and collaboration. Researchers attending the workshop will have an opportunity to publish their work in a dedicated venue, create new collaborations and take active part in the growing community of researchers working in the field.

The workshop will also be a meeting place between academia and industry, thus uniting academic excellence and industrial experience and needs. This will allow participants from academia to learn about the industrial experience in practical application of CT to real-life testing problems, and together with the colleagues from industry identify the pain points that are obstacles to wider application of CT, and should be addressed in future research. Industrial participants will have an opportunity to meet the leading scientists in the field, and hear about the latest advances and innovations.

Topics of interest include, but are not limited to:

• Combinatorial testing workflow
  • Modeling the input space for CT -- traditionally, the input to CT algorithms has been a set of parameters, respective values, and constraints on value combinations. This input should capture correctly the points of variability in testing the system. While this input is clearly crucial for effectively using CT, very little research has been carried out on how to simplify it for the user, and help her to easily achieve a correct representation of the system under test.
  • Efficient algorithms to generate test suites with small size for t-way testing for t > 2, involving support of constrains on combinations that are possible.
  • Determination of expected system behavior for each test case - while the test cases are automatically generated by the CT algorithm, determining the expected system behavior is currently usually a manual task.
  • Executing CT test suites -- the result of CT algorithms is a list of tests, where a test is represented by a value to each parameter. There is still significant effort in transforming this representation into actual tests that a tester or testing tool can execute.
  • Combinatorial testing based fault localization.
  • Implementation of CT in existing testing infrastructures.
  • Handling changes in test requirements -- current CT methods focus on one-time generation of a test plan from a given set of requirements. However, test requirements are almost never static, and tend to change between different releases and versions. On the other hand, generating a new set of tests for each release is not practical either.
• Applicability of combinatorial testing
  • Comparison and combination of CT with other dynamic verification methods.
  • Investigation of historical records of failures to determine the kind of CT which may have detected underlying faults.
  • Empirical studies and feedback from practical applications of CT.
  • Combinatorial testing for concurrent and real-time systems.
  • CT for testing cloud computing systems and use of combinatorial methods in cloud architecture.
  • Application of CT in other domains, e.g., study of gene regulation and other biotechnology applications.
• Combinatorial and complementing methods
  • Combinatorial analysis of existing test suites -- analyze a test suite not generated by a CT algorithm in light of a combinatorial test space.
  • Test plan reduction and completeness -- both statically, and under changing test requirements.
  • CT and coverage metrics -- combining the two, and studying the relation between them.

Accepted papers will be published as part of the IEEE ICST 2012 proceedings.

• Itai Segall, IBM Haifa Research Labs
• Richard Kuhn, National Institute of Standards and Technology (NIST)
• Raghu Kacker, National Institute of Standards and Technology (NIST)
• Yu (Jeff) Lei, University of Texas at Arlington