Case generation by constraints combinatorial testing

BAI Xiaoying; HUANG Jun

doi:10.16511/j.cnki.qhdxxb.2017.26.001

PDF(1267 KB)

Journal of Tsinghua University(Science and Technology) ›› 2017, Vol. 57 ›› Issue (3) : 225-233. DOI: 10.16511/j.cnki.qhdxxb.2017.26.001

COMPUTER SCIENCE AND TECHNOLOGY

Case generation by constraints combinatorial testing

{{article.zuoZhe_EN}}

Author information +

History +

Abstract

The inputs parameters to software systems usually have various constraints correlated to each other. However, the input often violates the required constraints and constraint correlations. When there are many parameters with a large input domain, combinatorial testing can be used to reduce the test cost while ensuring test coverage. This paper defines a test adequacy criterion for constraint coverage using constrained combinatorial testing to detect conflicts caused by violating the constraints and constraint combinations. The system was applied to a member registration service for an online payment platform. Three typical combinatorial algorithms, OA (orthogonal array), IPO (in parameter order) and OFOT (one factor one time), were compared in forward their speeds, defect detection efficiency, and test case size using different configurations and parameter settings. The results showed that the OA algorithm was fast, produced fewer, more stable test cases with a reasonable defect detection efficiency, so this algorithm is good for iterative optimizations.

Key words

software testing / combinatorial testing / constraints combination / test generation

Cite this article

EndNote

Ris (Procite)

Bibtex

Download Citations

BAI Xiaoying, HUANG Jun. Case generation by constraints combinatorial testing[J]. Journal of Tsinghua University(Science and Technology). 2017, 57(3): 225-233 https://doi.org/10.16511/j.cnki.qhdxxb.2017.26.001

References

"[1] Nie C, Leung H. A survey of combinatorial testing[J].ACM Computing Surveys (CSUR), 2011, 43(2):33-63. [2] Petke J. Constraints:The future of combinatorial in-teraction testing[C]//Proceedings of the Eighth International Workshop on Search-Based Software Testing. Piscataway, NJ, USA:IEEE Press, 2015:17-18. [3] Kilani Y, Alsarhan A, Bsoul M, et al. Soft Computing Applications[M]. Timisoara:Springer International Publishing, 2016. [4] Mercan H, Yilmaz C. Towards unified combinatorial interaction testing[J]. arXiv Preprint arXiv, 2016:1602. 06599. [5] Hnich B, Prestwich S D, Selensky E, et al. Constraint models for the covering test problem[J]. Constraints, 2006, 11(2-3):199-219. [6] Cohen M B, Colbourn C J, Ling A C H. Augmenting simulated annealing to build interaction test suites[C]//14th International Symposium on Software Reliability Engineering 2003. Denver, CO, USA:IEEE Press, 2003:394-405. [7] Cohen M B, Dwyer M B, Shi J. Exploiting constraint solving history to construct interaction test suites[C]//Testing:Academic and Industrial Conference Practice and Research Techniques-Mutation, 2007. Washington DC, USA:IEEE Press, 2007:121-132. [8] Cohen M B, Dwyer M B, Shi J. Constructing interaction test suites for highly configurable systems in the presence of constraints:A greedy approach[J]. IEEE Transactions on Software Engineering, 2008, 34(5):633-650. [9] Czerwonka J. Pairwise testing in the real world:Practical extensions to test-case scenarios[C]//Proceedings of 24th Pacific Northwest Software Quality Conference. Portland, Oregon, USA, 2006:419-430. [10] Tung Y W, Aldiwan W S. Automating test case gen-eration for the new generation mission software system[C]//2000 IEEE Aerospace Conference Proceedings. Big Sky, MT, USA:IEEE Press, 2000:431-437. [11] Cohen M B, Dwyer M B, Shi J. Interaction testing of highly configurable systems in the presence of constraints[C]//Proceedings of the 2007 International Symposium on Software Testing and Analysis. London, UK:ACM, 2007:129-139. [12] Cherif A, Imine A. A constraint-based approach for generating transformation patterns[J]. arXiv Preprint arXiv, 2015:1512.07684. [13] Choi Y, Byun T. Constraint-based test generation for automotive operating systems[J]. Software & Systems Modeling, 2015:1-18. [14] Bryce R C, Colbourn C J. Prioritized interaction testing for pairwise coverage with seeding and constraints[J]. Information and Software Technology, 2006, 48(10):960-970. [15] Banbara M, Matsunaka H, Tamura N, et al. Generating combinatorial test cases by efficient SAT encodings suitable for CDCL SAT solvers[C]//Logic for Programming, Artificial Intelligence, and Reasoning. Dakar, Senegal:Springer Berlin Heidelberg, 2010:112-126. [16] 侯可佳, 白晓颖, 周立柱. 一种基于多约束组合的多租户系统配置测试技术[J].计算机学报,2016, 39(2):237-252.HOU Kejia, BAI Xiaoying, ZHOU Lizhu. Web service test data generation using interface semantic contract[J]. Chinese Journal of Computers, 2016, 39(2):237-252. (in Chinese) [17] Lei Y, Tai K C. In-parameter-order:A test generation strategy for pairwise testing[C]//High Assurance Systems Engineering Symposium, 1998. Proceedings. Third IEEE International. Washington DC, USA:IEEE Press, 1998:254-261. [18] Lei Y, Kacker R, Kuhn D R, et al. IPOG/IPOG-D:Efficient test generation for multiway combinatorial testing[J]. Software Testing, Verification and Reliability, 2008, 18(3):125-148."