Research: Testing and Verification / 研究: テスティング・検証


Verification and Validation (V&V) are crucial activities to develop, deliver and maintain necessary software with enough quality in timely and cost-effective manner. We are conducting researches on (semi-) automated testing and verification techniques, mutation analysis, test coverage measurement and fault (bug) localization mainly targeting Web, Asynchronous JavaScript (Ajax) and Rich Internet Applications (RIA).

Automated Abstraction and Verification of RIA/Ajax Applications

VerificationAlthough asynchronous technologies such as Ajax make Rich Internet Applications (RIAs) responsive, they can result in unexpected behavior due to nondeterministic client-side processing and asynchronous communication. One difficulty in understanding such erroneous behavior lies in the unpredictable contexts of the running system. Dynamic behavior analysis techniques do not help to verify the correctness of certain “blind spots” in the execution path. In this work, we present a static approach for extracting all possible state transitions described in source code from the RIAs. Our approach is based on the assumption that user, server and self interactions with the RIAs can change the states of the application. Our method consists of four steps: 1) using given specifications of Ajax relevant to interactions as rules, 2) creating a call graph, annotating interactions on it and extracting interaction controls, 3) abstracting the call graph to extract relationships among the interactions, and 4) refining the relationships with the interaction controls. By extracting the state machines of test scenarios of the correct and wrong behavior, it can help developers to pinpoint the statements in the source code that lead to the erroneous behavior. Our approach has been evaluated against a few experimental cases and we conclude that it can extract comprehensible state machines in a reasonable time.

When developing asynchronous JavaScript and XML (Ajax) applications, developers implement Ajax design patterns for increasing the usability of the applications. However, unpredictable contexts of running applications might conceal faults that will break the design patterns, which decreases usability. We propose a support tool called JSVerifier that automatically verifies interaction invariants; the applications handle
their interactions in invariant occurrence and order. We also present a selective set of interaction invariants derived from Ajax design patterns, as input. If the application behavior breaks the design patterns, JSVerifier automatically outputs faulty execution paths for debugging. The results of our case studies show that JSVerifier can verify the interaction invariants in a feasible amount of time, and we conclude that it can help developers increase the usability of Ajax applications.

  • Yuta Maezawa, Hironori Washizaki and Shinichi Honiden, “Extracting Interaction-Based Stateful Behavior in Rich Internet Applications”, In proceesings of the 16th European Conference on Software Maintenance and Reengineering (CSMR ’12), 2012. (PDF)
  • 前澤 悠太, 鷲崎 弘宜, 本位田 真一, “インタラクションに着目したRich Internet Applications の欠陥発見の支援”, 情報処理学会論文誌, 特集「ソフトウェア工学」, Vol.54(No.2), 2013年02月 (PDF)
  • Yuta Maezawa, Hironori Washizaki, Yoshinori Tanabe and Shinichi Honiden, “Automated Verification of Pattern-based Interaction Invariants in Ajax Applications,” Proceedings of 28th IEEE/ACM International Conference on Automated Software Engineering (ASE2013), pp.XX-YY, Silicon Valley, California, November 11-15, 2013. (PDF)
  • Yuta Maezawa, Kazuki Nishiura, Hironori Washizaki, Shinichi Honiden, “Validating Ajax Applications Using a Delay-Based Mutation Technique,” Proceedings of the 29th IEEE/ACM International Conference on Automated Software Engineering (ASE 2014), Västerås, Sweden, September 15-19, 2014. (to appear)

Mutation Analysis for Ajax Application Testing

MutationWhen developers test modern web applications that use JavaScript, challenging issues lie in their event-driven, asynchronous, and dynamic features. Many researchers have assessed the adequacy of test cases with code coverage criteria; however, in this paper, we show that the code coverage-based approach possibly misses some faults in the applications. We propose a mutation analysis approach for estimating the fault-finding capability of test cases. We assume that developers can find overlooked fault instances and improve the test cases with the estimated capability. To create a set of faulty programs, i.e., mutants, we classify the JavaScript features in web applications and then define a comprehensive set of mutation operators. We conducted a case study on a real-world application and found that our approach supported the improvement of test cases to expose hand-seeded faults by an extra ten percent.

  • Kazuki Nishiura, Yuta Maezawa, Hironori Washizaki, Shinichi Honiden, “Mutation Analysis for JavaScript Web Applications Testing,” Proceedings of 24th International Conference on Software Engineering and Knowledge Engineering (SEKE 2013), pp.159-165, Hyatt Harborside at Logan Int’l Airport, Boston, June 27-29, 2013.(PDF)

Gray-Box Testing for Web Applications and Page Object Design Pattern

POGenWeb applications are complex; they consist of many subsystems and run on various browsers and platforms. This makes it difficult to conduct adequate integration testing to detect faults in the connections between subsystems or in the speci c environments. Therefore, establishing an efficient integration testing method with the proper test adequacy criteria and tools is an important issue. In this paper, we propose a new test coverage called template variable coverage. We also propose a novel technique for generating skeleton test code that includes accessor methods and improves the template variable coverage criterion, using a tool that we developed called POGen. Our experiments show that template variable coverage correlates highly with the capability to detect faults, and that POGen can reduce testing costs.

  • Kazunori Sakamoto, Tomohiro Kaizu, Daigo Hamura, Hironori Washizaki and Yoshiaki Fukazawa, “POGen: A Test Code Generator Based on Template Variable Coverage in Gray-Box Integration Testing for Web Applications,” 16th International Conference on Fundamental Approaches to Software Engineering (FASE 2013), March 16-24, 2013.(PDF)
  • 坂本一憲,海津智宏,波村大悟,鷲崎弘宜,深澤良彰, “Webアプリの動的部分に着目したグレーボックス統合テストとテンプレート変数カバレッジの提案”, 第19回 ソフトウェア工学の基礎ワークショップ FOSE 2012 in 湯布院, 2012. (PDF)
  • PageObjectGenerator (POGen): Generates skeleton test code for WebDriver from html template files
  • 青井翔平, 坂本一憲,鷲崎弘宜,深澤良彰, “DePoT: Webアプリケーションテストにおけるテストコード自動生成テスティングフレームワーク”, 第19回 ソフトウェア工学の基礎ワークショップ FOSE 2012 in 湯布院, 2012. (PDF)

Test Coverage Measurement and Spectrum-Based Fault (Bug) Localization

CoverageTest coverage is an important indicator of whether software has been sufficiently tested. However, there are several problems with the existing measurement tools for test coverage, such as their cost of development and maintenance, inconsistency, and inflexibility in measurement. We propose a consistent and flexible measurement framework for test coverage that we call the Open Code Coverage Framework (OCCF). It supports multiple programming languages by extracting the commonalities from multiple programming languages using an abstract syntax tree to help in the development of the measurement tools for the test coverage of new programming languages. OCCF allows users to add programming language support independently of the test-coverage-criteria and also to add test-coverage-criteria support independently of programming languages in order to take consistent measurements in each programming language. Moreover, OCCF provides two methods for changin the measurement range and elements using XPath and adding user code in order to make more flexible measurements. We implemented a sample tool for C, Java, and Python using OCCF. OCCF can measure four test-coverage-criteria. We also confirmed that OCCF can support C#, Ruby, JavaScript, and Lua. Moreover, we reduced the lines of code (LOCs) required to implement measurement tools for test coverage by approximately 90% and the time to implement a new test-coverage-criterion by over 80% in an experiment that compared OCCF with the conventional non-framework-based tools.

Bug localization technique is effectual approach for locating software bugs using the number of failing and passing test cases that execute the statement. However, existing bug localization tools only support one type of programming languages and one type of suspiciousness metrics. In this paper, We presents customizable bug localization framework supporting multiple programming languages and multiple suspiciousness metrics. By apply each suspiciousness metrics to multiple programming languages and comparing, we show the availability of our proposed.

  • Kazunori Sakamoto, Fuyuki Ishikawa, Hironori Washizaki, and Yoshiaki Fukazawa, “Open Code Coverage Framework: A Framework for Consistent, Flexible and Complete Measurement of Test Coverage Supporting Multiple Programming Languages,” IEICE Transactions on Information and Systems, Vol.E94-D, No.12, pp.2418-2430, 2011.(PDF)
  • Open Code Coverage Framework: Test-Coverage Measurement Framework supporting multiple programming languages (複数のプログラミング言語に対応したテストカバレッジ測定フレームワーク)
  • 下條 清史, 坂本 一憲, 鷲崎 弘宜, 深澤 良彰, “複数言語対応かつ算出式変更可能なバグローカリゼーションフレームワークの提案”, 電子情報通信学会ソフトウェアサイエンス研究会3月 (IEICE-SIGSS), 2013. (PDF, SS13_Shimojo)
  • Susumu Tokumoto, Kazunori Sakamoto, Kiyofumi Shimojo, Tadahiro Uehara and Hironori Washizaki, “Semi-automatic Incompatibility Localization for Re-engineered Industrial Software,” Proceedings of the 7th IEEE International Conference on Software Testing, Verification and Validation (ICST 2014), pp.XX-YY, Cleveland, March 31-April 4, 2014. (to appear)

Acceptance Testing utilizing UseCases

In software development using use cases, such as use-case-driven object-oriented development, test scenarios for the acceptance test can be built from use cases. However, the manual listing of execution flows of complex use cases sometimes results in incomplete coverage of the possible execution flows, particularly if the relationships between use cases are complicated. Moreover, the lack of a widely accepted coverage definition for the acceptance test results in the ambiguous judgment of acceptance test completion. We propose a definition of acceptance test coverage using use cases and an automated
generation procedure for test scenarios and skeleton codes under a specified condition for coverage while automatically identifying the execution flows of use cases. Our technique can reduce the incomplete coverage of execution flows of use cases and create objective standards for judging acceptance test completion. Moreover, we expect an improvement of the efficiency of acceptance tests because of the automated generation procedure for test scenarios and skeleton codes.

  • Susumu Kariyuki, Atsuto Kubo, Hironori Washizaki and Yoshiaki Fukazawa, “Acceptance-Test based on the Inter-Use Case Relations,” The 5th World Congress for Software Quality (5WCSQ), 2011.(PDF)

コメントを残す

メールアドレスが公開されることはありません。 * が付いている欄は必須項目です

次のHTML タグと属性が使えます: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>