Resources

Sign in

Don't have an account?

Signing up is free and easy
Home -> Our Services -> Programming Tools BU -> C Compilers -> Validation of a compiler for 32-bit MCU

Validation of a compiler for 32-bit MCU

Requirements

One of our Japanese customers required validation their 'C' compiler. The compiler to be validated had already gone through various stages of verification and validation and was already in the market. The purpose of this validation was to have an additional stage of validation to ascertain the quality of the compiler.

The validation had to be completed and results have to be reported in about 2 month's period. The acceptance criteria were that 5 critical defects had to be identified and reported to the customer. Critical defects are defined as those for which compiler produced an invalid output without giving a warning or error message (like invalid object code).

Challenges

  • Automation of testing
  • Planning the testing tasks for the given time-frame
  • Reorganizing the tests to be run based on defects identified

Solution

  • Environment
    OS
    		Windows
    Platform
    • Acme Quality Validation System (AQVS) - a proprietary compiler regression test system
    Test suites Selected from the following:
    • acmet test base
    • GCC test cases
    • Actual embedded system applications developed at acmet
    • Standard test suites
    Test base
    • Standard test suites like Perennial, and Plum Hall
    • Acme generated test vectors
    • GCC test vectors
  • Description

    The first task was to automate the testing using the customer's development tools. acmet had a tool called acmet Compiler Quality Validation System (AQVS), which can be used to automate compiler regression tests. Within the first week of the project, AQVS was customized to use customer's development tools and regression testing was started.

    In 2 months there is no way to running all the compiler tests that were available. So first an estimate was made on how many tests could be run within 75% of the given time frame while 25% could be used for defect analysis, reporting and finding more defects based on an identified defect pattern. The priority of running the tests was decided based on which tests have more chances of catching defects. A compiler which is already in market would have cleared all the standard test suites. So it was given the lowest priority.

    During the testing phase whenever a defect is identified it was analyzed and the defect pattern was identified. Then test cases were generated around the identified defect pattern to identify similar defects. Once a defect pattern is identified for a defect, a small test program that reproduces the defect pattern was created. The test program was well documented so that the report could easily be understood by the customer.

    After a defect is identified, the known defects / issues of the compiler under test were checked from the website. If the defect was one of the known defects / issues, then those defects was not processed further.

    The defect reports that were prepared were fairly detailed and provided a complete description of the reported defect. For defects where the compiler output was wrong (like invalid object code), the failure was analyzed and the exact location of failure was identified. Our hypothesis for the reason for failure was also provided. This was possible because of the compiler domain knowledge that acmet possessed.

    All the planned tests were completed within the given time-frame. At the end of the project identified 18 defects were identified in the compiler under test; out of which 11 were critical defects. The customer was happy with the effectiveness of our tests, with our analysis of the defects and the defect reports that were raised.