Back

Home Our Services Programming Tools Business Unit Compilers Evaluation of standard libraries

Evaluation of standard libraries

• Requirements
One of our Japanese customers required evaluation of standard libraries. The customer wanted to introduce a third party library to their compiler tool set. Before doing that, they wanted to evaluate the library. The objective of this evaluation is to find out the practical problems in using the standard library with their compiler tool set.

The evaluation requirements were the following:


• Evaluate accuracy of math routines
• Evaluate execution speed of library routines

The evaluation of libraries has to be done for 6 different architectures.
• Challenges
• Deciding the test inputs for the purpose of evaluation
• Presentation of evaluation results


• Solution
• Environment

OS	Windows
Language/Platform	Assembly language for the target architecture.
Tools	IDE for the respective architectures.

• Description

On going through the requirements and by way of discussion with the customers, the following tasks were identified to evaluate the library:



• Creation of test input specifications for 363 library functions
• Creation of test programs for 363 library functions.
• Testing of library functions with test programs

The team first defined the task flow for evaluation of a library routine. Then estimation for the evaluation was done by evaluating a small number of test cases and then calculating the average time taken for evaluation of one library routine.

It will not be sufficient if we have just one test input for each of the library functions. For some library routines, based on the input values, the library routine could take different paths. So we will have to provide a range of input values for each library routine. We referred the implementation to decide the range of values. Profiling was done to measure the execution cycles for each of the routines for different test inputs.

Most of the project tasks had to be done manually and could not be automated. Since we had to perform evaluation for 363 routines; each with range of inputs for 6 different architectures, it eventually became a tedious and repetitive task. To review the results the only way was to run the test again and see if same results are obtained. So the reviews took more time.

The most challenging task was to figure out how to present the result of this evaluation meaningfully. Considering the nature of the task, if the results are not presented meaningfully, it is difficult to infer anything from the results. Based on a series of discussions with the customer we decided on a template for updating the results. The evaluation results were organized such that it can be interpreted unambiguously. Summary graphs were provided for the evaluation results for ease of understanding.

The evaluation was completed in the planned schedule and the result of evaluation was useful for the customer.

Back