Benchmarking optimization software with performance profiles.

Journal Article · Tue Jan 01 04:00:00 EST 2002 · Math. Program.

DOI:https://doi.org/10.1007/s101070100263· OSTI ID:943229

Dolan, E D; More, J J

The benchmarking of optimization software has recently gained considerable visibility. Hans Mittlemann's work on a variety of optimization software has frequently uncovered deficiencies in the software and has generally led to software improvements. Although Mittelmann's efforts have gained the most notice, other researchers have been concerned with the evaluation and performance of optimization codes. Recent examples are cited. The interpretation and analysis of the data generated by the benchmarking process are the main technical issues addressed in this paper. Most benchmarking efforts involve tables displaying the performance of each solver on each problem for a set of metrics such as CPU time, number of function evaluations, or iteration counts for algorithms where an iteration implies a comparable amount of work. Failure to display such tables for a small test set would be a gross omission, but they tend to be overwhelming for large test sets. In all cases, the interpretation of the results from these tables is often a source of disagreement. The quantities of data that result from benchmarking with large test sets have spurred researchers to try various tools for analyzing the data. The solver's average or cumulative total for each performance metric over all problems is sometimes used to evaluate performance. As a result, a small number of the most difficult problems can tend to dominate these results, and researchers must take pains to give additional information. In this paper, the authors propose performance profiles -- distribution functions for a performance metric -- as a tool for benchmarking and comparing optimization software. They show that performance profiles combine the best features of other tools for performance evaluation.

• Research Organization: Argonne National Laboratory (ANL)
• Sponsoring Organization: SC; NSF
• DOE Contract Number: AC02-06CH11357
• OSTI ID: 943229
• Report Number(s): ANL/MCS/JA-38499
• Journal Information: Math. Program., Journal Name: Math. Program. Journal Issue: 2 ; Jan. 2002 Vol. 91; ISSN 0025-5610; ISSN MHPGA4
• Country of Publication: United States
• Language: ENGLISH

Related Subjects

97 MATHEMATICS AND COMPUTING
99 GENERAL AND MISCELLANEOUS
ALGORITHMS
BENCHMARKS
COMPUTER CODES
DATA
DATA ANALYSIS
DISTRIBUTION FUNCTIONS
EVALUATION
METRICS
OPTIMIZATION
PERFORMANCE