A methodological proposal to eliminate ambiguities in the comparison of vehicle routing problem solving techniques

In the field of vehicle routing problems it is very common to use benchmarks (sets of problem instances) to evaluate new solving techniques or algorithms. The purpose of these benchmarks is to compare the techniques based on the results or solutions obtained. Typically, the benchmarks include the values of optimal solutions (if they have been obtained) or values of the best known solutions. In many cases, details of how these results were obtained are not described. This may generate controversy and difficults the comparisons of techniques. This paper shows an example of ambiguity in the results of an instance of the most used VRPTW (Vehicle Routing Problem with Time Windows) bechmark. We show that when analyzing the optimal solution and the best approximate solution of a specific problem, the two results are equivalent. Finally, we will propose a set of guidelines to consider when publishing the results obtained by a new algorithm.