Balancing Performance, Efficiency and Robustness in Open-World Machine Learning via Evolutionary Multi-objective Model Compression

When deploying machine learning models on resource-constrained hardware, reducing the memory footprint required by the model without compromising its performance is critical. Moreover, in open-world scenarios models often operate in dynamic and unpredictable environments where the data distribution evolves over time. Robust models can generalize well to unforeseen circumstances, including out-of-distribution inputs that may not have been encountered during the training phase. This adaptability is essential to handle the inherent variability of real-world data. This work formulates a multi-objective optimization problem that aims at optimizing the quantization resolution of the parameters of an already trained machine learning model based on three conflicting goals: maximizing the performance of the model on its designated learning task, minimizing the memory footprint of the compressed model, and enhancing its robustness against out-of-distribution data. Given the complexity of the resulting combinatorial optimization problem, we employ multi-objective evolutionary algorithms to efficiently obtain an approximation of the Pareto front balancing among the aforementioned objectives. Experiments with a randomized neural network compressed under the proposed formulation are run over several benchmark classification datasets. Different multi-objective solvers are employed to compare their effectiveness in terms of the convergence and diversity of their produced Pareto estimations. Additionally, we assess the achieved equilibrium between the three objectives against a floating-point implementation of the same model. Our experiments reveal that both the computational resources and the robustness of the model can be optimized via evolutionary quantization without significantly sacrificing its performance for the task at hand.