Resilience to the Flowing Unknown: An Open Set Recognition Framework for Data Streams

Modern digital applications extensively integrate Artificial Intelligence models for automated decision-making. However, these AI-based systems encounter reliability and safety challenges when handling continuous data streams in dynamic scenarios. This work explores the concept of resilient AI systems that must operate in the face of unexpected events and unseen patterns. This is a common issue that regular closed-set classifiers encounter in streaming scenarios, as they are designed to compulsory classify any new observation into one of the training patterns (i.e., the so-called over-occupied space problem). In batch learning, the Open Set Recognition field addresses this issue by requiring models to maintain classification performance when processing unknown patterns. This work investigates the application of an Open Set Recognition framework that combines classification and clustering to address the over-occupied space problem in streaming scenarios. We devise a benchmark comprising different classification datasets with varying ratios of known to unknown classes, and experiments compare the performance of the proposed framework with that of individual incremental classifiers. Discussions held over the obtained results highlight situations where the framework performs best and the limitations of incremental classifiers in open-world streaming environments.