Machine Learning Under Concept Drift

Abstract

Modern machine learning systems are increasingly deployed in real-world operational en- vironments, where the underlying assumption of data stationarity often breaks down. In domains such as urban traffic management, machine learning models are expected to deliver accurate predictions over time, even as external conditions—such as weather, roadworks, and usage patterns—change. These evolving environments give rise to concept drift, where the statistical relationship between input data and predicted outcomes shifts, potentially de- grading prediction reliability without obvious warning. This presents significant operational and safety challenges for systems that must continuously adapt, such as transportation networks providing real-time traveler information and route optimization. Addressing these challenges, we design, implement, and evaluate a simulation-based, multi- task traffic prediction platform with automated concept-drift monitoring and adaptation. The platform monitors three distinct prediction tasks: estimated time of arrival (ETA), fuel consumption, and trip-stops prediction—estimating the number of intermediate stops within a trip segment. While the methodology is validated across all three tasks, this the- sis focuses primarily on the trip-stops prediction task as the central case study. The core difficulty lies in reliably identifying distributional changes amid temporal noise, across a range of prediction tasks, while minimizing false positive alarms which can be disruptive in production systems. While recent advances in ensemble drift detection have introduced sophisticated approaches, these methods often require manual parameter tuning for each task and can produce unreliable drift alarms that could trigger unnecessary safety interven- tions. The limitations of existing single-detector methods—including parameter sensitivity and unacceptable false alarm rates—motivate the need for conservative ensemble approaches that prioritize false positive control over detection speed. To address these issues, this thesis proposes a task-agnostic ensemble drift detection archi- tecture and validates it in a rigorous simulation environment using SUMO (Simulation of Urban Mobility). The system incorporates automated drift monitoring and dynamic model adaptation to support the complex demands of multi-task traffic forecasting. Notably, the platform avoids any task-specific parameter tuning by employing a consensus-based ensem- ble of statistical detectors to identify significant data drifts. This design enables continuous operation across multiple drift cycles through automated calibration. It effectively balances the need for timely adaptation with the imperative to avoid unnecessary interruptions Experimental results demonstrate that the approach achieves reliable drift detection with zero false alarms under stable conditions, while adapting effectively to abrupt distributional changes—such as those induced by simulated rain events. Adaptation is achieved rapidly and efficiently, restoring predictive accuracy to pre-drift levels and maintaining service quality across tasks. The framework thus provides practical insights for deploying robust, adaptive machine learning systems in dynamically evolving environments, and highlights the impor- tance of automated calibration and complete lifecycle management. While the results are validated through a simulation-based evaluation framework, the principles, methods, and platform architecture are generalizable to a wide range of operational machine learning de- ployments where reliability and adaptability are paramount.