When Better Models Make Worse Systems

Better predictions do not always produce better outcomes. Models are deployed in human systems, where people adapt to them, react to them, and sometimes work around them. A model can improve its own score while making the larger system worse.

Driver-assistance systems are one example. As systems such as Autopilot became more capable, some drivers became more willing to disengage from driving altogether. That overtrust contributed to delayed reactions and crashes: a classic case of automation complacency.

Social-media recommendation systems also became better at predicting engagement. The result was often more emotionally provocative, polarising, or misleading material in people's feeds, because those reliably produced attention. The model succeeded at engagement while degrading information quality and wellbeing.

Navigation apps can optimise routes for individual drivers, reducing their journey times. But when many people follow those routes at once, the traffic may simply move onto residential streets or create new bottlenecks. What is efficient for each driver is not necessarily efficient for the network.

Medical alert systems show the same problem in a different setting. A predictive model may identify high-risk patients reasonably well, but excessive alerts create alert fatigue. Clinicians can eventually start ignoring warnings, including the genuine ones, making the overall system less effective.

The lesson is simple: model accuracy is only one part of a system's performance. Automation bias, feedback loops, objective misalignment, and human adaptation can turn a local improvement into a broader failure. The thing being optimised is not always the thing we actually care about.