"many humans are inclined to trust machines even when they are warned not to. The phenomenon of "automation bias" has been documented repeatedly" [automation bias]
The quote names a documented psychological mechanism that directly contradicts the assumption underlying human-control policy: humans systematically fail to override AI recommendations even when explicitly instructed to maintain skepticism.
Reasoning from this article
The article cites multiple empirical instances of this failure (GPS drivers, fire-exit robots, Volvo crash test at 30%), then introduces the term 'cognitive surrender' from Wharton academics to describe the deeper phenomenon: humans don't just fail to override AI—they actively adopt the AI's judgment as their own. This is not a training or interface problem but a structural feature of human cognition under automation, making it a persistent vulnerability in any system relying on human final authority.