Human Oversight and Overload: Two Hidden and Costly Burdens of AI-Assisted Software Engineering

AI-assisted software engineering and social science require human oversight to ensure reliability, according to recent research. Studies highlight the need for human review and validation of AI-generated artifacts due to their uneven reliability.

Researchers have identified two major burdens associated with AI-assisted software engineering: human oversight and cognitive overload. Human oversight is necessary because AI-generated artifacts require review, validation, and sometimes rework by engineers^[1]. The flood of AI suggestions can also lead to cognitive overload, stretching developers mentally^[1]. In AI-assisted social science, human oversight is equally crucial. A study found that large language models are increasingly used for tasks once reserved for trained researchers, but their outputs can be unreliable^[3]. In one experiment, 72% of runs failed in an unconstrained multi-agent baseline^[3]. However, by imposing certain architectural commitments, the failure rate was reduced to 16%^[3]. The study's authors noted that deterministic computation and human gates contribute independently to reliability gains^[3]. Another study proposed a policy to steer human validation toward tasks where AI is least reliable, using upper confidence bounds learned online^[2].