Rapidly escalating complexity in AI tasks: Implications for utilization and potential consequences.
AI research has introduced a novel method to assess its performance – by measure the time it takes for AI models to complete tasks compared to humans. Although AI excels in text prediction and knowledge tasks, it lags when assisting with more substantial projects like remote executive work.
These performance differences have led scientists to examine AI models based on the duration of tasks they can finish, contrasted with the time human beings require. In a study published on the preprint database arXiv on March 30, researchers from AI organization Model Evaluation & Threat Research (METR) analyzed various AI models, and their ability to perform tasks ranging from straightforward Wikipedia fact-checking to complex programming tasks that take human experts multiple hours, such as writing CUDA kernels or fixing a subtle PyTorch bug.
The study leveraged testing tools like HCAST and RE-Bench. HCAST offers 189 software tasks designed to assess AI agents' capabilities in areas like machine learning, cybersecurity, and software engineering, while RE-Bench features seven challenging open-ended machine-learning research tasks, including optimizing a GPU kernel. By utilizing these tools, the researchers could test AI models on a wide array of tasks demanding varying degrees of skill and time.
To assess the complexity of these tasks, the study also rated them for "messiness," which characterizes tasks that require real-time coordination among several elements and are more representative of real-world assignments. In addition, the researchers developed software atomic actions (SWAA) to establish how quickly humans could complete single-step tasks, ranging from one to 30 seconds, measured using METR employees as a baseline.
The results indicated that AI models could accomplish tasks taking less than four minutes with almost a 100% success rate. However, this rate plummeted to 10% for tasks requiring over four hours. Older AI models underperformed at longer tasks in comparison to newer systems.
As the researchers projected, based on these trends, AI could automate one month worth of human software development by 2032. This study could pave the way for a new benchmark for understanding the actual intelligence and abilities of AI systems, providing a meaningful interpretation of absolute performance beyond relative comparisons with humans.
As Sohrob Kazerounian, distinguished AI researcher at Vectra AI, put it, "this new metric may not change the direction of AI development, but it will help track progress on certain types of tasks where AI systems will be valuable." Eleanor Watson, an AI ethics engineer at Singularity University, agrees, calling it a "valuable and intuitive" metric that directly reflects real-world complexity, providing insights into AI's capacity for maintaining goal-directed behavior over extended periods.
In essence, this study's findings underscore the rapidly advancing capabilities of AI, demonstrating its potential impact on society and the need for continued research to harness these advancements while minimizing their potential risks. The growth in AI's ability to perform lengthy tasks could transform industries, reshape the workforce, and redefine the way we interact with machines in the years to come.
Science and technology news have reported on the use of artificial intelligence (AI) models, which have been analyzed by researchers from Model Evaluation & Threat Research (METR) based on the duration of tasks they can finish compared to humans. Specifically, the study used testing tools like HCAST and RE-Bench to evaluate AI models on a wide array of tasks, and rated them for "messiness" to assess their complexity and applicability to real-world assignments.
