Stanford/MIT Study: GPT Boosts Support Agent Productivity by up to 35%

A new study by researchers from Stanford and MIT presents a first-of-its-kind analysis of the effects of large language model (LLM)-powered software on over 5,000 customer service agents at a Fortune 500 company. The software, which was powered by OpenAI’s GPT-3, delivered a significant impact that may foreshadow accelerated shifts in the labor landscape driven by generative AI's rapid influence.

A Pioneering Wide-Scale, Long-Term Analysis

The study examined more than 3 million chat sessions conducted by 5,179 agents over a year-long period encompassing both pre- and post-AI deployment. Its uniqueness stems from its focus on real-world conditions, rather than controlled lab tests, and its consideration of AI's impact across an extended timeframe on thousands of real workers.

Although the specific software remains unnamed, the researchers disclosed that it was based on OpenAI’s GPT-3 and fine-tuned on customer service interactions. The agents, primarily located overseas, worked for an undisclosed Fortune 500 company specializing in business process software.

Boosting Productivity for New and Low-Skill Agents

The researchers observed a 13.8% increase in the number of chats agents successfully resolved per hour. Specifically, the gains the researchers witnessed were:

• A decline in the time it takes an agent to handle an individual chat.

• An increase in the number of chats that an agent is able to handle per hour.

• A small increase in the share of chats that are successfully resolved.

The improvement was particularly notable among lower-performing agents, who experienced a 35% increase in successful resolutions per hour. This substantial reduction in skill gaps allowed less-skilled workers to perform nearly as well as their highest-skilled counterparts, the researchers found.

Conversely, higher-skilled agents saw little to no increase in productivity. In some cases, they even experienced a decrease, which researchers hypothesized was due to the AI tool's distracting nature.

Notably, new agents also experienced significant performance improvements. Agents with two months of tenure, aided by AI, performed on par with agents who had been working for over six months without AI assistance.

Implications for Knowledge Worker Industries

Generative AI's rapid emergence is already transforming various industries and the work professionals do within them. We have reported on the declining job opportunities for creative artists in China as AI image generation gains widespread adoption in the gaming industry. The impact of generative AI on other roles is just beginning to surface.

For customer service agents, increased retention rates and better performance among lower-skilled agents are clear outcomes of generative AI implementation. These improvements could lead to cost savings for companies as they benefit from higher performance levels and longer employee tenures.

However, the elevation of the lowest performers to the level of the highest performers may have other consequences for the broader labor market in customer support. As AI improves the quality of the least-skilled agents, job requirements and compensation may change as companies no longer need to compete for top talent.

The researchers highlight the importance of considering compensation for workers who contribute data to AI systems. “Our findings raise questions about whether and how workers should be compensated for the data that they provide to AI systems,” the team writes, “High-skill workers, in particular, play an important role in model development but see smaller direct benefits in terms of improving their own productivity.”

As generative AI continues to make a splash in numerous roles, overall efficiency gains may result in a reduction of available support agent roles. Notably, the software examined in the study was based on GPT-3. With the rapid advancements in LLMs, even more capable software utilizing OpenAI's recently released GPT-4 could have profound consequences, including a further increase in support agent efficiency.

Given the early stage of generative AI, the researchers agree: numerous consequences “deserve further scrutiny.”