The next frontier of AI: AlphaCode can match the programming prowess of average coders

Artificial intelligence software is becoming surprisingly adept at carrying on conversations, winning board games and generating works of art – but what about creating software? In a recently published article, Google DeepMind researchers claim that their AlphaCode program can keep pace with the average human coder in standardized programming competitions.

“This result marks the first time that an artificial intelligence system has performed competitively in programming contests,” the researchers report in this week’s issue of the journal Science.

There’s no need to raise alarm bells about Skynet just yet: DeepMind’s code-generating system averaged 54.3% in simulated ratings in recent programming contests on the Codeforces platform, which is a very “average” average.

“Competitive programming is an extremely difficult challenge, and there is a huge gap between where we are now (solving about 30% of problems in 10 submissions) and the best programmers (solving >90% of problems in a single submission) “, DeepMind research scientist Yujia Li, one of the main authors of the scientific paper, told GeekWire in an email. “The remaining problems are also much more difficult than the problems we are currently solving.”

Nevertheless, the experiment points to a new frontier in AI applications. Microsoft is also exploring the frontier with a code suggestion program called Copilot which is offered via GitHub. Amazon has a similar software tool, called CodeWhisperer.

Oren Etzioni, founding CEO of the Allen Institute for Artificial Intelligence in Seattle and CTO of the AI2 incubator, told GeekWire that the recently published research highlights DeepMind’s status as a major player in the application. of AI tools known as large language models, or LLMs.

“It’s an impressive reminder that OpenAI and Microsoft don’t have a monopoly on impressive LLM exploits,” Etzioni said in an email. “Far from it, AlphaCode outperforms both GPT-3 and Microsoft’s Github Copilot.”

AlphaCode is arguably also notable for How? ‘Or’ What it programs as it is for how well it programs. “Perhaps most surprising about the system is what AlphaCode doesn’t do: AlphaCode doesn’t contain any explicit built-in knowledge about the structure of computer code. Instead, AlphaCode relies on a purely “data-driven” approach to writing code, learning the structure of computer programs by simply observing a lot of existing code,” wrote J. Zico Kolter, computer scientist at the Carnegie Mellon University, in a scholarly commentary. on the study.

AlphaCode uses a large language model to create code in response to natural language descriptions of a problem. The software leverages a massive data set of programming problems and solutions, as well as a set of unstructured code from GitHub. AlphaCode generates thousands of proposed solutions to the problem at hand, filters those solutions to eliminate invalid ones, groups the surviving solutions into groups, and then selects a single example from each group for submission.

“It might seem surprising that this procedure has any chance of creating correct code,” Kolter said.

Kolter said AlphaCode’s approach could eventually be integrated with more structured machine language methods to improve system performance.

“If ‘hybrid’ ML methods that combine data-driven learning with technical knowledge can work better on these tasks, let them try,” he wrote. “AlphaCode has rolled the dice.”

Li told GeekWire that DeepMind continues to refine AlphaCode. “While AlphaCode is a ~0%-30% milestone, there is still a lot of work to do,” he wrote in his email.

Etzioni agreed that “there is a lot of wiggle room” in the quest to create code generator software. “I expect rapid iteration and improvements,” he said.

“We are only 10 seconds away from the ‘big bang’ of generative AI. Many more impressive products on a wider variety of data, both textual and structured, will be available soon,” Etzioni said. We are feverishly trying to understand how far this technology goes.”

As work progresses, AlphaCode could reignite the long-running debate about the promises and potential dangers of AI, just as DeepMind’s AlphaGo program did when it demonstrated mastery of AI. machine on the old game of Go. And programming isn’t the only area where the rapid advance of AI is causing controversy:

When we asked Li if DeepMind had any qualms about what it was creating, he provided a thoughtful response:

“AI has the potential to help solve humanity’s greatest challenges, but it must be built responsibly and safely, and used for the benefit of all. Whether it benefits or harms us and society depends on how we deploy it, how we use it, and what kinds of things we decide to use it for.

“At DeepMind, we take a thoughtful approach to AI development: inviting scrutiny of our work and not releasing technology until we consider the consequences and mitigate the risks. Guided by our values, our responsible innovation culture is centered on responsible governance, responsible research and responsible impact (you can consult our operating principles here).

In addition to Li, lead authors of the Science research paper, “Competition-level Code Generation With AlphaCode”, include DeepMind’s David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling , Felix Gimeno , Agustin Dal Lago, Thomas Hubert, Peter Choy and Cyprien de Masson d’Autume. Thirteen other researchers are listed as co-authors. A pre-printed version of the paper and additional materials is available through ArXiv.