Papperoni

2021

Program Synthesis With Large Language Models

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le

Open PDF Google Scholar

citations

Cite Score

AI summary

This paper explores the limits of large language models for program synthesis, introducing two new benchmarks, MBPP and MathQA-Python, and showing that synthesis performance scales log-linearly with model size, achieving up to 83.8% accuracy on MathQA-Python with fine-tuning.

Main Contributions

Introduces two new datasets for Python code synthesis: Mostly Basic Programming Problems (MBPP) with 974 tasks and MathQA-Python with 23914 problems.
Demonstrates that large language models perform surprisingly well at few-shot synthesis of Python programs from natural language, with performance scaling log-linearly with model size (up to 59.6% on MBPP few-shot, 83.8% on MathQA-Python fine-tuned).
Shows that models can engage in dialog about code and improve performance with natural language feedback, halving the error rate with four turns of dialog.
Analyzes the semantic grounding of models, finding that even the largest models are generally unable to predict program execution output given specific inputs.
Investigates the impact of prompt examples and test cases on synthesis performance, and confirms that solutions typically generalize to held-out test cases with minimal overlap with pre-training data.

Abstract

This paper explores the limits of the current generation of large language models for program synthesis in general purpose programming languages. We evaluate a collection of such models (with between 244M and 137B parameters) on two new benchmarks, MBPP and MathQA-Python, in both the few-shot and fine-tuning regimes. Our benchmarks are designed to measure the ability of these models to synthesize short Python programs from natural language descriptions. The Mostly Basic Programming Problems (MBPP) dataset contains 974 programming tasks, designed to be solvable by entry-level programmers. The MathQA-Python dataset, a Python version of the MathQA benchmark, contains 23914 problems that evaluate the ability of the models to synthesize code from more complex text. On both datasets, we find that synthesis performance scales log-linearly with model size. Our largest models, even without finetuning on a code dataset, can synthesize solutions to 59.6% of the problems from MBPP using few-shot learning with a well-designed prompt. Fine-tuning on a held-out portion of the dataset improves performance by about 10 percentage points across most model sizes. On the MathQA-Python dataset, the largest fine-tuned model achieves 83.8% accuracy. Going further, we study the model’s ability to engage in dialog about code, incorporating human feedback to improve its solutions. We find that natural language feedback from a human halves the error rate compared to the model’s initial prediction. Additionally, we conduct an error analysis to shed light on where these models fall short and what types of programs are most difficult to generate. Finally, we explore the semantic grounding of these models by fine-tuning them to predict the results of program execution. We find that even our best models are generally unable to predict the output of a program given a specific input.

Citation Graph

Loading graph...

References [102]

Sort:

Filter:

[1]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite