Very large models trained on very large datasets display emergent prompting and priming behaviors: by conditioning on a task description and/or priming example pairs, the model generates an output for a given input and gets non-trivial performance on a diversity of benchmarks (cf GPT3). However, these behaviors feel somewhat accidental. When training a large enough left-to-right (a.k.a. causal) language model on enough text, the model learns non-trivial “skills” that are secondary to the LM object for which supervised signal was thought to be necessary. However, prompting and priming are a fickle art right now because of a model’s brittleness to the format of the prompt and examples (AutoPrompt, How can we know what LMs know?, What makes good in-context examples for GPT-3?). We want to make these prompting abilities more explicit and robust rather than solely relying on poorly understood emergent behaviors, by providing as much signal that resembles prompting as possible.
Towards this goal, the biggest “modeling” question is how we augment the training signal to encourage the model to learn prompting behaviors i.e. condition a model on an input (and potentially some additional context and metadata), give it a prompt that asks to process the input and to react to it, and generate the output response. Ideally, we would like to extract 100x scale more signal for prompting from our data sources compared to causal language modeling on raw web text.
The desired outcomes of this working group are:
A/ A very large multilingual model trained with the best-identified training objective and data format for our moonshot goal. Multiple sizes can be trained, but we expect that the largest one will be in the order of magnitude of GPT3 (~150 to 200 billion parameters) and will be trained on the French national supercomputer.
B/ A public codebase gathering the code, tools (and potentially processed dataset) developed in this project. Code should be documented so that it can be a starting point for future endeavors.
C/ Focused paper(s) describing the model, the modeling approach with experiments, and analysis of the design choices.
The moonshot goal of the project is to answer in natural language any prompted query expressed in natural language in a 0-shot manner.