Link to paper
The full paper is available here.
You can also find the paper on PapersWithCode here.
Abstract
- Language models used as mutation and crossover operators for evolutionary neural architecture search algorithm
- Combination of evolutionary prompt engineering and soft prompt-tuning (EvoPrompting) produces diverse and high performing models
- EvoPrompting successful at designing accurate and efficient neural network architectures across a variety of machine learning tasks
Paper Content
Introduction
- Scaling of Transformers has produced language models with impressive performance
- Language models can learn how to code, do math, and solve reasoning problems
- Limitations of language models in solving complex problems and creating novel solutions
- EVOPROMPTING improves ability to propose novel and diverse solutions to complex reasoning problems
- EVOPROMPTING uses evolutionary search to create and curate data to improve LM in-context prompting examples
- Few-shot prompting with EVOPROMPTING enables LMs to create architectures that outperform those designed by human experts
- EVOPROMPTING discovers novel graph neural network architectures that outperform current state-of-the-art
Related work
- Transformer models are popular for natural language systems
- Transformer models can be used to write code, do math, and solve reasoning problems
- Brown et al. (2020) demonstrated that LMs can be prompted with in-context examples
- Numerous works have used prompting to unlock latent LM abilities
- Prompts can be tuned using probabilistic inference techniques
- Evolutionary search can be used to design prompts for in-context learning
- LM replaces mutation and crossover functions in evolutionary search
- LM used as crossover operator to produce variations of text-based genotypes
- Deep sequence models have been used to improve machine learning workflows
Architecture search problem formulation
- The target task is denoted by T
- D is a dataset consisting of input-output pairs
- π θ is a probability distribution over vocabulary V
- Code segments can be sampled from π θ
- EVAL T is an evaluation function that trains the model architecture given by code c on D
- The goal is to identify code samples that maximize the reward EVAL T
Lms for evolutionary crossover and mutation
- Goal of algorithm is to generate set of k neural network architectures that maximize reward
- Use black-box evolutionary approach to generate, score, and select best architectures
- Evolution works well in this domain because of sparse high quality solutions
- Algorithm uses LM for crossover and mutation operations
- Search space includes any neural network architecture that can be represented in Python
- LM is pre-trained on massive datasets containing source code files
- LM can be used as self-adaptive crossover operator
- Scoring function is negative product of validation error and model size
- Initialize with seed architectures that are known to be well-designed
- Create few-shot prompts using source code and evaluation metrics
- Use LM to generate n samples per prompt
- Apply fitness-based selection to identify top candidate models
- Train LM for next round using child models not previously selected
Experiments and results
- Evaluated meta-learning algorithm on two datasets
- π θ0 is a 62B parameter PALM model
- Used AdamW optimizer to train each child model
- Scored child models according to best validation accuracy
- Seeded search with 4 seed models
- Compared EVOPROMPTING with 3 baselines
- EVOPROMPTING finds smaller and more accurate models
- EVOPROMPTING excels at optimizing convolutional architectures
- Performance suffers when any individual component is removed
- Trajectory of meta-learning algorithm round over round
Clrs
- MNIST-1D task is used to evaluate a meta-learning algorithm
- CNN architectures perform well on this task
- CLRS algorithmic reasoning benchmark is used to assess meta-learning framework
- Benchmark evaluates neural networks’ ability to learn algorithmic reasoning
- Algorithms are represented as graphs
- Triplet-GMPNN model performs best
- AdamW optimizer used to train each child model
- Search over triplet representations of Triplet-GMPNN model
- Nine seed models used
- EVOPROMPTING used to search 3 individual algorithms
- New models discovered in model searches over single algorithms
- New models generalized to other algorithms unseen during model search
- Simple models suggested by model search
Conclusion
- Pre-trained language models (LMs) can be embedded in evolutionary algorithms to improve performance on neural architecture design tasks
- EVOPROMPTING can optimize convolutional architectures for MNIST-1D and develop new GNNs for CLRS algorithmic benchmark
- EVOPROMPTING can discover novel, competitive, and state-of-the-art architectures that optimize for accuracy and model size
- EVOPROMPTING is general enough to be adapted to other kinds of reasoning tasks
- EVOPROMPTING is more efficient than other search methods, reaching a maximum fitness with fewer samples
- EVOPROMPTING combines evolutionary search with soft-prompt tuning
- EVOPROMPTING uses evaluation function to train model and return lowest validation error
- EVOPROMPTING uses four seed models for MNIST-1D model search and nine seed models for CLRS model search
- EVOPROMPTING uses triplet representations such as MAXMEAN, CONCATREP, TANHEXPANDTRIPLETS, DIV2MEAN, and others
- EVOPROMPTING uses fully-connected multi-layer perceptrons and linear layers for node and edge representations