Link to paper
The full paper is available here.
You can also find the paper on PapersWithCode here.
Abstract
- Deep learning has become the most important technique in Artificial Intelligence (AI).
- Deep learning has been used in areas such as protein folding, drug discovery, integrated chip design, and weather prediction.
- AI idea production is more capital-intensive than traditional R&D.
- AI-augmented R&D has the potential to speed up technological change and economic growth.
Paper Content
Introduction
- AI has potential to impact idea production and economic growth
- Deep learning has led to rapid progress in AI
- AI has potential to replace human labor
- Existing research has focused on AI’s impact on final good production
- AI has potential to change innovation process
- Empirical testing of AI’s impact on R&D has mixed results
- AI increases returns to data and knowledge production
- AI has potential to produce more dramatic effects on productivity growth than changes in final goods production
- Deep learning increases capital intensity of R&D
- Higher capital intensity of R&D implies faster productivity growth
- Results are robust to outliers and alternative model specifications
Capital-intensive r&d and deep learning
The role of capital in idea production
- Deep learning may affect the growth rate of knowledge by impacting the productivity of research capital.
- Investment in capital can increase the marginal product of labour in R&D and thus increase investment in the R&D sector.
- Empirical work has validated the importance of physical capital investment in idea production.
- Academic institutions have spent over $2bn per year on Capital Equipment or Software for R&D since 2010.
- Deep neural networks can evade the trade-off between bias and variance by deploying more computational capital.
- Test error falls according to a power law in the scale of deep learning models and the amount of compute used.
- The amount of computational capital used in milestone models doubles roughly every 6 months.
R&d capital intensity in a semi-endogenous growth model
- Consider a simple semi-endogenous growth model
- Two sectors: goods-producing and R&D
- Labour and capital used in both sectors
- Ideas are non-rivalrous and used equally
- Constant returns to scale in goods production
- Diminishing returns in production of new ideas
- Constant saving rate and capital depreciation rate
- Population grows at exogenous rate
- Positive shock to R&D elasticity to capital increases idea accumulation and economic growth
- Highly capital intensive R&D could result in super-normal productivity growth
- US R&D tends to be highly labour-intensive
- Deep learning has higher returns to capital than other types of R&D
Data
- Estimates of R&D elasticity of human capital for image classification are 0.246 and 0.350.
- Estimates of R&D elasticity of human capital for object detection are 0.352 and 0.319.
- Implied optimal R&D expenditure breakdown suggests capital-cost shares between 29% and 44%.
Data on computer vision experiments
- Dataset covers 151 models published between 2012 and 2021
- Dataset is an augmented version of Thompson et al. 2020
- Compute estimates are derived from underlying papers following procedure described by Sevilla et al. 2022b
- Dataset includes ImageNet and MS COCO datasets
Data on authors and publications
- Dataset includes papers from 1993-2021 in computer science subfields related to machine learning
- Authors of papers matched to entries in Scopus using string distance-based matching
- 90.1% of authors matched, 96% of matches correct
- Timeseries constructed for each author showing publications, h-index, and citations
- Grant funding, institutional rankings, and journal/conference influence also included
Empirical strategy
- Assumption that idea production using deep learning depends on three factors: labour, specialised capital goods, and total factor productivity
- Estimate this using data by replacing Ȧ(t) with performance of deep learning models, C(t) with computational inputs, and S(t) with estimates of scientific human capital inputs
Empirical specification
- Technology grows exponentially on the balanced growth path
- Performance on machine learning tasks is measured as a type of predictive accuracy
- Performance relates to technology according to the logistic function
- Logistic function implies a power-law between the scale of compute deployed and the level of error achieved
- Proportional increase in accuracy provides a decomposition of technological progress
- Model assumes knowledge is non-rivalrous
Operationalizing innovations
- Estimating the model requires operationalizing proportional performance improvements in terms of observables.
- Performance gains are measured using baseline data, where authors indicate the touchstone models they are building on.
- Performance gains are defined as proportional improvement over the performance of a relevant baseline model.
- This operationalization is chosen because it is common practice in the machine learning literature and it lines up with the notion of the change in stock of knowledge in R&D-based growth models.
- Appropriate baseline levels of performance are found by surveying the models used as baseline results in the relevant literature and taking the median of their performance.
Modelling human capital
- Need to construct measure of scientific human capital used for computer vision models
- Measure should be predictive of outcomes influenced by human capital
- Prior work used various measures of quality/status of scientists/engineers
- These measures are weakly predictive of outcomes where human capital is important
- Strategy is to construct a deep neural network to develop a single-dimensional representation of total quality-adjusted research input
- Representation is highly predictive of key bibliometric and publication-related outcomes
Our machine learning approach to estimating human capital
- A neural network was trained to predict bibliometric and publication-related outcomes
- A diagrammatic overview of the data pipeline and training set-up is provided in Figure 2a
- The neural network architecture is shown in Figure 2b
- The architecture consists of 15 sets of hidden layers followed by a single unit - the “human capital” unit
- The human capital result is then concatenated with the publication date and fed into a series of independent sub-branches
- The approach implements an encoder and a set of decoder regression models
Validating our estimates
- Our measure predicts more than 55% of the variation in citations and journal quality rankings
- 4-5 times more predictive than other commonly used proxies
- Predictive performance is better than simpler approaches found in the literature
- Obtained prediction errors 40% lower than Lasso regressions
- No natural unit for human capital, unclear how citations and journal quality relate to scientific merit
Empirical analysis
- Estimate production functions for two AI tasks: image classification and object detection
- Estimate using OLS, except where heteroskedascitity is present, in which case use GLS model by Maximum Likelihood
- Results for models A1-B2 displayed in Table 4
- Pooled model with distinct time-fixed effects combines data across both computer vision tasks
- Estimates of models C1-C2 displayed in Table 5
- R&D elasticity of capital for image classification is 0.111-0.140%, for object detection is 0.246
- Pooled estimates between 0.145 and 0.176
- All results statistically significant at 5% level, most also significant at 0.1% level
R&d with deep learning
- Widespread adoption of deep learning would act as a one-time shock, raising the capital intensity of knowledge production in the economy
- Depending on the model specification, the results from image classification would imply a productivity growth rate between 1.6% and 1.8%, whereas those from object detection would imply a rate between 3.1% and 3.9%
- With the widespread adoption of deep learning, productivity growth rate would rise to between 2.1% and 2.4%, a 1.7-2 fold increase relative to the 1.2% average U.S. productivity growth from 1948 to 2021
Robustness and external validity
- Results in Section 7 are robust to outliers and different time periods
- Assumptions about labour and capital are consistent with data
- Discuss generalizability of estimates from computer vision to other R&D tasks
Sensitivity to outliers
- It is known that certain empirical results can be reversed by removing less than 1% of the sample.
- The authors tested the robustness of their results by re-running their analysis multiple times on random sub-samples of their datasets.
- The authors found that the point estimates were mostly robust to the removal of any small subset of observations.
Alternative model specifications
- Estimation strategy for A t takes advantage of cross-sectional variation at each time point.
- Pool publications into groups of contemporaries published around the same time.
- Bias-variance trade-off when specifying more granular time periods.
- Fixed time periods as yearly intervals.
- Estimates are mostly similar for all relevant datasets.
- Estimates of R&D elasticity to capital (β) are relatively lower for image classification tasks than for object detection tasks.
Sensitivity to model assumptions about the substitutability of human scientists
- Assume elasticity of substitution equals 1 (Cobb-Douglas production function)
- Test if results hold with lesser degree of substitutability of human scientists
- Substitutability assumption important because model implies compute stock will grow faster than stock of scientists
- Estimate equation 15 to investigate substitutability assumption
- Estimate using translog production function
- Bootstrapping estimates of σ clustered around unity
- Estimates of σ slightly above 1, implying capital intensity of deep learning-based R&D is an underestimate
Limits to external validity stemming from our choice of domain
- Analysis focuses on two computer vision tasks
- Deep learning often uses common techniques, algorithms, and architectures
- Most deep learning systems use a ‘deep’ computational graph with parameters learned through gradient descent
- AlexNet in 2012 was one of the first instances of these key features
- Different architectures are used across domains
- Transformer-based models are used across many domains
- Findings may generalize to domains outside of computer vision
- Neural scaling laws suggest relation between model size and performance is universal
How data availability influences our estimates
- Computer vision has a relative abundance of quality labelled data
- Protein folding has less data available and is expensive to generate
- Returns to compute are lower in low-data regimes
- AI-based ideas production is rapidly expanding in data-rich sectors
- Neural scaling laws suggest data and compute can substitute for data
- Economically important R&D tasks may have complementary investments in generating necessary datasets
- Low-data regimes may be short-lived or atypical for R&D tasks
Discussion
Summary and implications
- Deep learning is being adopted in the R&D sector and could lead to permanent acceleration of technological change
- Data from two computer vision tasks suggests deep learning is more capital-intensive than other forms of R&D
- A novel machine learning-based method was introduced to infer human capital from scientific publications
- Future research should focus on the diffusion of deep learning, capital intensity of deep learning, and the knowledge production of AI-augmented R&D
Selection issues and measurement error
- Data used in empirical work may suffer from selection bias
- Dataset contains models for which authors reported enough to infer how model was trained and how much computation was needed
- Bias could be introduced if reporting is more common among computationally-intensive papers
- Bias could positively affect estimates of returns to physical capital
- Leading machine learning conferences require/encourage researchers to report details about hardware usage
- Variation in amount of compute used spans over four orders of magnitude
- Estimate of effectiveness of compute could be overestimated
- Human capital estimates could also be overestimated
- Measurement-error issues with estimates of amounts of training computation
- Multiple methods used to reduce variance of estimates
Conclusion
- AI has potential to impact productivity and output growth
- Deep learning can be used to efficiently harness more computational capital in R&D
- Greater capital intensity in R&D can lead to permanent increases in productivity growth and economic growth
- Deep learning-based computer vision research is significantly more capital-intensive than other R&D sectors
- Widespread adoption of deep learning-based R&D could double U.S. productivity growth rates
- Data on deep learning models was collected from 46 publications
- Data on machine learning baseline results was collected from arXiv, Microsoft Project Academic Knowledge, and Scopus
- 223,703 authors on 115,235 machine learning papers from 1993 to 2021 were studied
- Features studied include number of publications, number of total citations, and authors’ h-index
- Grants received by institutions was collected from Dimensions database
- Institutional rankings for Computer Science was generated using Computer Science publication data
- Human capital measure was evaluated
- Implied optimal R&D expenditure breakdown was calculated
- Predicted productivity growth under widespread deployment of AI in R&D was estimated
- Elasticity of substitution estimates were generated
- Deep learning production function estimates were generated for image classification and object detection
- Pooled deep learning production function estimates were generated for computer vision experiments
- Alternative window lengths were studied