Link to paper

The full paper is available here.

You can also find the paper on PapersWithCode here.

Abstract

  • Deep learning has become the most important technique in Artificial Intelligence (AI).
  • Deep learning has been used in areas such as protein folding, drug discovery, integrated chip design, and weather prediction.
  • AI idea production is more capital-intensive than traditional R&D.
  • AI-augmented R&D has the potential to speed up technological change and economic growth.

Paper Content

Introduction

  • AI has potential to impact idea production and economic growth
  • Deep learning has led to rapid progress in AI
  • AI has potential to replace human labor
  • Existing research has focused on AI’s impact on final good production
  • AI has potential to change innovation process
  • Empirical testing of AI’s impact on R&D has mixed results
  • AI increases returns to data and knowledge production
  • AI has potential to produce more dramatic effects on productivity growth than changes in final goods production
  • Deep learning increases capital intensity of R&D
  • Higher capital intensity of R&D implies faster productivity growth
  • Results are robust to outliers and alternative model specifications

Capital-intensive r&d and deep learning

The role of capital in idea production

  • Deep learning may affect the growth rate of knowledge by impacting the productivity of research capital.
  • Investment in capital can increase the marginal product of labour in R&D and thus increase investment in the R&D sector.
  • Empirical work has validated the importance of physical capital investment in idea production.
  • Academic institutions have spent over $2bn per year on Capital Equipment or Software for R&D since 2010.
  • Deep neural networks can evade the trade-off between bias and variance by deploying more computational capital.
  • Test error falls according to a power law in the scale of deep learning models and the amount of compute used.
  • The amount of computational capital used in milestone models doubles roughly every 6 months.

R&d capital intensity in a semi-endogenous growth model

  • Consider a simple semi-endogenous growth model
  • Two sectors: goods-producing and R&D
  • Labour and capital used in both sectors
  • Ideas are non-rivalrous and used equally
  • Constant returns to scale in goods production
  • Diminishing returns in production of new ideas
  • Constant saving rate and capital depreciation rate
  • Population grows at exogenous rate
  • Positive shock to R&D elasticity to capital increases idea accumulation and economic growth
  • Highly capital intensive R&D could result in super-normal productivity growth
  • US R&D tends to be highly labour-intensive
  • Deep learning has higher returns to capital than other types of R&D

Data

  • Estimates of R&D elasticity of human capital for image classification are 0.246 and 0.350.
  • Estimates of R&D elasticity of human capital for object detection are 0.352 and 0.319.
  • Implied optimal R&D expenditure breakdown suggests capital-cost shares between 29% and 44%.

Data on computer vision experiments

  • Dataset covers 151 models published between 2012 and 2021
  • Dataset is an augmented version of Thompson et al. 2020
  • Compute estimates are derived from underlying papers following procedure described by Sevilla et al. 2022b
  • Dataset includes ImageNet and MS COCO datasets

Data on authors and publications

  • Dataset includes papers from 1993-2021 in computer science subfields related to machine learning
  • Authors of papers matched to entries in Scopus using string distance-based matching
  • 90.1% of authors matched, 96% of matches correct
  • Timeseries constructed for each author showing publications, h-index, and citations
  • Grant funding, institutional rankings, and journal/conference influence also included

Empirical strategy

  • Assumption that idea production using deep learning depends on three factors: labour, specialised capital goods, and total factor productivity
  • Estimate this using data by replacing Ȧ(t) with performance of deep learning models, C(t) with computational inputs, and S(t) with estimates of scientific human capital inputs

Empirical specification

  • Technology grows exponentially on the balanced growth path
  • Performance on machine learning tasks is measured as a type of predictive accuracy
  • Performance relates to technology according to the logistic function
  • Logistic function implies a power-law between the scale of compute deployed and the level of error achieved
  • Proportional increase in accuracy provides a decomposition of technological progress
  • Model assumes knowledge is non-rivalrous

Operationalizing innovations

  • Estimating the model requires operationalizing proportional performance improvements in terms of observables.
  • Performance gains are measured using baseline data, where authors indicate the touchstone models they are building on.
  • Performance gains are defined as proportional improvement over the performance of a relevant baseline model.
  • This operationalization is chosen because it is common practice in the machine learning literature and it lines up with the notion of the change in stock of knowledge in R&D-based growth models.
  • Appropriate baseline levels of performance are found by surveying the models used as baseline results in the relevant literature and taking the median of their performance.

Modelling human capital

  • Need to construct measure of scientific human capital used for computer vision models
  • Measure should be predictive of outcomes influenced by human capital
  • Prior work used various measures of quality/status of scientists/engineers
  • These measures are weakly predictive of outcomes where human capital is important
  • Strategy is to construct a deep neural network to develop a single-dimensional representation of total quality-adjusted research input
  • Representation is highly predictive of key bibliometric and publication-related outcomes

Our machine learning approach to estimating human capital

  • A neural network was trained to predict bibliometric and publication-related outcomes
  • A diagrammatic overview of the data pipeline and training set-up is provided in Figure 2a
  • The neural network architecture is shown in Figure 2b
  • The architecture consists of 15 sets of hidden layers followed by a single unit - the “human capital” unit
  • The human capital result is then concatenated with the publication date and fed into a series of independent sub-branches
  • The approach implements an encoder and a set of decoder regression models

Validating our estimates

  • Our measure predicts more than 55% of the variation in citations and journal quality rankings
  • 4-5 times more predictive than other commonly used proxies
  • Predictive performance is better than simpler approaches found in the literature
  • Obtained prediction errors 40% lower than Lasso regressions
  • No natural unit for human capital, unclear how citations and journal quality relate to scientific merit

Empirical analysis

  • Estimate production functions for two AI tasks: image classification and object detection
  • Estimate using OLS, except where heteroskedascitity is present, in which case use GLS model by Maximum Likelihood
  • Results for models A1-B2 displayed in Table 4
  • Pooled model with distinct time-fixed effects combines data across both computer vision tasks
  • Estimates of models C1-C2 displayed in Table 5
  • R&D elasticity of capital for image classification is 0.111-0.140%, for object detection is 0.246
  • Pooled estimates between 0.145 and 0.176
  • All results statistically significant at 5% level, most also significant at 0.1% level

R&d with deep learning

  • Widespread adoption of deep learning would act as a one-time shock, raising the capital intensity of knowledge production in the economy
  • Depending on the model specification, the results from image classification would imply a productivity growth rate between 1.6% and 1.8%, whereas those from object detection would imply a rate between 3.1% and 3.9%
  • With the widespread adoption of deep learning, productivity growth rate would rise to between 2.1% and 2.4%, a 1.7-2 fold increase relative to the 1.2% average U.S. productivity growth from 1948 to 2021

Robustness and external validity

  • Results in Section 7 are robust to outliers and different time periods
  • Assumptions about labour and capital are consistent with data
  • Discuss generalizability of estimates from computer vision to other R&D tasks

Sensitivity to outliers

  • It is known that certain empirical results can be reversed by removing less than 1% of the sample.
  • The authors tested the robustness of their results by re-running their analysis multiple times on random sub-samples of their datasets.
  • The authors found that the point estimates were mostly robust to the removal of any small subset of observations.

Alternative model specifications

  • Estimation strategy for A t takes advantage of cross-sectional variation at each time point.
  • Pool publications into groups of contemporaries published around the same time.
  • Bias-variance trade-off when specifying more granular time periods.
  • Fixed time periods as yearly intervals.
  • Estimates are mostly similar for all relevant datasets.
  • Estimates of R&D elasticity to capital (β) are relatively lower for image classification tasks than for object detection tasks.

Sensitivity to model assumptions about the substitutability of human scientists

  • Assume elasticity of substitution equals 1 (Cobb-Douglas production function)
  • Test if results hold with lesser degree of substitutability of human scientists
  • Substitutability assumption important because model implies compute stock will grow faster than stock of scientists
  • Estimate equation 15 to investigate substitutability assumption
  • Estimate using translog production function
  • Bootstrapping estimates of σ clustered around unity
  • Estimates of σ slightly above 1, implying capital intensity of deep learning-based R&D is an underestimate

Limits to external validity stemming from our choice of domain

  • Analysis focuses on two computer vision tasks
  • Deep learning often uses common techniques, algorithms, and architectures
  • Most deep learning systems use a ‘deep’ computational graph with parameters learned through gradient descent
  • AlexNet in 2012 was one of the first instances of these key features
  • Different architectures are used across domains
  • Transformer-based models are used across many domains
  • Findings may generalize to domains outside of computer vision
  • Neural scaling laws suggest relation between model size and performance is universal

How data availability influences our estimates

  • Computer vision has a relative abundance of quality labelled data
  • Protein folding has less data available and is expensive to generate
  • Returns to compute are lower in low-data regimes
  • AI-based ideas production is rapidly expanding in data-rich sectors
  • Neural scaling laws suggest data and compute can substitute for data
  • Economically important R&D tasks may have complementary investments in generating necessary datasets
  • Low-data regimes may be short-lived or atypical for R&D tasks

Discussion

Summary and implications

  • Deep learning is being adopted in the R&D sector and could lead to permanent acceleration of technological change
  • Data from two computer vision tasks suggests deep learning is more capital-intensive than other forms of R&D
  • A novel machine learning-based method was introduced to infer human capital from scientific publications
  • Future research should focus on the diffusion of deep learning, capital intensity of deep learning, and the knowledge production of AI-augmented R&D

Selection issues and measurement error

  • Data used in empirical work may suffer from selection bias
  • Dataset contains models for which authors reported enough to infer how model was trained and how much computation was needed
  • Bias could be introduced if reporting is more common among computationally-intensive papers
  • Bias could positively affect estimates of returns to physical capital
  • Leading machine learning conferences require/encourage researchers to report details about hardware usage
  • Variation in amount of compute used spans over four orders of magnitude
  • Estimate of effectiveness of compute could be overestimated
  • Human capital estimates could also be overestimated
  • Measurement-error issues with estimates of amounts of training computation
  • Multiple methods used to reduce variance of estimates

Conclusion

  • AI has potential to impact productivity and output growth
  • Deep learning can be used to efficiently harness more computational capital in R&D
  • Greater capital intensity in R&D can lead to permanent increases in productivity growth and economic growth
  • Deep learning-based computer vision research is significantly more capital-intensive than other R&D sectors
  • Widespread adoption of deep learning-based R&D could double U.S. productivity growth rates
  • Data on deep learning models was collected from 46 publications
  • Data on machine learning baseline results was collected from arXiv, Microsoft Project Academic Knowledge, and Scopus
  • 223,703 authors on 115,235 machine learning papers from 1993 to 2021 were studied
  • Features studied include number of publications, number of total citations, and authors’ h-index
  • Grants received by institutions was collected from Dimensions database
  • Institutional rankings for Computer Science was generated using Computer Science publication data
  • Human capital measure was evaluated
  • Implied optimal R&D expenditure breakdown was calculated
  • Predicted productivity growth under widespread deployment of AI in R&D was estimated
  • Elasticity of substitution estimates were generated
  • Deep learning production function estimates were generated for image classification and object detection
  • Pooled deep learning production function estimates were generated for computer vision experiments
  • Alternative window lengths were studied