Link to paper

The full paper is available here.

You can also find the paper on PapersWithCode here.

Abstract

  • Ivy is a templated Deep Learning (DL) framework.
  • Ivy unifies the core functions of existing DL frameworks.
  • Ivy supports TensorFlow, PyTorch, MXNet, Jax and NumPy.
  • Ivy releases four pure-Ivy libraries for mechanics, 3D vision, robotics, and differentiable environments.
  • Ivy can reduce lines of code with a runtime overhead of less than 1%.

Paper Content

Introduction

  • There is a trade-off between run-time efficiency and ease of development in software projects.
  • Python is the front-runner language for Deep Learning (DL) frameworks.
  • DL frameworks use pre-compiled C++ code in the backend for runtime efficiency and control.
  • Most developers use Python functions for DL projects.
  • Ivy abstracts existing DL frameworks to make them compatible with each other.

Towards general differentiable programming

  • DL models increasingly use a hybrid of neural networks and parameter-free, “handdesigned” components
  • Parameter-free computation blocks can still pass gradients for end-to-end learning
  • Convolutional architecture is an example of inductive bias in the computation graph
  • Works combine DL with SfM for geometric reconstructions
  • Gradient based optimization pre-dates DL in many applied fields
  • Path planning has seen interesting intersections with DL
  • Other fields exploit parameter-free computation in end-to-end graphs
  • Libraries provide domain specific functions with support for gradient propagation

A templated framework

  • Ivy takes inspiration from template metaprogramming and template methods to abstract DL frameworks.
  • Ivy abstracts core tensor functions, which are often semantically similar but syntactically unique.
  • Ivy enables functions, layers and libraries to be implemented once with support for all prominent modern Python DL frameworks.
  • Ivy encourages users to join the Ivy community by implementing their own functions, layers and libraries.

Deep learning frameworks

  • Deep learning progress has advanced rapidly in the past decade
  • Many frameworks were designed for matrix and tensor operations before modern DL
  • NumPy is a widely used package for scientific computing
  • NumPy does not support automatic differentiation and back-propagation
  • Python is the front-runner language for DL interfaces
  • There is semantic consistency between the core tensor APIs of modern python DL libraries

Deep learning libraries

  • Many field-specific libraries exist for deep learning
  • Officially supported framework extensions are becoming common
  • Libraries can become obsolete when frameworks become obsoleted
  • No solution for building large framework-agnostic libraries for users of both present and future DL Python frameworks

Deep learning abstractions

  • Previous works provide framework-level abstractions for DL
  • Keras provided abstractions at the level of classes and models
  • Ivy simplifies and reduces the abstraction to the core tensor APIs
  • TensorLy offers a framework agnostic functional API
  • Ivy offers a comprehensive API applicable to a wide variety of fields

Ivy core

  • Ivy API provides functions for implementing a variety of examples
  • User selects backend framework for Ivy functions
  • Ivy functions are unit tested against each backend framework
  • Core Ivy API can be extended to include additional functions

Framework-specific namespaces

  • Ivy API wraps native functions to provide consistent syntax and call signatures.
  • Ivy API extends functionality in cases where native functions are lacking.
  • Ivy API follows most common syntax and selects most general variant among backends.
  • Ivy API has framework-specific namespaces with updated syntax and call signatures.
  • Ivy API makes minor changes to enable direct bindings to native functions.

Framework-agnostic namespace

  • Framework-specific namespaces provide unification of syntax and call signatures.
  • Framework-agnostic Ivy namespace is used to create new framework-agnostic code.
  • Framework-agnostic functions are implemented in two lines of code.
  • User specifies which backend framework to use at runtime.

Local framework specification

Global framework setting

  • A framework can be set globally for all future function calls using ivy.set_framework().
  • The framework can be unset using ivy.unset_framework().
  • The framework can also be set globally for a block of code using the with command.

Ivy classes

  • Ivy is a fully functional framework
  • Ivy provides classes that build on the functional primitives
  • Ivy provides trainable neural network layers and stateful optimizers
  • High level classes are built directly on top of Ivy’s framework-agnostic functional API
  • Details of layers and optimizers are beyond the scope of this paper

Ivy libraries

  • Ivy provides a core framework-agnostic API
  • Ivy provides four libraries for mechanics, 3D vision, robotics, and differentiable RL environments
  • Ivy Mech provides functions for conversions of orientation, pose, and positional representations
  • Ivy Vision provides functions for camera geometry, image projections, coordinate frame transformations, forward warping, inverse warping, optical flow, depth triangulation, voxel grids, point clouds, implicit rendering and signed distance functions
  • Ivy Robot provides functions and classes for gradient-based motion planning and trajectory optimization
  • Ivy Gym provides differentiable implementations of classic control tasks from OpenAI Gym

A spectrum of users

  • Ivy can be used in a variety of ways
  • Three hypothetical groups of Ivy users: Ivy contributors, Ivy creators and Ivy library users
  • Ivy contributors use Ivy Core to develop an Ivy library
  • Ivy library users use existing Ivy libraries to supplement their own projects
  • Ivy creators use both Ivy core and the Ivy libraries to implement substantial parts of their own personal project in Ivy

End-to-end integration

  • Ivy libraries can be integrated into computation graphs for end-to-end training
  • Ivy libraries can be used for gradient-based methods outside of deep learning
  • Example of combining Ivy libraries for motion planning of a drone in a scene with obstacles
  • Code for example provided in Appendix A.3
  • Visualization and simulation done with PyRep and CoppeliaSim

Framework evaluations

  • Ivy abstraction reduces development time
  • LoC analysis shows Ivy and its libraries reduce lines of code
  • Runtime analysis of Ivy core assesses overhead of wrapping backend functions

Line of code analysis

  • Ivy reduces the lines of code required for developers
  • Writing a library once with joint support of all DL frameworks reduces code by 20%
  • Ivy libraries offer a variety of commonly used functions in different areas of applied DL
  • 100 lines of code required with Ivy and its libraries
  • 400 lines of code required without Ivy and its libraries
  • 455 lines of code required with Ivy but without its libraries
  • 1820 lines of code required without Ivy and its libraries

Ivy core runtime analysis

  • 53 of 101 core functions incur no overhead for any backend framework
  • Each function is called with inputs of flattened size 1 x 10^4
  • Runtime analysis for each core function averaged across backend frameworks
  • Ivy overhead is minimal in both compiled and eager mode
  • Vision library incurs largest Ivy overhead due to frequent usage of gather and scatter functions

Conclusion and future work

  • Presents Ivy, a templated deep learning framework
  • Supports TensorFlow, PyTorch, MXNet, Jax, and Numpy
  • Offers potential for creating framework-agnostic DL libraries
  • Provides four initial Ivy libraries for mechanics, 3D vision, robotics, and differentiable environments
  • Encourages developers to join the Ivy community by writing their own functions, layers and libraries
  • Will continue extending the derived libraries and adding new libraries for additional research fields
  • Will continue developing Ivy Core to remain compatible with all the latest DL framework developments
  • Will strive to support the community of open DL research through the framework
  • Allows developers to stay in control regarding the depth of the Ivy abstraction in their own projects
  • Enables the network to be trained using the native framework’s own optimizers and trainers
  • Presents a spectrum of potential Ivy users
  • Presents an example application using Ivy
  • Presents runtimes for each Ivy core method
  • Presents lines of code to implement the demo
  • Presents percentage slowdown when using Ivy