Link to paper
The full paper is available here.
You can also find the paper on PapersWithCode here.
Abstract
- Ivy is a templated Deep Learning (DL) framework.
- Ivy unifies the core functions of existing DL frameworks.
- Ivy supports TensorFlow, PyTorch, MXNet, Jax and NumPy.
- Ivy releases four pure-Ivy libraries for mechanics, 3D vision, robotics, and differentiable environments.
- Ivy can reduce lines of code with a runtime overhead of less than 1%.
Paper Content
Introduction
- There is a trade-off between run-time efficiency and ease of development in software projects.
- Python is the front-runner language for Deep Learning (DL) frameworks.
- DL frameworks use pre-compiled C++ code in the backend for runtime efficiency and control.
- Most developers use Python functions for DL projects.
- Ivy abstracts existing DL frameworks to make them compatible with each other.
Towards general differentiable programming
- DL models increasingly use a hybrid of neural networks and parameter-free, “handdesigned” components
- Parameter-free computation blocks can still pass gradients for end-to-end learning
- Convolutional architecture is an example of inductive bias in the computation graph
- Works combine DL with SfM for geometric reconstructions
- Gradient based optimization pre-dates DL in many applied fields
- Path planning has seen interesting intersections with DL
- Other fields exploit parameter-free computation in end-to-end graphs
- Libraries provide domain specific functions with support for gradient propagation
A templated framework
- Ivy takes inspiration from template metaprogramming and template methods to abstract DL frameworks.
- Ivy abstracts core tensor functions, which are often semantically similar but syntactically unique.
- Ivy enables functions, layers and libraries to be implemented once with support for all prominent modern Python DL frameworks.
- Ivy encourages users to join the Ivy community by implementing their own functions, layers and libraries.
Related work
Deep learning frameworks
- Deep learning progress has advanced rapidly in the past decade
- Many frameworks were designed for matrix and tensor operations before modern DL
- NumPy is a widely used package for scientific computing
- NumPy does not support automatic differentiation and back-propagation
- Python is the front-runner language for DL interfaces
- There is semantic consistency between the core tensor APIs of modern python DL libraries
Deep learning libraries
- Many field-specific libraries exist for deep learning
- Officially supported framework extensions are becoming common
- Libraries can become obsolete when frameworks become obsoleted
- No solution for building large framework-agnostic libraries for users of both present and future DL Python frameworks
Deep learning abstractions
- Previous works provide framework-level abstractions for DL
- Keras provided abstractions at the level of classes and models
- Ivy simplifies and reduces the abstraction to the core tensor APIs
- TensorLy offers a framework agnostic functional API
- Ivy offers a comprehensive API applicable to a wide variety of fields
Ivy core
- Ivy API provides functions for implementing a variety of examples
- User selects backend framework for Ivy functions
- Ivy functions are unit tested against each backend framework
- Core Ivy API can be extended to include additional functions
Framework-specific namespaces
- Ivy API wraps native functions to provide consistent syntax and call signatures.
- Ivy API extends functionality in cases where native functions are lacking.
- Ivy API follows most common syntax and selects most general variant among backends.
- Ivy API has framework-specific namespaces with updated syntax and call signatures.
- Ivy API makes minor changes to enable direct bindings to native functions.
Framework-agnostic namespace
- Framework-specific namespaces provide unification of syntax and call signatures.
- Framework-agnostic Ivy namespace is used to create new framework-agnostic code.
- Framework-agnostic functions are implemented in two lines of code.
- User specifies which backend framework to use at runtime.
Local framework specification
Global framework setting
- A framework can be set globally for all future function calls using ivy.set_framework().
- The framework can be unset using ivy.unset_framework().
- The framework can also be set globally for a block of code using the with command.
Ivy classes
- Ivy is a fully functional framework
- Ivy provides classes that build on the functional primitives
- Ivy provides trainable neural network layers and stateful optimizers
- High level classes are built directly on top of Ivy’s framework-agnostic functional API
- Details of layers and optimizers are beyond the scope of this paper
Ivy libraries
- Ivy provides a core framework-agnostic API
- Ivy provides four libraries for mechanics, 3D vision, robotics, and differentiable RL environments
- Ivy Mech provides functions for conversions of orientation, pose, and positional representations
- Ivy Vision provides functions for camera geometry, image projections, coordinate frame transformations, forward warping, inverse warping, optical flow, depth triangulation, voxel grids, point clouds, implicit rendering and signed distance functions
- Ivy Robot provides functions and classes for gradient-based motion planning and trajectory optimization
- Ivy Gym provides differentiable implementations of classic control tasks from OpenAI Gym
A spectrum of users
- Ivy can be used in a variety of ways
- Three hypothetical groups of Ivy users: Ivy contributors, Ivy creators and Ivy library users
- Ivy contributors use Ivy Core to develop an Ivy library
- Ivy library users use existing Ivy libraries to supplement their own projects
- Ivy creators use both Ivy core and the Ivy libraries to implement substantial parts of their own personal project in Ivy
End-to-end integration
- Ivy libraries can be integrated into computation graphs for end-to-end training
- Ivy libraries can be used for gradient-based methods outside of deep learning
- Example of combining Ivy libraries for motion planning of a drone in a scene with obstacles
- Code for example provided in Appendix A.3
- Visualization and simulation done with PyRep and CoppeliaSim
Framework evaluations
- Ivy abstraction reduces development time
- LoC analysis shows Ivy and its libraries reduce lines of code
- Runtime analysis of Ivy core assesses overhead of wrapping backend functions
Line of code analysis
- Ivy reduces the lines of code required for developers
- Writing a library once with joint support of all DL frameworks reduces code by 20%
- Ivy libraries offer a variety of commonly used functions in different areas of applied DL
- 100 lines of code required with Ivy and its libraries
- 400 lines of code required without Ivy and its libraries
- 455 lines of code required with Ivy but without its libraries
- 1820 lines of code required without Ivy and its libraries
Ivy core runtime analysis
- 53 of 101 core functions incur no overhead for any backend framework
- Each function is called with inputs of flattened size 1 x 10^4
- Runtime analysis for each core function averaged across backend frameworks
- Ivy overhead is minimal in both compiled and eager mode
- Vision library incurs largest Ivy overhead due to frequent usage of gather and scatter functions
Conclusion and future work
- Presents Ivy, a templated deep learning framework
- Supports TensorFlow, PyTorch, MXNet, Jax, and Numpy
- Offers potential for creating framework-agnostic DL libraries
- Provides four initial Ivy libraries for mechanics, 3D vision, robotics, and differentiable environments
- Encourages developers to join the Ivy community by writing their own functions, layers and libraries
- Will continue extending the derived libraries and adding new libraries for additional research fields
- Will continue developing Ivy Core to remain compatible with all the latest DL framework developments
- Will strive to support the community of open DL research through the framework
- Allows developers to stay in control regarding the depth of the Ivy abstraction in their own projects
- Enables the network to be trained using the native framework’s own optimizers and trainers
- Presents a spectrum of potential Ivy users
- Presents an example application using Ivy
- Presents runtimes for each Ivy core method
- Presents lines of code to implement the demo
- Presents percentage slowdown when using Ivy