{"id":17998,"date":"2022-02-25T13:47:17","date_gmt":"2022-02-25T18:47:17","guid":{"rendered":"https:\/\/www.crim.ca\/blogue\/contributing-to-libtorch-recent-architectures-and-vanilla-training-pipeline\/"},"modified":"2023-06-14T10:03:39","modified_gmt":"2023-06-14T14:03:39","slug":"contributing-to-libtorch-recent-architectures-and-vanilla-training-pipeline","status":"publish","type":"blogue","link":"https:\/\/www.crim.ca\/en\/blogue\/contributing-to-libtorch-recent-architectures-and-vanilla-training-pipeline\/","title":{"rendered":"Contributing to LibTorch: recent architectures and \u201cvanilla\u201d training pipeline"},"content":{"rendered":"

In August 2021, a\u00a0PR<\/a>\u00a0aimed at adding a SOTA architecture (namely EfficientNet) to TorchVision, a Python-based PyTorch package for computer vision experiments, was submitted on GitHub. Even though deep learning practitioners are used to testing new architectures that are regularly posted on this platform, this is certainly a welcome contribution. On the other hand, C++ contributions to PyTorch (or more precisely to LibTorch, which is PyTorch\u2019s C++ API) and TorchVision from the official maintainers are limited, particularly for independent contributors, so the need for new contributions is even greater.<\/p>\n

This post describes the C++ implementation of a pair of recent architectures, EfficientNet and NFNet, as well as a testing tool that uses these architectures. The whole package is\u00a0available on GitHub<\/a>.<\/p>\n

The proposed package represents a wider pedagogical contribution aimed at providing a concrete example of development of a training pipeline using LibTorch.<\/p>\n

Implementing EfficientNet<\/h2>\n

In 2019, researchers from Google Brain used neural architecture search to design a new baseline network called B0, whose scalable properties (width, depth, image resolution) allow the creation of a family of models (from B1 through B7) called EfficientNets. They define a compound parameter alpha that helps determine the depth and input resolution of the most appropriate architecture with respect to available resources, while also setting the width of the convolutional filters layers making up each layer.<\/p>\n

The architecture consists of 9 stages, each of which being made of 1 to 4 layers (thick blue rectangles in the Figure below), where a layer is either a convolutional layer or a series of \u201cmobile inverted bottleneck blocks\u201d (MBConv blocks). The term MBConv6 means that six of those blocks make up that particular layer. Note the stride error flagged by the red rectangle (28x28x80 should be 14x14x80, see\u00a0https:\/\/github.com\/lukemelas\/EfficientNet-PyTorch\/issues\/13<\/a>).<\/p>\n

\"\"<\/p>\n

This C++ Implementation of the architecture with LibTorch closely follows Lukemelas\u2019\u00a0PyTorch implementation<\/a>. At the time of writing, the swish() activation function wasn\u2019t available in LibTorch, so a basic implementation is provided in the source code.<\/p>\n

Implementing NFNet and Gradient Clipp<\/h2>\n

The NFNet family of architectures has attracted\u00a0a lot of attention<\/a>\u00a0recently due to its SOTA performance on the ImageNet dataset while avoiding the use of batch normalization. To do so, the authors draw not only on their previous work on Normalizer-Free ResNets, but they also make a number of contributions in order to stabilize and optimize the new architecture:<\/p>\n