MIT develops NAS algorithm that is 200 times more efficient than conventional methods

Current NAS techniques are cumbersome and costly

Recently, developers have been using algorithms to automatically design artificial neural networks that are more efficient than human-designed neural networks.  The technique, referred to as neural architecture search (NAS), produces artificial neural networks that are more efficient and more accurate than the networks designed by humans.

But NAS is costly and it takes thousands of graphical processing unit (GPU) hours to produce a neural network.  As an example, it takes Google around 48,000 GPU hours to produce a conventional neural network (CNN) such as the ones commonly used for image classification.

New NAS algorithm will be an AI game-changer

Now, researchers at MIT have developed an NAS algorithm that can design machine-learning models in a fraction of the time.  The algorithm is capable of learning specialized CNNs for target hardware platforms in just 200 GPU hours.

The research was presented in a paper authored by Song Han, Han Cai and Ligeng Zhu, to be presented at the International Conference on Learning Representations this May.

NAS algorithm uses a variety of techniques to speed up processing

An NAS works by constructing what it deems the most efficient architecture within a given "search space."  In this case, the researchers used the ImageNet database to train the architecture.  But instead of letting the algorithm proceed as usual by considering all the possible paths, they allowed it to drastically pare down the number of paths using path-level binarization and pruning.  The algorithm only keeps paths that are optimized for accuracy and efficiency, immensely simplifying the CNN architecture and cutting down the amount of time and memory needed to complete the task.

The outputted CNNs also run faster than their predecessors across all platforms – 1.8 faster, for example, when tested on mobile phones.  The algorithm is capable of measuring the latency of a device without needing an entire “farm” of them.  It then leverages the latency and uses it to optimize the architecture.

Interestingly, some of the CNN architectures designed by the NAS algorithm were in styles that had been dismissed by researchers because they were thought to be inefficient.  The algorithm discovered they were in fact suitable to specific types of hardware.  Just another case of AI and human ingenuity working together!

New NAS algorithm will democratize AI

Whereas currently only massive companies like Google can afford the time and money to produce artificial neural networks using NAS, the new algorithm developed by MIT will enable smaller start-ups to make use of NAS.  Song Han says the researchers hope to “democratize AI” by “[enabling] both AI experts and nonexperts to efficiently design neural network architectures with a push-button solution that runs fast on a specific hardware.”

He added that while NAS algorithms cannot replace human engineers, the researchers hope the NAS algorithm will be able to take on some of the more repetitive work.  This would leave more time for human researchers to perform higher-level adjustments.

The new NAS algorithm is the latest in a string of recent innovations such as Google's Coral Dev Board and USB accelerator for the Edge AI TPU chip that are pushing AI closer and closer to being broadly accessible.