As Silicon Valley is the main cradle of the largest high-tech corporation, including Apple, it’s only fitting to have a Silicon Valley perspective on where we stand with digitalisation and emerging technologies like AI, machine learning and deep learning. Bhairav Mehta, Data Science Manager at Apple gave a comprehensive overview of the latest developments and trends, but also covered important historical breakthroughs that lead to the current state of AI and ML at the Data Innovation Summit 2019.
The last decade of the 20th century gave rise to the digital revolution brought on by the internet and digital computation technologies. The digital economy is undermining and at the same time reinventing the traditional way of doing business. It also provides a prolific ground for the propagation of these technologies that are changing the way we live and work.
The digital economy gave birth to what Bhairav refers to as the Unicorn Club of companies, not exclusive only to Silicon Valley, but covering the whole world. These are companies with more than a billion dollars valuation that started around 2014. What happened then to trigger this flood of unicorn startups that totally disrupted the economy?
There were many technological developments happening at that period, Bhairav explains, such as the rise of 4G, the cloud and the algorithmic and data explosion, that are responsible for creating the unicorns, Apple being one of them.
All the advanced technologies that drive the digital economy today don’t exist separately, they are a part of the digital correlation – an umbrella concept that covers cloud computing, artificial intelligence, Internet of Things, fog computing and data analytics.
“These are the five industries that are driving the industry today, especially the data economy we are living in”, specifies Bhairav.
A lot of buzz is surging around these technologies, some of them hold the top spot while some of them are fleeting, according to Gartner Hype Cycle. Deep learning held the Hype Cycle thone in 2018, but it was overthrown by 5G in 2019. However, Bhairav’s presentation focuses on the deep learning techniques that marked 2019.
Who created AI
There are several distinguished persons throughout history, known as the Godfathers of AI, who are credited for establishing the foundation and contributing to AI and deep learning as they are today. These are some of the most important milestones for AI in history:
- 1958 – Frank Rosenblatt created the single-layer perceptron
- 1995 – Vladimir Vapnik co-inventor of Kernel methods (support vector machine method)
- 1998 – Yann LeCun created of convolutional networks
- 2006 Geoffrey Hinton invented fast learning algorithms for the Restricted Boltzmann machine
- 2007 – Yoshua Bengio proposed the Stacked Auto-encoders
- 2012 – Google Brain’s system of 16,000 computers successfully trained itself to recognise a cat based on cat images from Youtube
- 2014 – Ian Goodfellow and his colleagues invented the generative adversarial network (GAN).
Types of machine learning algorithms
Before doing deeper into the latest developments of deep learning, Bhairav laid out some machine learning 101 fundamentals.
He explains the 5 tribes of machine learning algorithms and the favoured algorithms among them today:
- Symbolists – rules and decision trees
- Bayesians – Naive Bayes or Markov
- Connectionists – Neural networks
- Evolutionaries – Genetic programms
- Analogizers – Support vectors.
As the main focus of Bhairav presentation is deep learning, we are shifting the attention to deep learning applications and neural networks.
Computer vision and convolution neural networks
Computer vision today has surpassed the human performance zone with regard to the error rate in image classification, states Bhairav. Computer vision is currently at 2-3% error rate, while the human’s performance zone is around 5-10%. This advancement in computer vision has helped solve all the advanced unstructured data problems.
If humans perceive around 4 terabytes of content every second, how much of that information can be trained through neural networks, Bhairav asks?
When people are looking at an object, they immediately tell objects apart pretty easy.
However, image perception is no so simple for computers.
When a convolutional network sees an object, it has to break it down into layers:
1) Visible layer (input pixels)
2) First hidden layer (edges)
3) Second hidden layer (corners and contours)
4) Third hidden layer (object parts)
And as a result, the neural network identifies and classifies it as a person, an animal or an object.
Region proposal based CNN (R-CNN)
With regional proposal based convolutional neural network we are able to segment and identify multiple objects in an image. For this process to take place, we need region proposals which are a sort of lookup tables that are fed into the model so it’s able to identify the specific objects we are looking for in the image. The biggest use case of region proposal based CNNs is in self-driving cars where objects are identified through a smart camera.
Text analytics with natural language processing (NLP)
The text analytics sphere has been undergoing some substantial revolution, adds Bhairav. One major shift is applying standard NLP and NLU model to deep learning by creating word embeddings layers or word vectors.
There are pre-trained models for text classification, much like the pre-trained computer vision models, that can do unsupervised text classification. They can classify and cluster the text together based on the sentiment and colour-code it.
For each text block, sentiment can be identified. For that purpose, a word vector and word embeddings are created and the inferred embedding numbers are fed into neural networks just like pixels in images to train a sentiment engine with them.
Federated learning model
Bhairav rounds up his discussion with federated learning, which refers to how our smartphones can be used in model training and processing. As Bhairav points out most of the phones are equipped with AI chips and high memory of several gigabytes or even terabytes and yet 2.3 billion phones are underutilised in terms of processing capacity.
These high-performance smartphones can be used to train machine learning models. Thanks to the unique user experience in every phone, the models can be enriched based on the usage of the customer. This way a federated machine learning model can be created which provides a custom experience for each phone, and every phone, in turn, sends out its model which improves the training process, emphasises Bhairav.