Surviving the Hype in Machine Learning and AI
The amount of news and marketing dedicated to Artificial Intelligence and Machine Learning over the past couple of years has been amazing. In my humble view, no single technology has been “sold” with such universal applicability as AI and ML.
We have cheered for success stories where AI is beating masters in games humans play, like the game of Go.
More recently AI beat human pro players in Poker.
In fact, machine learning is at the very peak of inflated expectations according to Gartner’s famous Hype Cycle of Emerging Technologies 2016.
What follows is the Trough of Disillusionment as some over-hyped promises may fail to meet expectations. In some cases, limits and capabilities of the technology may not be well aligned to the problems being solved.
So how does one survive the hype in Machine Learning and AI? Here are a few guiding principles grounded within two decades of emerging technology consulting.
Ecosystem Stack Versus Single Technology Solution
When considering a solution using ML and AI one needs to understand that despite the hype, these technologies do not exist in isolation. A lot of the promise of AI and ML in delivering sophisticated models is reliant on a successful and mature big data infrastructure to be in place.
Most ML algorithms work best when they train on really massive amounts of data. Most AI, ML applications today use a class of algorithms called supervised learning. Supervised learning requires large amounts of correctly labelled data to train on before yielding acceptable predictive analytics. Acquiring large amounts of related data, cleansing it, labeling it accurately, and structuring it consistently requires significant investments, before you even start applying ML algorithms on this data.
This is the reason you see many AI, ML, Data Science job advertisements focus on Big Data infrastructure technologies (Hadoop, Spark, Kafka) as much if not more than AI, ML specific tools and technologies (Python, R, SAS).
Along with data, AI, ML algorithms need lots of computing power for running millions of simulations and processing big data. Specialized chips or Graphical Processing Units (GPUs) designed by Nvidia are just one of the key components on the hardware side. Writing software to run on parallel or distributing computing architectures poses another challenge that needs to be considered.
Let us also appreciate that the underlying ecosystem technologies including the famous Hadoop stack, is going through its own hype cycle and related market corrections for big players like Hortonworks.
In summary, there are number of moving parts to the Ecosystem Stack view for implementing an AI, ML solution. Just the promise of a well marketed ML API or AI framework should not sway your decision in favor of significant investment in this field.
Organization Capability Versus Specialized Function
Most successful companies which have demonstrable AI, ML solutions in the market are built from ground up as data-oriented, Cloud-based, digital businesses adopting and many times inventing latest AI, ML technologies. Google has been building the foundational technologies including Big Data infrastructure for several years now, before taking big bets on AI and ML organization-wide, across most of their products.
When bringing AI, ML to your organization you need careful thought around introducing it as a “Lean Startup” within the “Innovation” function verses treating it as an organization skill and as a large change management program.
If you want to play it safe and take the blue pill then please set your expectations from the results accordingly. Setting up an AI, ML, data science department or function will only yield results which are prototypes, isolated from most of your organization, and at best deliver point solutions.
Taking the red pill is a journey you need to take at organization capability maturity level. It is a fundamental change in your business model in most cases. Before running with a cool voice, speech, or image recognition algorithm, ask yourself this. Is this a core skill for my organization? Will my customers see me as an AI, ML solution provider?
Understandably these are tough calls. I have seen even technology savvy companies, struggle to cross the chasm, changing from ML as a function or department, over to the entire organization as a capability maturity. So, the journey does not become easier if you try the blue pill first!
A middle-ground viable alternative is to consider outsourcing this capability to a boutique consultancy with the long-term view of bringing the “experiment” in-house as it scales.
There are many more guiding principles. I am listing a few here. If there is more interest in this article, please indicate this with likes, shares, and comments.
- Importance of ML Workflow and Tooling
- Adopting Best and Avoiding Worst of R&D Cultures
- Building High-Performance ML, AI Teams
- Productizing ML and AI for Brick-and-Click Businesses
- Industry Specific AI, ML Solutions, and Capability Maturity
- Managing the AI, ML Talent Gap
I will write more on topics that interest my readers.
