Machine Learning Product Management

A short engineering perspective underneath the surface

Flavio Clesio / @flavioclesio

flavioclesio.com / flavioclesio at gmail dot com

07.2020
About me

Flávio Clésio
● Machine Learning Engineer
● MSc. Production Engineering (Computational Intelligence)
● Some of my thoughts flavioclesio.com
● Some conference talks (Strata Hadoop World, Spark Summit,
PAPIS.io, The Developers Conference et aliae)
flavioclesio ● Independent Researcher (Applied Machine Learning) in spare
time

2
Summary
● Artificial Intelligence and Machine Learning

● General Trends in AI and ML

● Current State of Affairs

● Machine Learning Projects: “Unknown unknowns” instead “Known unknowns”

● Elements of Machine Learning Projects

● Machine Learning Projects: Engineering endeavors

● Product Management in Machine Learning

● Machine Learning Projects: Key lessons

3
Disclaimer
Subjects or topics not discussed in this talk:

● Agile / Scrum / Kanban / CMMI Methodologies

● Corporate Culture

● Spotify “Squad Model”

● Slides with Terminator pics

● Regulations in AI
4
Artificial Intelligence
and Machine
Learning

5
What is Artificial Intelligence?

[1] - The 2020 State of AI and Machine Learning Report

6
What is Artificial Intelligence?

[1] - The 2020 State of AI and Machine Learning Report

7
From Artificial Intelligence to Machine Learning

[27] - Two years in the life of AI, ML, DL and Java

8
General Trends in AI
and ML

9
Trend in Research

[28] - Too many machine learning papers?

10
Trend in general interest

[29] - 100+ AI Use Cases & Applications in 2020: In-Depth Guide

11
Trend in Organizations/Corporations: Increase in the
faith that AI can help businesses...

[1] - The 2020 State of AI and Machine Learning Report

12
Trend in Organizations/Corporations: Increase in the
faith that AI can help businesses...

[1] - The 2020 State of AI and Machine Learning Report

13
How much money is in the table?
● Budgets greater than $5M doubled YoY. [1]

● ~$15.7tr Potential contribution to the global economy by 2030 from AI [23]

● +26% Up to 26% boost in GDP for local economies from AI by 2030 [23]

14
How much money is in the table?
● ~45% of total economic gains by 2030 will come from product
enhancements, stimulating consumer demand. This is because AI will drive
greater product variety, with increased personalization, attractiveness and
affordability over time. [23]

● ~45% of work activities could potentially be automated by today’s

technologies, and 80% of that is enabled by machine learning.[10]

15
How much money is in the table?

[30] - The macroeconomic impact of artificial intelligence

16
How much money is in the table?
With all this money over the table, early adopters and big companies are doing
great and everyone else is missing out, right?

Well… Not really…

17
Current State of
Affairs

18
Current State of Affairs (#1)

[11] - Why do 87% of data science projects never make it into production?
19
Current State of Affairs (#2)

[26] - Failure rates for analytics, AI, and big data projects = 85% – yikes!
20
Current State of Affairs (#3)

[1] - The 2020 State of AI and Machine Learning Report

21
Current State of Affairs (#4)
[19] DEC/2019 - 1,388 responses
● Though a problem, the lack of ML and AI skills isn’t the biggest impediment
to AI adoption. Almost 22% of respondents identified a lack of institutional
support as the most significant issue.

22
Current State of Affairs (#5)

[1] - The 2020 State of AI and Machine Learning Report

23
Current State of Affairs

Focus of today’s talk: Machine Learning

Projects from an Engineering
Perspective

24
Machine Learning
Projects: “Unknown
unknowns”
instead “Known
unknowns”
25
Machine Learning Projects: “Unknown unknowns”
instead “Known unknowns”

[31] - The Future of MLOps … and how did we get here?

26
Machine Learning Projects: “Unknown unknowns”
instead “Known unknowns”

[31] - The Future of MLOps … and how did we get here?

27
Machine Learning Projects: “Unknown unknowns”
instead “Known unknowns”

[8] - But what is this “machine learning engineer” actually doing?

28
Machine Learning Projects: “Unknown unknowns”
instead “Known unknowns”

[5] - The AI Hierarchy of Needs

29
Machine Learning Projects: Behavior instead flow
● Software Engineering Projects: Deterministic and transparent representation
of a business flow

● Machine Learning Projects: Sometimes non-deterministic, opaque, and

represents explicit and implicit embedded behaviors from data

30
Machine Learning Projects: Old project
methodologies it’s not enough

[2] - KDD, semma and CRISP-DM: A parallel overview

31
Machine Learning Projects: Uncertainty is the only
certainty
● Challenges abound: non-deterministic outcomes, uncertainty, opacity,
fairness issues, and other factors make AI a difficult sell to decision-makers
and upper management. [4]

32
Machine Learning Projects: Uncertainty is the only
certainty
● Because it’s so different from traditional software development, where the
risks are more or less well-known and predictable, AI rewards people and
companies that are willing to take intelligent risks, and that have (or can
develop) an experimental culture.[4]

33
Elements of
Machine Learning
Projects

34
Machine Learning Projects: Main components

[16] - CD for Machine Learning Automating the end-to-end lifecycle of Machine Learning applications
35
Machine Learning Projects: Technical Roles and
touchpoints

[16] - CD for Machine Learning Automating the end-to-end lifecycle of Machine Learning applications
36
Machine Learning Projects: Touchpoints and
workflow

[16] - CD for Machine Learning Automating the end-to-end lifecycle of Machine Learning applications
37
Machine Learning
Projects:
Engineering
endeavors
38
Machine Learning: Engineering Endeavors
Machine Learning: The High-Interest Credit Card of Technical Debt
● Machine Learning and Complex Systems
● Complex Models Erode Boundaries
● Data Dependencies Cost More than Code Dependencies
● System-level Spaghetti
● Dealing with Changes in the External World [18]

39
Machine Learning: Engineering Endeavors
AI Is Different—Here’s Why
● Most AI models decay overtime
● AI systems are more complex than traditional software
● Because statistics [17]

40
Machine Learning: Engineering Endeavors

[17] - What to Do When AI Fails

41
Product
Management in
Machine Learning

42
Product Management for Machine Learning

[13] - Executive Briefing: Why managing machines is harder than you think
43
Machine Learning Projects: Important questions
needed from Product Managers
● The data PM understands the technological infrastructure involved in
building products at a technical level.
● What kind of infrastructure is needed to support the product?
● Do machine learning models need to be scored in real-time or can they be
prescored offline?
● What is the plan for retraining models on new data?
● How will the model’s success be evaluated over time?
● What is the complexity cost for implementing the model in production? [9]

44
Machine Learning Projects: Management Skills
The Data Expertise of the AI PM
● Skill-Data Lifecycle and Pipeline Management
● Skill-Experimentation and Measurement
● Skill-DS/ML/AI Development Process
● [15]

45
Machine Learning Projects: Management Skills

[13] - Executive Briefing: Why managing machines is harder than you think
46
Machine Learning Projects: Small, fast, interactive
experimentation

[13] - Executive Briefing: Why managing machines is harder than you think
47
Machine Learning
Projects: Key
lessons

48
Machine Learning Projects: Key lessons
[20] […]the key to long-term success for AI companies is to own the challenges
and combine the best of both services and software […]
● Eliminate model complexity as much as possible
● Choose problem domains carefully – and often narrowly – to reduce data
complexity
● Plan for high variable costs
● Embrace services
● Plan for change in the tech stack
● Build defensibility the old-fashioned way

49
Machine Learning Projects: Key lessons
[24] In general, for Machine Learning to make sense for a business, your
problem should have these characteristics:
● Requires complex logic that’s impractical to solve with human-defined rules,
or heuristics.
● The problem will be scaling up very fast.
● Requires personalization at scale.
● Require rules that change quickly over time.
● Has a known, pre-defined end result.
● Does not require 100% accuracy.

50
Machine Learning Projects: Key lessons
[25] Key Challenges to AI Success
● Challenge #1: Data is the key to success, but difficult to harness
● Challenge #2: Data science and engineering silos, resulting in poor
collaboration
● Challenge #3: The explosion of ML frameworks and technologies adds
complexity

51
Closing thoughts

52
Closing thoughts
● There is money on the table
● Lack of AI/ML skills it’s one of the main sources of failure in ML projects
● MVPs, Prototyping, small iterations and if it promising, scale
● Fairness, Transparency, Accountability is a real issue and needs to be
considered in any project
● Real understanding and pragmatism can cut the hype and help Product and
Engineering teams ship

53
 Thank you!
Machine Learning Product Management
A short engineering perspective underneath the surface

Author: Flavio Clesio

flavioclesio.com / flavioclesio at gmail dot com

Presenter: Flavio Clesio / @flavioclesio

Facilitators: [names]

08.07.2020
References
[1] - The 2020 State of AI and Machine Learning Report

[2] - KDD, semma and CRISP-DM: A parallel overview

[3] - Machine learning requires a fundamentally different deployment approach

[4] - What you need to know about product management for AI

[5] - The AI Hierarchy of Needs

[6] - Rules of Machine Learning: Best Practices for ML Engineering

[7] - Managing Machine Learning Projects

[8] - But what is this “machine learning engineer” actually doing?

[9] - Rise of the Data Product Manager

55
References
[10] - The First Wave of Corporate AI Is Doomed to Fail

[11] - Why do 87% of data science projects never make it into production?

[12] - Machine Learning Product Management: Lessons Learned

[13] - Executive Briefing: Why managing machines is harder than you think

[14] - What you need to know about product management for AI

[15] - Practical Skills for The AI Product Manager

[16] - Continuous Delivery for Machine Learning Automating the end-to-end lifecycle of Machine Learning applications

[17] - What to Do When AI Fails

[18] - Machine Learning: The High-Interest Credit Card of Technical Debt

56
References
[19] - AI adoption in the enterprise 2020

[20] - The New Business of AI (and How It’s Different From Traditional Software)

[21] - AI Playbook

[22] - A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence

[23] - PwC’s Global Artificial Intelligence Study: Exploiting the AI Revolution

[24] - When and How to Add Machine Learning to a Product Roadmap

[25] - CIO Survey: Top 3 Challenges Adopting AI and How to Overcome Them

[26] - Failure rates for analytics, AI, and big data projects = 85% – yikes!

[27] - Two years in the life of AI, ML, DL and Java

57
References
[28] - Too many machine learning papers?

[29] - 100+ AI Use Cases & Applications in 2020: In-Depth Guide

[30] - The macroeconomic impact of artificial intelligence

[31] - The Future of MLOps … and how did we get here?

[32] - The age of analytics: Competing in a data-driven world

58