Downloaded 16 times
![Themes inspire Initiatives & Epics for the Backlog
Ensure
compliance
Achieve Compliance objective by adhering to anti-money laundering
(AML) regulations; Applying machine learning to automatically monitor
customer transaction data to identify anomalies for investigation.
Resulting in reduced risk of fines; Partnering with Compliance and IT
teams; and Integrating financial transaction and customer identity data.
Achieve [Objective] by [Target outcome]; Proposed [Hypothesis];
Resulting in [Predicted Benefit]; Partnering with [Other Teams and
Partners]; and Integrating [sources and data types].
Develop multiple Initiatives. There is more than one way to achieve
objectives.
Achieve Compliance objective by ensuring information security;
Applying natural language processing and machine learning to
automatically identify sensitive content and off-policy distribution
for review; Resulting in reducing risk of fines and reputational risk;
Partnering with Compliance and IT teams; and Integrating enterprise
email data.](https://image.slidesharecdn.com/dataleaderssummit2018final-181023090101/75/Data-Leaders-Summit-Barcelona-2018-24-2048.jpg)
![Objectives
Treat each prioritised Epic as a
hypothesis to be tested
Think of
Interesting
Epics
(Formulate
Hypotheses)
Prioritise
Epics
Create
Stories &
Test
Refine,
Alter,
Expand, or
Reject
Hypotheses
Gather
Data to test
Predictions Develop Testable Predictions
(If my hypothesis is correct
then I expect [benefit])
Pilot
Experiment
Measure
Embed](https://image.slidesharecdn.com/dataleaderssummit2018final-181023090101/75/Data-Leaders-Summit-Barcelona-2018-33-2048.jpg)
Uploaded byHarvinder Atwal
Data Leaders Summit Barcelona 2018
The document outlines strategies for leveraging machine learning and intelligent automation to enhance business efficiency and productivity. It discusses the importance of aligning data science initiatives with corporate objectives, employing various data-driven methodologies to improve customer experience, compliance, and operational performance. The document emphasizes the need for agile collaboration and effective data governance to create a culture of measurement and continuous feedback in data-driven organizations.
More Related Content
Similar to Data Leaders Summit Barcelona 2018
![Strategic Picks — Prioritising the Right Agentic Use Cases [2/6]](https://cdn.slidesharecdn.com/ss_thumbnails/session-2-250902135931-4c97717a-thumbnail.jpg?width=640&height=640&fit=bounds)
Data Leaders Summit Barcelona 2018
- 1. Case Study Interactive:Productionising machine learning to automate the enterprise: How can you pin-point which core business processes to transform with increased automation and streamline daily workflows to boost in-house efficiencies?
- 2. // Harvinder Atwal MoneySuperMarket //Web dunnhumby {"previous" : "Insight Director, Tesco Clubcard"} Lloyds Banking Group {"previous" : "Senior Manager, Customer Strategy and Insight"} {"Current" : "Head of Data Strategy and Advanced Analytics"} @harvindersatwal British Airways {"previous" : "Senior Operational Research Analyst"} {"about" : "me"} @gmail.com
- 3. £2B SAVINGS 2017 estimate totalof UK savings 1993 24.9M 24 million £323M 989 We started life as mortgages 2000 Adults choose to share their data with us Average monthly users 2017 Revenue 2017 Product Providers
- 4. 3 major waysData Science can help the organisation Product Creation Customer Experience Business efficiency
- 5. Intelligent Automation isa solution to improving Business Efficiency Robotic Process Automation Machine Learning Artificial Intelligence Business rules to execute tasks with existing software systems. Systems performing tasks normally requiring human intelligence Predictive and Prescriptive Analytics driven decision making Act Like a Human Think Like a Human
- 6. Robotic Process Automation(RPA) is process rather than data-driven Read Email Open Attachment Enter Data into ERP System
- 7. “The science ofgetting computers to act without being explicitly programmed” – Andrew Ng 7 Output Output Machine Learning AlgorithmRules f(x) Input Data Code Regular programming Machine Learning Input Data Rules f(x) Robotic Process Automation is really Software Engineering
- 8. Our Marketing Personalisationrelies on Intelligent Automation through Prescriptive Analytics From one version to 1400+ customised variants of the newsletter +19% Increase in Revenue Per Send
- 9. Intelligent Automation throughML Anomaly detection prioritises alerts Facebook use ML to review code releases Automated Maintenance Schedules
- 10. Can’t we just AIit? WT F AI solves all some problems
- 11. We use NaturalLanguage processing to customise content Worried about whether you can afford a personal loan? With UK interest rates at record lows, it’s worth checking to see how reasonable the cost could be. Whether you need to borrow to buy something, or you want to bring your existing debts under one roof, have a look at these competitive deals we’ve assembled. Thanks to our Smart Search tool, you can get an idea of the loans you’re likely to be accepted for before you proceed with your application. Same message but Language tailored to the customer’s Financial Attitude
- 12. Associated Press useNLP tools to create articles quickly, such as business earnings reports and localized election coverage Classify documents according to their meaning and relevance to ongoing litigation Automated Claims handling and Fraud detection in Insurance
- 14. Algorithms are becomingless and less important
- 15. Automated Claims Processing AutomatedProduct Descriptions Automated Trend Detection at Unilever Automated Image Selection at AirBnb & TripAdvisor
- 16. Netflix - AutomatedArtwork HireVue – Automated Interviews Factory Automation at Kawasaki Heavy Industries
- 17. Ideas aren’t inshort supply Demand Forecasting Capacity Forecasting Marketing automation Supply chain management, automatic ordering Automatic scaling of infrastructure Document Classification Image Annotation Customer Service Machine Translation Anomaly Detection Product Recommendation Fraud Detection Image Selection Text Generation Predictive Maintenance Automated Pricing Automated routing Medical diagnosis
- 18. Deciding what to workon – A better way
- 19. Data Science can’thappen in a vacuum Situational Awareness is needed
- 20. Alignment of datascience with the rest of the organisation and it's goals
- 21. Your business alreadyhas hypothesis for what creates value Actively avoid work on anything else It’s the Corporate Strategy and Objectives (everyone is aligned behind)
- 22. Measurement of everythinggives feedback of not just individual deliverables (fast loop) but also the organisation’s hypothesis of what adds value (slow loop) Situational Awareness Themes (Objectives) Initiatives Epics Stories Initiatives Initiatives Themes (Objectives) Epics Epics Epics Epics Epics Stories Stories Stories Stories Stories Stories Stories Stories Stories Stories Stories Corporate strategy is broken down into many options (Epics) for Agile delivery
- 23. Improve employee retentionEnsure compliance Increase reliability of operations Capitalise on physical facilities Reduce energy usage per unit of production Improve and maintain workplace safety Reduce error rates Improve customer retentionImprove customer satisfaction Improve customer serviceAcquire new customers from innovative offerings Grow percentage of sales from new products Differentiate the product Understands customer needs Increase share of wallet Increase share of market Cross-sell more products Reliable products/services Grow Revenue Increase Margin Reduce Costs Work on the Right Things: Organisational Objectives are your Themes
- 24. Themes inspire Initiatives& Epics for the Backlog Ensure compliance Achieve Compliance objective by adhering to anti-money laundering (AML) regulations; Applying machine learning to automatically monitor customer transaction data to identify anomalies for investigation. Resulting in reduced risk of fines; Partnering with Compliance and IT teams; and Integrating financial transaction and customer identity data. Achieve [Objective] by [Target outcome]; Proposed [Hypothesis]; Resulting in [Predicted Benefit]; Partnering with [Other Teams and Partners]; and Integrating [sources and data types]. Develop multiple Initiatives. There is more than one way to achieve objectives. Achieve Compliance objective by ensuring information security; Applying natural language processing and machine learning to automatically identify sensitive content and off-policy distribution for review; Resulting in reducing risk of fines and reputational risk; Partnering with Compliance and IT teams; and Integrating enterprise email data.
- 25. HIRE DATA SCIENTISTS How businessesthink they become data-driven 1 2 3 MONEY FLOWS HOARD DATA 4
- 26. We know there’smore to it Opportunity Data Modelling Deployment Benefits Realisation Refresh How well do we plan and prioritise the projects we work on? How aligned are plans to the business objectives, stakeholders and customer needs identified earlier? How integrated is our Internal data? How structured and clean is our data? How secure is our data? How real-time is our data? How well do we track the benefits of our work? How effective is the feedback loop to new opportunities? How good is our ability to sell our work? How easily can we create and validate models? Do we have access to a wide range of Algorithms? Do we have monitoring in place? Are models explainable? How easily can we deploy data products across touchpoints? How quickly can we create and update reporting? How well do we drive business and customer change? How well do we review/revalidate our output and processes? How often do we reuse existing processes? How easily can we launch experiments? How easily can we update existing data products?
- 27. Meet their needs •Engage and consult on interest area • Increase level of interest • Move to right hand box Key Player • Focus on this area • Involve in Governance • Engage and consult regularly Least Important • Inform via General Communications • Increase level of interest • Aim to move to right box Show Consideration • Involve in low risk projects • Keep informed and consult • Potential supporter Work with the right people STAKEHOLDER POWER STAKEHOLDER INTEREST STAKEHOLDER POWER STAKEHOLDER INTEREST Which stakeholders are more likely to respond to your recommendations and requests?
- 28. Create an Epicprioritisation Matrix Objective Epic Stakeholders Power and Interest Data Availability, Quality and Integration Resour ces Deployment Capability Benefits Measurement Expected Business Value Ensure compliance Applying machine learning to automatically monitor customer transaction data to identify anomalies. Resulting in reducing risk of fines H/L H/H/M M L H 5 Applying natural language processing and machine learning to automatically identify sensitive content and off-policy distribution; Resulting in reducing risk of fines and reputational risk. H/M L/M/L H L M 3 Improve customer retention Applying machine learning to identify customers at risk of churn. Resulting in higher retention. M/H M/H/M M M H 6 Applying machine learning to identify customer LTV for differentiated service. Resulting in higher retention. M/H M/H/M M M H 5 Improve customer service Applying natural language processing to prioritise customer messages for resolution. Resulting in higher NPS. M/H M/H/M M M H 2 … ….. ../.. ../../.. .. .. .. ..
- 29. Reduce Batch sizesof work and have options to keep flow going
- 30. Get Specific As aData Engineer I need to create a data pipeline of transaction data for Data scientists As a Data Scientist I need to train and validate a machine learning model for accurate predictions As a Data Engineer I need to operationalise and deploy the machine learning to make predictions As a Data Scientist I need to explore data to understand the quality and value for modelling As a Data Scientist I need to engineer features to improve model accuracy As a Data Scientist I need to monitor model performance to know when to retrain the model Big, coarse grained Small, specific Applying machine learning to automatically monitor customer transaction data to identify anomalies. Resulting in reducing risk of fines
- 32.
- 33. Objectives Treat each prioritisedEpic as a hypothesis to be tested Think of Interesting Epics (Formulate Hypotheses) Prioritise Epics Create Stories & Test Refine, Alter, Expand, or Reject Hypotheses Gather Data to test Predictions Develop Testable Predictions (If my hypothesis is correct then I expect [benefit]) Pilot Experiment Measure Embed
- 34. There’s one processthat needs to be transformed before all others
- 35. model.fit(X_train, y_train) is actuallythe easiest part
- 36. Only 22% ofcompanies are currently seeing a significant return from data science expenditures* *Obligatory conference presentation quote from GartnerForresterMcKinsey Consulting. Sorry.
- 37.
- 38. Multiple challenges inthe process of turning data into value on existing infrastructure Business Problem Evaluate available data Request Data Access from IT Request Compute Resources from IT Negotiate with IT for requested resources Wait for resources to be provisioned Install Languages and tools Configure connectivity, Access and security RAM/CPU Availability, scaling, monitoring Request network Config Change Request to install another package Model building Compose PowerPoint to share results Edit Confluence to document work Negotiate with business stakeholder on deployment timeline Wait for Data Engineering to implement the model Test Newly implemented model to ensure valid results Request Modifications to model due to unexpected results Release model to production and schedule Document release notes and deployment steps Prepare for change management
- 39. Data Science trappedin laptops
- 40. Thinking real-life DataScience is a Kaggle competition
- 41. Treating Data Scienceas a Death Star Technology Project
- 42. DATA STORAGE Cloud File StorageDistributed File System NoSQL DB RDBMS COMPUTE INFRASTRUCTURE ResourceManagement/Monitoring/Auditing Scheduling ProjectandDataGovernance DataEngineering Distributed SQLQuery Engine Distributed Compute Framework Compute Instances Coding Workspace & Language Libraries Output Files ANALYTICS LAYER Machine Learning libraries Data Visualisation libraries BI Tools Interactive dashboards/ Web Apps Security/IdentityAccesscontrol APIs Data Prep/Exploration tools Summary Analysis, Analysis of Experiments, Segmentation, Machine Learning, Data Matching Revision/Deployment Tools Interactive Dashboards/ Web App development Applications (Business Layer) Insight Marketing Optimisation External Data Products Internal Reporting Website Optimisation Commercial Optimisation Production Code DevOps/Infrastructure DBAs ETL DQM MetadataManagement Agile Data Science does not solve tech complexity problem Container Service Resource Vertical requirementsDATA PRODUCTS (Presentation/ Service Layer) Deployment,OrchestrationandScaling ConfigurationManagement RevisionControl KnowledgeManagement DataScientists DATA SOURCES Stream Processing Framework
- 43. Insight does notscale!. Using data to generate ad hoc Decision Support Insight INSTEAD OF ACTION
- 44. Money is wasted Timeis wasted Talent is wasted
- 45.
- 46. Collaboration is key SharedBuy-in from Senior management Organizational behavior structured around the ideal data-journey model Shared Priorities Shared Trust in data Shared Rewards based on measured outcomes, not outputs
- 47. Test & Collect Model EmbedRoll Out Feedback Plan Pilot test Collect Data Build Model, Identify segments Adjust model to fit organisation Re-engineer business processes to support segmented execution Train organisation Creation of fast feedback loop
- 48. Data cycles aremeasured to eliminate bottlenecks
- 49. Shortened Data Cyclesto be Agile Data Engineering Dev Ops/Infrastructure DB Management Cloud File Storage Distributed File System NoSQL DB RDBMS Distributed SQLQuery Engine Distributed Compute Framework Compute Instance Container Service Data Prep/ Exploration tools Coding Workspace & Language Libraries Machine Learning Data Visualisation Interactive Dashboards/ Web App development Version/ Deployment Tool Output Files BI Tools Interactive dashboard s/Web Apps APIs Knowledge Management Security/Identity Access control Revision Control Configuration Management Orchestration and scaling Project and Data Governance Scheduling Resource Management/Monitoring/Auditing ETL DQM Data Scientists Epic Customer Feedback & Iteration Data Product Strategy Story Stream Processing Data Sources
- 50.
- 51. DataOps was popularisedby Andy Palmer in a 2015 Blog post
- 52. DataOps is anindependent approach to data analytics Data Analytics team moves at lightening speed using highly optimized tools and processes across the whole data lifecycle Agile Collaboration to break down silos and work on “The Right Things” that add value Lean Manufacturing like focus on eliminating waste & bottlenecks, improving quality, monitoring and control Iterative project management Continuous delivery Automated test and deployment Monitoring Self-serve Quality Governance Organisational alignment Ease of use PredictabilityReproducibility Strategic Objectives
- 53.
- 54. Why do wehave brakes on a car?
- 55. Accept the deliverypipeline is governed by rules and constraints
- 56. Trust part 1:Make the “What you do to data” people in the organisation happy Identity and Access Management Custom role permissions Audit trail logs Data Loss Prevention Encryption of Data at Rest Encryption of Data in Motion Resource Monitoring Firewall rules Resource and Object Isolation Penetration Testing Code Encryption and Backup Segregation of Duties Authorisation protocols Data Access and Privacy Policy Metadata Management Data Lineage Tracking Data Stewards and Owners
- 57. Trust part 2:Make the “What you do with data” people in the organisation happy Data Quality Testing Transformation Testing End-User Testing ETL Integration Testing Metadata Testing Data Completeness Testing ETL Regression Testing Incremental ETL Testing Reference Data Testing ETL Performance Testing
- 58.
- 59. Configuration Management For consistentlyreproducible computational environments
- 60. Continuous Integration: CommitCode Regularly Data Cleaning Master Data Cleaning Dev Branch Feature Extraction Dev Feature Extraction Master Model Train Master Model Train Dev Branch Machine Learning Pipeline Product Development (e.g. App, Website, Marketing system, Operational System, Dashboard, etc.)
- 61. Run tests andreview code (please integrate safely)
- 62. Continuous Delivery andBeyond: Accelerating Deployment Dev Integration testApplication test Acceptance test Production Continuous Integration Dev Integration testApplication test Continuous Delivery Dev Integration testApplication test Acceptance test Production Continuous Deployment Automated Manual
- 63. Chemistry is notabout tubes DataOps is not about tools (but the right ones help)
- 64. Align your spine Needs Principles Practices Tools Values Howdo you know it is the best possible tool? How do you know that the Practices actively help the system? How do you know which Principles you want to apply? “We use _____ to get our work done” “We DO Self-Service and DataOps to continuously create VALUE for the customer and business” We LEVERAGE Agile and Lean PRINCIPLES to change the system and make sure resources work on the right thing We OPTIMISE for Speed, Accuracy, Experimentation/Feedback and Security. We are here to SATISFY THE NEED to help customers save money and the business to execute it’s strategy It all starts at Needs. Why does this system exist in the first place? Source: Kevin Trethewey, Danie Roux, Joanne Perold
- 65. Avoid building yourown anything or being on the bleeding edge. Cost of Delay is high.
- 66. Data Scientists needa way to manage their projects end-to- end with self-service data AND ARCHITECTURE Business Problem Evaluate available data Request Data Access from IT Request Compute Resources from IT Negotiate with IT for requested resources Wait for resources to be provisioned Install Languages and tools Configure connectivity, Access and security RAM/CPU Availability, scaling, monitoring Request network Config Change Request to install another package Model building Compose PowerPoint to share results Edit Confluence to document work Negotiate with business stakeholder on deployment timeline Wait for Data Engineering to implement the model Test Newly implemented model to ensure valid results Request Modifications to model due to unexpected results Release model to production and schedule Document release notes and deployment steps Prepare for change management
- 67. Modern serverless andmanaged infrastructure makes it easy to create data products just bring code and data A single unified platform reduces data fragmentation, overcomes business silos and helps enforce consistent governance
- 68. Data Science Platformsadd further self-serve capabilities Data Access, Prep and Exploration Jupyter, Rstudio, Zeppelin, etc. Automation and Machine Learning Run experiments, track and compare results Delivery and Model Management Publish APIs, Interactive web apps Schedule reports Collaboration and Version Control Discover, discuss and build on existing work Compute Environment Library Customised software stack Compute Grid Orchestrate hardware for development and deployment Source: Domino Data Labs
- 69. The market forplatforms is exploding
- 70. Data Scientist Data Analyst Data Engineer Self-serve enables reducedDataOps roles ETL Quality Testing Descriptive Analytics Advanced Analytics BI Dev Ops IT Ops DBAs X X X Business Stakeholders Operations Sys adminX Developers ML Product Managers
- 71. Implement AI: ActionableIntelligence
- 72. #1 Align withthe Organisation through Agile Collaboration
- 73. Find the FASTEST,CHEAPEST path between data and consumers #2 Eliminate wasted effort The Optimist The Pessimist The Lean Thinker THE GLASS IS HALF FULL THE GLASS IS HALF EMPTY WHY IS THE GLASS TWICE AS BIG AS IT SHOULD BE?
- 74. #3 Deliver Products not Projects Prioritize solutionsthat fit into a DataOps workflow over others
- 75. #4 Build ameasurement and feedback culture
- 76. #5 Embrace Development best practisein Data Science Version Control, Configuration management, Continuous Integration, Continuous Operations
- 77. #6 KEEP CALM AND BUILDTRUST IN DATA Put Effective Data Governance, Security and Testing in place
- 78. #7 Invest intools and process to reduce bottlenecks and increase quality Managed Infrastructure and Serverless Cloud, Automation and Data Science Platforms
- 79. #8 Decentralise Self-serviceanalytics AND cloud infrastructure
- 80. #9 Organise aroundthe ideal data journey instead of teams Fewer roles, more end-to-end ownership, less friction Store Share UseManageAcquire Process Data Engineering Data Scientists Data Analysts Business Stakeholders
- 81.
- 82. Data Science Today Customer Data ? Hamster wheel Analytics The Roadblock TheAimless crash and burn The “So What happened?” The “We did it once, why doesn’t it work again?”
- 83. The DataOps DataScience Factory Epic Customer Data Product Strategy Story Data Rest of Business Analytics Agile Collaboration Data Governance Automated testing Value Measurement Version Control Configuration Management Self-Serve Infrastructure Automation Continuous Integration
- 84.
- 86. // Harvinder Atwal// Web var current: { companyName : "MoneySuperMarket", position : "Head of Data Strategy" + " and Advanced Analytics" }; var previous1: { companyName : "Dunnhumby", position : "Insight Director," + " Tesco Clubcard" }; var previous2: { companyName : "Lloyds Banking Group", position : "Senior Manager" }; var previous3: { companyName : "British Airways", position : "Senior Operational Research Analyst" }; {"about" : "me"} var username = "harvindersatwal"; var linkedIn = "/in/" + username; var twitter = "@" + username; var email = username + "@gmail.com";
Editor's Notes
- #7 Process automation can expedite back-office tasks in finance, procurement, supply chain management, accounting, customer service and human resources, including data entry, purchase order issuing, creation of online access credentials, or business processes that require “swivel-chair” access to multiple existing systems Automated processes in the remote management of IT infrastructures can consistently investigate and solve problems for faster process throughput. RPA can improve service desk operations and the monitoring of network devices. As in voice recognition software or automated online assistants, developments in how machines process language, retrieve information and structure basic content mean that RPA can provide answers to employees or customers in natural language rather than in software code. This technology can help to conserve resources for large call centers and for customer interaction centers.
- #9 Content, Timing
- #13 http://www.slate.com/articles/technology/technology/2014/07/automated_insights_to_write_ap_earnings_reports_why_robots_can_t_take_journalists.html?via=gdpr-consent https://www.shift-technology.com/solutions/#force
- #16 https://motionscloud.com/
- #17 https://medium.com/netflix-techblog/ava-the-art-and-science-of-image-discovery-at-netflix-a442f163af6
- #21 The key to adding value is to adapt and borrow principles from Agile Software development starting with alignment of data science with the rest of the organisation and it's goals.
- #22 Work only on the organisation’s biggest strategic objectives – those stakeholders have aligned behind. Objectives the business hypothesises will add the most value.
- #23 We don’t know upfront what is going to work.
- #28 Who is your ideal internal stakeholder/client? Which ones are more likely to respond to your recommendations and requests? What is their persona(s) Who are your key stakeholders and what stage of the data-driven enterprise buy-in process are they at? Who are you influential stakeholders? Who else is talking about your team/output (and what are they saying)? Who are your actual clients? Does this fit with our ideal client? Do we need to change who we serve if they are different?
- #29 Lots of ways to prioritise Sometimes you actually have some qual/quant data to estimate expected business value. Else you have to estimate.
- #42 Star wars is not a metaphor for good vs Evil but Waterfall vs Agile
- #44 Too much wastage in the process and hard to impact customers directly
- #53 DataOps is an automated, process-oriented methodology, used by big data teams, to improve the quality and reduce the cycle time of data analytics. While DataOps began as a set of best practices, it has now matured to become a new and independent approach to data analytics.[1] DataOps applies to the entire data lifecycle[2] from data preparation to reporting, and recognizes the interconnected nature of the data analytics team and information technology operations.[3] From a process and methodology perspective, DataOps applies Agile software development, DevOps[3] and the statistical process control used in lean manufacturing, to data analytics.[4] In DataOps, development of new analytics is streamlined using Agile software development, an iterative project management methodology that replaces the traditional Waterfall sequential methodology. Studies show that software development projects complete significantly faster and with far fewer defects when Agile Development is used. The Agile methodology is particularly effective in environments where requirements are quickly evolving — a situation well known to data analytics professionals.[5] DevOps focuses on continuous delivery by leveraging on-demand IT resources and by automating test and deployment of analytics. This merging of software development and IT operations has improved velocity, quality, predictability and scale of software engineering and deployment. Borrowing methods from DevOps, DataOps seeks to bring these same improvements to data analytics.[3] Like lean manufacturing, DataOps utilizes statistical process control (SPC) to monitor and control the data analytics pipeline. With SPC in place, the data flowing through an operational system is constantly monitored and verified to be working. If an anomaly occurs, the data analytics team can be notified through an automated alert.[6] DataOps seeks to provide the tools, processes, and organizational structures to cope with this significant increase in data.[7] Automation streamlines the daily demands of managing large integrated databases, freeing the data team to develop new analytics in a more efficient and effective way.[9] DataOps embraces the need to manage many sources of data, numerous data pipelines and a wide variety of transformations.[3] DataOps seeks to increase velocity, reliability, and quality of data analytics.[10] It emphasizes communication, collaboration, integration, automation, measurement and cooperation between data scientists, analysts, data/ETL(extract, transform, load) engineers, information technology (IT), and quality assurance/governance.[11] It aims to help organizations rapidly produce insight, turn that insight into operational tools, and continuously improve analytic operations and performance.[11]
- #56 This is sometimes really hard for Data Scientists who experiment with data on laptops to accept.
- #58 Add Data and Logic Tests
- #59 Version control is the foundation upon which a lot of delivery is built. At a minimum, reviewers of a publication and future researchers should be able to: 1) Download all data and software used to generate the results. 2) Run tests and review source code to verify correctness. 3) Run a build process to execute the computation. Version control makes it possible to maintain an archived version of the code used to produce a particular result. Examples include Git and Subversion. 3) Automated build systems document the high-level structure of a computation: which programs process which data, what outputs they produce, etc. Examples include Make and Ant.
- #60 Configuration management tools document the details of the computational environment where the result was produced, including the programming languages, libraries, and system-level software the results depend on. Examples include package managers like Conda that document a set of packages, containers like Docker that also document system software, and virtual machines that actually contain the entire environment needed to run a computation.
- #61 In an enterprise setting where multiple data scientists could be working on a single project, the first step to doing data science work that scales is implementing version control, whether that’s GitHub, GitLab, Bitbucket, or another solution. Once your team has the ability to track code changes, the next step is to create a process in which they regularly commit their code to the master branch of your repository.
- #62 2) During development, automated tests make programs more likely to be correct; they also tend to improve code quality. During review, they provide evidence of correctness, and for future researchers they provide what is often the most useful form of documentation. Examples include unittest and nose for Python and JUnit for Java.
- #63 You can move beyond Continuous Integration to make deployment even faster. Traditionally, data science deployment has been a multi-step process that puts the onus on engineering: Engineers would refactor, test, and automate or schedule a data scientist’s model before slowly rolling it out, sometimes months after it was originally built. Developers that embrace continuous delivery are pushing new application features or changes into production quickly, sometimes with the click of a button. Increasingly, cloud and data science platforms are filling this void with features such as the ability to deploy models as APIs or schedule code runs which means that as soon as new development passes your tests it can be deployed into production with no dependencies on other teams.
- #64 Which brings me on to tools Just as chemistry is not about the tubes but the process of experimentation. DataOps is not tied to a particular technology, architecture, tool, language or framework. However, some tools are better at supporting DataOps collaboration, orchestration, agility, quality, security, access and ease of use.
- #65 Whenever, choosing tools is best to never start with the tools themselves. I like to use the spine model by Tre the wey, Roux and Perold. So to decide on the tool you need to understand the practices you employ, in order to understand what practices to employ you need to define your principles, to define your principles you need to know your values, and to know your values you need start with the needs you’re trying to fulfil. We have a set of clear DataOps Practices we want to employ so we have a clear idea of what tools will be fit for purpose. http://spinemodel.info/explanation/introduction
- #66 But first a bit of advice. You should avoid building your own anything or being on the bleeding edge. Any technology or tool that is really useful will end up being refined or commoditised and turned into a service. Let someone else find the bugs, be the beta tester or end up in cul-de-sac. The other factor to take into account is Cost of Delay. It’s nearly always ignored in business cases. On paper be cheaper to build your own solution. However, the months, or years, you’re taking to do that is months, or years, you’re not benefiting from the solution and handing to your competitors. And it always takes twice as long to build your own solutions, even after you’ve factored in it’s going to take you twice as long as you think.
- #67 Because one of our principles is that we want to make data cycles shorter and shorter it’s important Data Scientists can self-serve not just the data but also the infrastructure, tools and packages
- #68 Modern Cloud architecture makes it very easy to create data products rapidly. Specifically, the move from Infrastructure and Platform as a Service to Software as a Service and Serverless architecture. That means you having not hardware or software to configure, you just bring your data and code and all the scaling and optimisation is done for you. The other advantage is you can use the same tools for dev and production, You can also the same data in dev and production as in the SAAS or Serverless word there’s no need for separation of environments. We’re so convinced of the benefits we’re actually moving our Data Analytics stack out of AWS onto Google Cloud Platform. Here’s an example of GCP reference architecture for Big data which isn’t a million miles from our architecture. There’s absolutely no infrastrucutre to manage within the environment. The other thing you can do is use the cloud as a centralised platform helping to break down organisational barriers and makes it easier to enforce governance rules.
- #69 Modern cloud takes care of the underlying tools but you can add further levels of abstraction and self-service to the compute infrastructure and data pipeline. Data Science Platforms provide tools that enable teams to work faster and deploy DataOps methodology very easily from choosing the computer infrastructure and environments to run their code on, to automated version control, collaboration tools and one-click deployment to APIs and Dashboards.
- #70 The requirements for this type of platform haven’t gone unnoticed, these are just some of the vendors we looked at before settling on Domino Data Labs. Each has their strengths and weaknesses, so which one is best depends on your use-cases.
- #71 There’s another positive side-effect of going down the DataOps route. You require fewer roles due to self-service. There no need for Specialist Dev Ops, Infrastructure Engineers, Sys Admins or DBAs. This reduces friction, hand-offs, bottlenecks You’re left with just four key roles, Data Scientists, Data Engineers, Data Analysts (who are a much under-invested in group as everyone wants to be a Data Scientist) and the Line of Business (these are the stakeholders and also those who will help integrate your Data Product into other applications.
- #72 Worrying About Artificial Intelligence when you can’t even produce a Sales report is not going to get you very far. You need to worry about being able to action data instead in alignment with the organisation’s strategy and goals.
- #75 80% of the battle is knowing what not to work on. You should not work on projects but products. Products are in constant use by consumers and have direct customer and business benefit. The benefit scales according to the number of customers who use them. A data product may be a machine learning model, a segmentation, a recommendation engine, a dashboard. They may be integrated into other products. They have an owner, you get feedback that helps you improve them through iteration. They are not one-off adhoc pieces of insight that get filed away.
- #82 Velocity is th
- #83 We need to solve all the problems with Data Science today: Hamster Wheel Analytics – Doing busywork for the organisation that makes us feel good because we’re putting in a lot of effort and clients appreciate but is never going to move the needle. The work we do that’s not repeatable because it was never documented The aimless crash and burn – Where we explore data to find the magical insights without a clear objective or worse the rest of the business has no interest in The Roadblock – Work we do that has no route to the customers because it is blocked by corporate silos, IT, Security, lack of infrastructure, tools or willingness to integrate into an end product and remains on a laptop. Work we do that does make a customer impact which we can’t measure because the feedback loop was never closed.
- #84 Instead we can move to the DataOps World – What I like to call the Data Science Factory. It starts with alignment with the rest of the business’ strategy to create options for Agile Delivery and collaboration to deliver them. Rapid delivery of Data Products because there is the governance, trust in data, self-service and automation A path to the end-consumer and feedback to measure value for the next iteration.