Uploaded byCraig Warman
Spark SQL & Machine Learning - A Practical Demonstration
The document provides an overview of Apache Spark, detailing its architecture, core components, and functionalities like Spark SQL and MLlib for machine learning, alongside practical demonstrations. It emphasizes Spark's speed advantage over Hadoop's MapReduce for in-memory data processing, and illustrates its use in collaborative filtering for recommender systems. Furthermore, it touches on Spark's integration with various programming languages and tools, highlighting important concepts such as resilient distributed datasets (RDDs) and dataframes.
More Related Content
Similar to Spark SQL & Machine Learning - A Practical Demonstration
Spark SQL & Machine Learning - A Practical Demonstration
- 1.
- 2. ® • Spark Background/Overview Brief Spark background Tthe Spark+Hadoopteam Spark's five main components • Spark SQL Architecture Features, Languages, How DataFrames work, The SQLContext, Data sources • Loading And Querying a Dataset with Spark SQL Live demonstration of setting up a SQLContext Loading it with data Running queries against it • Machine Learning with Spark MLlib Collaborative filtering basics Alternating Least Squares (ALS) algorithm Live demo of a simple recommender model and training- test loop iterations 2
- 3.
- 4. ® Spark began life in 2009 as a project within the AMPLab at the University of California, Berkeley. Spark became an incubated project of the Apache Software Foundation in 2013 Was promoted to Foundation top-level project in early 2014 • Is currently one of the most active projects managed by the Foundation Key drivers (Reasons why you might select Spark for a data project): •Rich and well-documented API designed specifically for interacting with data at scale. • Supports Java, Scala, Python, R, and SQL • Spark was optimized to run in memory from the beginning • Helps it process data far more quickly than 3
- 5. ® alternative approaches like Hadoop's MapReduce, which tends to write data to and from computer hard drives between each stage of processing. Spark is not, despite the hype, a replacement for Hadoop. Nor is MapReduce dead. • Hadoop is a platform that encompasses a wide variety of technologies •Spark is faster than MapReduce for iterative algorithms that fit data in memory. • So, even though you can run Spark in stand-alone mode, doing so means you completely miss out on Hadoop's ability to run multiple types of workloads (incl. advanced analytics with Spark) on the same data at the same time. • In other words, Spark without Hadoop is just another silo. 3
- 6. ® Important to remember: Hadoop is more than just MapReduce Spark and MapReduce are: • Scalable frameworks for executing custom code on a cluster •Nodes in the cluster work independently to process fragments of data and also combine those fragments together when appropriate to yield a final result • Can tolerate loss of a node during a computation • Require a distributed storage layer for common data view Spark is often deployed in conjunction with a Hadoop cluster • On its own, Spark isn’t well-suited to production workloads • It needs a resource manager • YARN, for instance, which takes responsibility for scheduling tasks across cluster nodes • A distributed filesystem 4
- 7.
- 8. ® Apache Spark consists of Spark Core and a set of libraries. Spark Core • This is the distributed execution engine • Implements a programming abstraction known as Resilient Distributed Datasets RDDs •Provides APIs for S=Java, Scala, and Python • Some documentation shows R as part of the core, others show it as a separate module. The R module was added to the 1.4 release. Spark SQL • This is the module for working with structured data using SQL • Supports the open source Hive project, along with ODBC and JDBC Spark Streaming 5
- 9. ® • Enables scalable and fault-tolerant processing of data streams • Supports sources like Flume (for data logs) and Kafka (for messaging) MLlib •Scalable machine learning library • Implements commonly-used machine learning and statistical algorithms • These algorithms are often iterative • Spark’s ability to cache the dataset in memory greatly speeds up such iterative data processing • So it’s an ideal processing engine for implementing such algorithms GraphX • Supports analysis of and computation over graphs of data • Includes a number of widely understood graph algorithms, including PageRank. • Graph databases are well-suited for analyzing interconnections • They’re useful for working with data that involve complex relationships (such as social media) This presentation will focus primarily on the Spark Core API using Scala, along with SQL and the machine learning library 5
- 10.
- 11.
- 12. ® There are two basic types of operations Transformations essentially change the data, so a new RDD is created because the original cannot be changed They’re lazily evaluated –meaning that they aren’t executed until a subsequent action has a need for a result This improves performance because it can avoid unnecessary processing Actions measure – but don’t change – the original data. They essentially force processing to take place 7
- 13.
- 14.
- 15.
- 16.
- 17.
- 18. ® These are Scala caseclasses that define schemas corresponding to their respective files 13
- 19.
- 20.
- 21.
- 22.
- 23.
- 24.
- 25.
- 26. ® Collaborative filtering (CF) is a technique used by some recommender systems. Recommender systems essentially seek to predict the 'rating' or 'preference' that a user would give to an item. So,Collaborative Filtering tries to make these predictions (we call this “filtering”) about the interests of a user by collecting information from many users (hence “collaborating”) 21
- 27.
- 28.
- 29.
- 30.
- 31. ® Let’s break off the users – We know that there are m number of them, that’s a given But there’s something we don’t know about these users There are certain influences that might cause these users to rate movies higher or lower It could be demographics -age-related, gender, profession, whatever. The point is, we don’t know what they are, we just know they’re present in the background somewhere. I used this reference regarding latent variables: http://stats.stackexchange.com/questions/162585/predicting- score-in-the-presence-of-latent-variables I used an explanation from Cambridge Coding Academy’s 26
- 32.
- 33.
- 34.
- 35. ® Likewise, let’s break off the movies – We know that there are n number of them, that’s a given Like users, there’s something we don’t know about these movies It’s those same featuresthat we just talked about… I used an explanation from Cambridge Coding Academy’s “Predicting User Preferences in Python using Alternating Least Squares” tutorial located here: http://online.cambridgecoding.com/notebooks/mhaller/predicti ng-user-preferences-in-python-using-alternating-least-squares 28
- 36.
- 37.
- 38.
- 39.
- 40.
- 41. ® Problem: We don’t know what the features are, or how many we have For that, we’lluse the trick of alternating least squares. The concept of dot product is explained here: https://www.mathsisfun.com/algebra/matrix-multiplying.html I used an explanation from Cambridge Coding Academy’s “Predicting User Preferences in Python using Alternating Least Squares” tutorial located here: http://online.cambridgecoding.com/notebooks/mhaller/predicti ng-user-preferences-in-python-using-alternating-least-squares 32
- 42.
- 43.
- 44.
- 45.
- 46.
- 47.
- 48.
- 49. ® We’ll then estimate the Movie Features matrix for every movie, column by column, using the non-empty cells from the Ratings Matrix and the values from the User Featuresmatrix. I used an explanation from Cambridge Coding Academy’s “Predicting User Preferences in Python using Alternating Least Squares” tutorial located here: http://online.cambridgecoding.com/notebooks/mhaller/predicti ng-user-preferences-in-python-using-alternating-least-squares 40
- 50.
- 51.
- 52.
- 53.
- 54.
- 55. ® So we have things that will influence how good our model is Regularization is a technique used in an attempt to solve the overfitting problem in statistical models. Here’s how it works, in a nutshell: Let’s assume we only have two features that influence movie ratings –age and gender. That’s it. Well probably this model will fail because it’s too simple. So let’s throw in more demographics such as profession, what part of the country they’re from, education level, etc. This makes the model more interesting and complex. Our model will do better in some ways, but worse in others due to this concept of overfitting This is because it’s sticking too much to the data – in other words, it can’t generalize 46
- 56. ® Which means we’re being influenced by “background noise” So overfitting happens when the model works well on training data using known noise but doesn't perform well on the actual testing data. This isn't acceptable. So we apply a regularization factor to controlthe model’s complexity, and thus help prevent overfitting The higher the regularization factor, the lower the overfitting, but the greater the bias. If you’re a visual person, imagine smoothing a regression line so that it’s less influenced by noisy data points. Incidentally, three of these are actually parameters into the algorithm Rank - Starting at 5 and increasing by 5 until the recommendation improvement rate slows down, memory and CPU permitting, is a good approach Iterations – Likewise, 5, 10, and 20 iterations for various combinations of rank and lambda are good starting points Lambda - Values of 0.01, 1 and 10 are good values to test. I used an explanation from Quora here: https://www.quora.com/What-is-regularization-in-machine- learning And here: https://cloud.google.com/solutions/recommendations-using- machine-learning-on-compute-engine 46
- 57. ® Split your dataset into three pieces: • One for training •One for validation • One for testing 47
- 58.
- 59.
- 60.
- 61.
- 62.
- 63.
- 64. ® In mathematical terms, it represents the sample standard deviation of the differences between predictedand observed values. I used an explanation from Wikipedia here: https://en.wikipedia.org/wiki/Root-mean-square_deviation 54
- 65.
- 66.
- 67.
- 68. ® Note the use of the cache method here –I want to keep this in memory 58
- 69.
- 70.
- 71. ® I’ve got three parameters, with a variety of values for each, for a total of 12 runs The model will train At the end of each runit’ll tell us the RMSE The lowest RMSE will be our final test run parameters. 61
- 72.
- 73. ® or more, the jury would issue a last call, giving all teams 30 days to send their submissions. Only then, the team with best submission was asked for the algorithm description, source code, and non-exclusive license, and, after successful verification; declared a grand prize winner. The contest would last until the grand prize winner was declared. Had no one received the grand prize, it would have lasted for at least five years (until October 2, 2011). After that date, the contest could have been terminated at any time at Netflix's sole discretion. Two progress prizes were awarded in 2007 and 2008 Finally, on September 18, 2009, Netflix announced team "BellKor'sPragmatic Chaos" as the prize winner References: https://en.wikipedia.org/wiki/Netflix_Prize https://www.techdirt.com/blog/innovation/articles/20120409/ 03412518422/why-netflix-never-implemented-algorithm-that- won-netflix-1-million-challenge.shtml 62
- 74.
- 75.
- 76.
- 77.
- 78.
- 79.
- 80. ® First, we’ll assume your data is present within the context of a Data Frame Remember this is structured data, just like a table in a relational database We can save Data Frame as simple JSON document files in the filesystem. This makes them reasonably portable. A better option is to materialize dataframes as Parquet files Parquet is a columnar format that is supported by many other data processing systems. The nice thing about a columnar-formatted datastoreis that they are well-suited for OLAP-type workloads against very large datasets (e.g., data warehouses) This is because the data is stored as columns rather than rows 68
- 81. ® Another side benefit is that they tend to compress very nicely as well You can still run SQL against these files Still, a third option is to save them to a Hive metastore using the saveAsTablecommand An existing Hive deployment is not necessary to use this feature. Spark will create a default local Hive metastore using Apache Derby for you. The saveAsTable command materializes the contents of a Data Frame and creates a pointer to the data in the Hive metastore If you liked the sound of the columnar Parquet file format then you’ll be happy to know that saveAsTable writes using Parquet as well I’ll demonstrate this third option with Tableau as my BI tool I used this image from this video: https://www.youtube.com/watch?v=BN4-GrZ1mGU I referred to this Wikipedia entry: https://en.wikipedia.org/wiki/Column-oriented_DBMS and this Spark documentation: https://spark.apache.org/docs/latest/sql- programming-guide.html 68
- 82.
- 83.
- 84.
- 85.
- 86.
- 87.
- 88.
- 89.
- 90.
- 91. ® • Spark Background/Overview Brief Spark background Tthe Spark+Hadoopteam Spark's five main components • Spark SQL Architecture Features, Languages, How DataFrames work, The SQLContext, Data sources • Loading And Querying a Dataset with Spark SQL Live demonstration of setting up a SQLContext Loading it with data Running queries against it • Machine Learning with Spark MLlib Collaborative filtering basics Alternating Least Squares (ALS) algorithm Live demo of a simple recommender model and training- test loop iterations 77
- 92.
- 93.