![Who am I?
• Wisely Chen ( thegiive@gmail.com )
• Sr. Engineer inYahoo![Taiwan] data team
• Loves to promote open source tech
• Hadoop Summit 2013 San Jose
• Jenkins Conf 2013 Palo Alto
• Spark Summit 2014 San Francisco
• Coscup 2006, 2012, 2013 , OSDC 2007,Webconf 2013,
Coscup 2012, PHPConf 2012 , RubyConf 2012](https://image.slidesharecdn.com/nckucsietalkslides-141009214416-conversion-gate01/85/Ncku-csie-talk-about-Spark-3-320.jpg)
![MLlib
• Data
• data: [(36, 2802, 4.0), (36, 256, 4.0), …]
• rank, numIter, lambda are int
• candidates : [(0, 2),(0, 3),(0, 4)…]
• model = ALS.train(data, rank, numIter, lambda)
• model.predictAll(candidates)](https://image.slidesharecdn.com/nckucsietalkslides-141009214416-conversion-gate01/85/Ncku-csie-talk-about-Spark-95-320.jpg)
Ncku csie talk about Spark
Download as PDF, PPTX3 likes1,095 views
This document provides an introduction and agenda for a presentation on Spark. It discusses how Spark is a fast engine for large-scale data processing and how it improves on MapReduce. Spark stores data in memory across clusters to allow for faster iterative computations versus writing to disk with MapReduce. The presentation will demonstrate Spark concepts through word count and log analysis examples and provide an overview of Spark's Resilient Distributed Datasets (RDDs) and directed acyclic graph (DAG) execution model.
More Related Content
Similar to Ncku csie talk about Spark(20)
Ncku csie talk about Spark
- 1. Introduction to Spark WiselyChen ([email protected]) Sr. Engineer at Yahoo
- 2. Agenda • Big datawill change the world? • What is Spark? • Demo (Start a spark cluster / Word Count) • Break : 10min • Spark Concept • Demo (ETL / MLib) • Q&A
- 3. Who am I? •Wisely Chen ( [email protected] ) • Sr. Engineer inYahoo![Taiwan] data team • Loves to promote open source tech • Hadoop Summit 2013 San Jose • Jenkins Conf 2013 Palo Alto • Spark Summit 2014 San Francisco • Coscup 2006, 2012, 2013 , OSDC 2007,Webconf 2013, Coscup 2012, PHPConf 2012 , RubyConf 2012
- 4. Taiwan Data Team Data! Highway BI! Report Serving! API Data! Mart ETL/ Forecast Machine! Learning
- 5. Big data willchange the world?
- 6.
- 7.
- 8.
- 9.
- 10. More Sensor • 2000 •Browser • Digital Camera(Photo) • 2000~2014 • Browser • Mobile (GPS,More Photo,Video) • Wearable Device(Pulse) • Google Glass(More Video) • Internet of thing(………)
- 11.
- 12. More Sensor • 2000 •Browser • Digital Camera(Photo) • 2000~2014 • Browser • Mobile (GPS,More Photo,Video) • Wearable Device(Pulse) • Google Glass(More Video) • Internet of thing(………)
- 13. Technology Improve • SloanDigital Sky Survey(SDSS) collected more data in its first few weeks than had been amassed in the entire history of astronomy. • The Large Synoptic Survey Telescope in Chile, due to come on stream in 2016, will acquire that quantity of data every five days.
- 14. New Area 30min Zebrafish experiment = 1TB http://research.janelia.org/zebrafish/
- 15. Hadoop handle bigdata well • 18M of hadoop related jobs on Yahoo Grid • Yahoo handle over 440 PB data daily • Most of job are ETL/SQL/BI
- 16. EBay’s data volume 2015: 130EB 2020 : 4000ZB Vadim Kutsyy “Data Science Empowering Personalization” in Big data innovation Summit 2014 Boston
- 17. Data is notonly bigger • We have more area of data • More Sensor • Sensor technology improve • New area
- 18.
- 19. Word Grammar Check •MS researcher Michele and Eric try to improve grammar check algorithm • They took four algorithm and feed in 10M, 100M and 1B words • In 10M words, sophisticated algorithm(86%) works bester than simpler algorithm(75%) • In 1B words, simpler algorithm(95%+) improved a lot, even better than sophisticated algorithm(94%)
- 20. –Google AI guruPeter Norvig, "The Unreasonable Effectiveness of Data” “Simple models and a lot of data trump more elaborate models based on less data” In translate area
- 21. Different type ofdata • In Harvard Data Mining Class, two team do Netflix recommendation challenge • Team A came up with a very sophisticated algorithm using the Netflix data. • Team B used a very simple algorithm, but they added in additional data beyond the Netflix set • Team B got much better results, close to the best results on the Netflix leaderboard
- 22. Taiwan Shopping UserAnalysis Man tend to view underwear. But they don’t buy it Male Users’ Top5 View Categories 1. Computer 2. Camera 3. ………. 4. ………. 5. Woman Underwear
- 23. 2 types ofdata Traffic Data Transaction Data User’s views / clicks “Weak intention” Large amount User’s checkout “Strong Intention” Small amour
- 24. Small Data Sophisticated Algo OK Result Big Data Simple Algo Better Result Data Set 1 SmartModel which leverage more area of data Data Set 2 Best Result
- 25.
- 26. Robot • FoxCon’s robotcan replace 70% worker • Google driverless car/big dog • Amazon warehouse robot
- 27.
- 28. It is notmovie it is happening
- 29. Sensor Data Machine Learning Robot User Behavior Recommendation Algorithm Recommendation to user (1/3 salesare from recommendation module) Amazon layoff the editor team and replaced by recommendation algorithm
- 30.
- 31. Sensor Data Machine Learning Robot DNNresearch, Behavio, Wavii,Flutter, autofuss, DeepMind, spider.io, Adometry, QQuest Visual, Jetpac Talaria, Stackdriver SCHAFT, Industrial Perception, Redwood Robotics, Meka Robotics, Holomni, Bot & Dolly, Boston Dynamics, Titan Aerospace, Nest Lab, MyEnergy, Skybox Imaging, Dropcam, Google buy 47 company at 13,14 IOT Google is top 1 leader in big data 24 company on the ring
- 32.
- 33.
- 34. The ring willchange the world and big data is the core of ring
- 35.
- 36.
- 37. Hadoop is notgood in machine learning • Efficiency • Difficult • Data Engineer • Data Scientist • Data Analyst • Algorithm: it is not so easy to parallelize your algorithm
- 38.
- 39. Hadoop is notgood in machine learning • Efficiency • Difficult : Data scientist don’t know how to do • Algorithm: it is not so easy to parallelize your algorithm
- 40. 3X~25X than MapReduceframework ! From Matei’s paper: http://0rz.tw/VVqgP Logistic regression RunningTime(S) 0 20 40 60 80 MR Spark 3 76 KMeans 0 27.5 55 82.5 110 MR Spark 33 106 PageRank 0 45 90 135 180 MR Spark 23 171
- 41. Hadoop is notgood in machine learning • Efficiency • Difficult • Algorithm: it is not so easy to parallelize your algorithm
- 42.
- 43. Language Support • Python: Data Scientist , Data Engineer • Java : Data Engineer • Scala : Data Engineer • SQL : Data Scientist, Data Analyst , Data Engineer • R : Data Scientist, Data Analyst • (will be official support in 1.2)
- 44. Python Word Count •file = spark.textFile("hdfs://...") • counts = file.flatMap(lambda line: line.split(" ")) • .map(lambda word: (word, 1)) • .reduceByKey(lambda a, b: a + b) • counts.saveAsTextFile("hdfs://...") Access data via Spark API Process via Python
- 45. Scala Word Count •val file = spark.textFile("hdfs://...") • val counts = file.flatMap(line => line.split(" ")) • .map(word => (word, 1)) • .reduceByKey(_ + _) • counts.saveAsTextFile("hdfs://...")
- 46. Java Wordcount • JavaRDD<String>file = spark.textFile("hdfs://..."); • JavaRDD<String> words = file.flatMap(new FlatMapFunction<String, String>() • public Iterable<String> call(String s) { return Arrays.asList(s.split(" ")); } • }); • JavaPairRDD<String, Integer> pairs = words.map(new PairFunction<String, String, Integer>() • public Tuple2<String, Integer> call(String s) { return new Tuple2<String, Integer>(s, 1); } • }); • JavaPairRDD<String, Integer> counts = pairs.reduceByKey(new Function2<Integer, Integer>() • public Integer call(Integer a, Integer b) { return a + b; } • }); • counts.saveAsTextFile("hdfs://...");
- 47. Highly Recommend • Scala: Latest API feature, Stable • Python • very familiar language • Native Lib: NumPy, SciPy
- 48. What is Spark •From UC Berkeley AMP Lab • Apache Spark™ is a very fast and general engine for large-scale data processing • Most activity Big data open source project since Hadoop
- 49.
- 50.
- 51.
- 52.
- 53. HDFS YARN MapReduce Hadoop vs Spark Spark HiveShark/SparkSQL
- 54. More than MapReduce HDFS SparkCore : MapReduce SparkSQL: Hive GraphX: Pregel MLib: Mahout Streaming: Storm Resource Management System(Yarn, Mesos)
- 55. How to useit? • 1. go to https://spark.apache.org/ • 2. Download and unzip it • 3. ./sbin/start-all.sh or ./bin/spark-shell
- 56. EC2 • ./ec2/spark-ec2 -kxxx -i xxx -s 3 launch CLUSTERNAME ! • http://spark.apache.org/docs/latest/ec2- scripts.html
- 57.
- 58. Python Word Count •file = spark.textFile("hdfs://...") • counts = file.flatMap(lambda line: line.split(" ")) • .map(lambda word: (word, 1)) • .reduceByKey(lambda a, b: a + b) • counts.saveAsTextFile("hdfs://...")
- 59.
- 60.
- 61. Why is Sparkso fast?
- 62. Most machine learning algorithmsneed iterative computing
- 63. a1.0 1.0 1.0 1.0 PageRank 1st Iter 2ndIter 3rd Iter b d c Rank Tmp Result Rank Tmp Result a1.85 1.0 0.58 b d c 0.58 a1.31 1.72 0.39 b d c 0.58
- 64. HDFS is 100xslower than memory Input (HDFS) Iter 1 Tmp (HDFS) Iter 2 Tmp (HDFS) Iter N Input (HDFS) Iter 1 Tmp (Mem) Iter 2 Tmp (Mem) Iter N MapReduce Spark
- 65. First iteration(HDFS)! take 200sec 3rd iteration(mem)! take 7.7 sec Page Rank algorithm in 1 billion record url 2nd iteration(mem)! take 7.4 sec
- 66. Memory Size Problem Cachestorage in local disk(2sec) Cache storage in memory(2sec) Network transfer(30 sec)
- 67. Just memory? • FromMatei’s paper: http://0rz.tw/VVqgP • HBM: stores data in an in-memory HDFS instance. • SP : Spark • HBM’1, SP’1 : first run • Storage: HDFS with 256 MB blocks • Node information • m1.xlarge EC2 nodes • 4 cores • 15 GB of RAM
- 68. 100GB data on100 node cluster Logistic regression RunningTime(S) 0 35 70 105 140 HBM'1 HBM SP'1 SP 3 46 62 139 KMeans RunningTime(S) 0 50 100 150 200 HBM'1 HBM SP'1 SP 33 8287 182
- 69.
- 70.
- 71. map,filter groupBy on ! non-partitioneddata union join with input! co-partitioned join with inputs not! co-partitioned Map(Narrow) Reduce(Wide)
- 72.
- 73.
- 74.
- 75.
- 76.
- 77. RDD • Resilient DistributedDataset • Interface of data, stored in RAM or on Disk • Built through parallel transformations
- 78. RDD RDD a RDDb val a =sc.textFile(“hdfs://....”) val b = a.filer( line=>line.contain(“Spark”) ) Value c val c = b.count() Transformation Action
- 79. Log mining a =sc.textfile(“hdfs://aaa.com/a.txt”)! err = a.filter(lambda t=> “ERROR” in t )! .filter(lambda t=> “2014” in t)! ! err.cache()! err.count()! ! m = err.filter(lambda t=>“MYSQL” in t)! ! ! .count()! a = err.filter(lambda t=> “APACHE” in t )! ! ! .count() Driver Worker! ! ! ! Worker! ! ! ! Worker! ! ! !Task TaskTask
- 80. Log mining a =sc.textfile(“hdfs://aaa.com/a.txt”)! err = a.filter(lambda t=> “ERROR” in t )! .filter(lambda t=> “2014” in t)! ! err.cache()! err.count()! ! m = err.filter(lambda t=>“MYSQL” in t)! ! ! .count()! a = err.filter(lambda t=> “APACHE” in t )! ! ! .count() Driver Worker! ! ! ! !Block1 RDD a Worker! ! ! ! !Block2 RDD a Worker! ! ! ! !Block3 RDD a
- 81. Log mining a =sc.textfile(“hdfs://aaa.com/a.txt”)! err = a.filter(lambda t=> “ERROR” in t )! .filter(lambda t=> “2014” in t)! ! err.cache()! err.count()! ! m = err.filter(lambda t=>“MYSQL” in t)! ! ! .count()! a = err.filter(lambda t=> “APACHE” in t )! ! ! .count() Driver Worker! ! ! ! ! RDD err Worker! ! ! ! ! RDD err Worker! ! ! ! ! RDD err Block1 Block2 Block3
- 82. Log mining a =sc.textfile(“hdfs://aaa.com/a.txt”)! err = a.filter(lambda t=> “ERROR” in t )! .filter(lambda t=> “2014” in t)! ! err.cache()! err.count()! ! m = err.filter(lambda t=>“MYSQL” in t)! ! ! .count()! a = err.filter(lambda t=> “APACHE” in t )! ! ! .count() Driver Worker! ! ! ! ! RDD err Worker! ! ! ! ! RDD err Worker! ! ! ! ! RDD err Block1 Block2 Block3
- 83. Log mining a =sc.textfile(“hdfs://aaa.com/a.txt”)! err = a.filter(lambda t=> “ERROR” in t )! .filter(lambda t=> “2014” in t)! ! err.cache()! err.count()! ! m = err.filter(lambda t=>“MYSQL” in t)! ! ! .count()! a = err.filter(lambda t=> “APACHE” in t )! ! ! .count() Driver Worker! ! ! ! ! RDD err Worker! ! ! ! ! RDD err Worker! ! ! ! ! RDD err Cache1 Cache2 Cache3
- 84. Log mining a =sc.textfile(“hdfs://aaa.com/a.txt”)! err = a.filter(lambda t=> “ERROR” in t )! .filter(lambda t=> “2014” in t)! ! err.cache()! err.count()! ! m = err.filter(lambda t=>“MYSQL” in t)! ! ! .count()! a = err.filter(lambda t=> “APACHE” in t )! ! ! .count() Driver Worker! ! ! ! ! RDD m Worker! ! ! ! ! RDD m Worker! ! ! ! ! RDD m Cache1 Cache2 Cache3
- 85. Log mining a =sc.textfile(“hdfs://aaa.com/a.txt”)! err = a.filter(lambda t=> “ERROR” in t )! .filter(lambda t=> “2014” in t)! ! err.cache()! err.count()! ! m = err.filter(lambda t=>“MYSQL” in t)! ! ! .count()! a = err.filter(lambda t=> “APACHE” in t )! ! ! .count() Driver Worker! ! ! ! ! RDD a Worker! ! ! ! ! RDD a Worker! ! ! ! ! RDD a Cache1 Cache2 Cache3
- 86. 1st iteration(no cache)! take sametime with cache! take 7 sec RDD Cache
- 87. RDD Cache • Datalocality • Cache A big shuffle! take 20min After cache, take only 265ms self join 5 billion record data
- 88.
- 89.
- 90.
- 91. a1.0 1.0 1.0 1.0 PageRank 1st Iter 2ndIter 3rd Iter b d c Rank Tmp Result Rank Tmp Result a1.85 1.0 0.58 b d c 0.58 a1.31 1.72 0.39 b d c 0.58
- 92.
- 93.
- 95. MLlib • Data • data:[(36, 2802, 4.0), (36, 256, 4.0), …] • rank, numIter, lambda are int • candidates : [(0, 2),(0, 3),(0, 4)…] • model = ALS.train(data, rank, numIter, lambda) • model.predictAll(candidates)
- 96. Homework • 1. InstallSpark and run word count (50%) • Data : http://www.gutenberg.org/ebooks/5000 • Output: total word number • 2. Write Movie Recommendation (50%) • Trainning Data : http://arbor.ee.ntu.edu.tw/~wisely/data/lesson.tgz • Input: 10 rating(1-5) on the 10 movie • Example: movie 123 rating is 3 , movie 45 is 5 • Output: Top 10 recommendation movie • Any algorithm is ok
- 98.
- 99. Background Knowledge • Tweetreal time data store into SQL database • Spark MLLib use Wikipedia data to train a TF- IDF model • SparkSQL select tweet and filter by TF-IDF model • Generate live BI report
- 100. Code • val wiki= sql(“select text from wiki”) • val model = new TFIDF() • model.train(wiki) • registerFunction(“similarity” , model.similarity _ ) • select tweet from tweet where similarity(tweet, “$search” > 0.01 )
- 101.
- 102.