Apache Flink Hands-On
Download as PPTX, PDF•1 like•694 views
Apache Flink is an open source stream processing framework that originated from a research project at TU Berlin in 2009. It is undergoing incubation at the Apache Software Foundation and allows for fast and reliable processing of big data. Flink provides high-level APIs for the Java and Scala programming languages to define and execute dataflow programs on top of distributed datasets.
Apache Flink Hands-On
- 1. Apache Flink Fast and reliable big data processing Aljoscha Krettek [email protected]
- 2. What is Apache Flink? • Project undergoing incubation in the Apache Software Foundation • Originating from the Stratosphere research project started at TU Berlin in 2009 • http://flink.incubator.apache.org • 59 contributors (doubled in ~4 months) • Has awesome squirrel logo
- 3. What is Apache Flink? Flink Client
- 4. Apache Flink ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); flink.incubator.apache.org … env.execute();
- 5. Apache Flink DataSet<String> input = env.readTextFile(“/hello/there”); flink.incubator.apache.org
- 6. Apache Flink DataSet<String> input = env.readTextFile(“hdfs:///hello/there”); flink.incubator.apache.org
- 7. Apache Flink DataSet<Tuple2<String,Integer> input = env.readCsvFile(“/hello/there”) flink.incubator.apache.org .fieldDelimiter(‘|’) .lineDelimiter(“n") .ignoreFirstLine() .types(String.class, Integer.class);
- 8. Apache Flink DataSet<Beer> beers = input.map( new MapFunction<?, Beer>() { flink.incubator.apache.org public Beer map(? in) { return new Beer(in.f0, in.f1); } });
- 9. Apache Flink DataSet<Beer> beers = input.map( in -> new Beer(in.f0, in.f1) ); flink.incubator.apache.org
- 10. Apache Flink val beers = input.map( in => new Beer(in._1, in._2) ) flink.incubator.apache.org
- 11. Apache Flink DataSet<Beer> filtered = beers.filter( new FilterFunction<Beer>() { flink.incubator.apache.org public boolean filter(Beer in) { return beer.name.contains(“augustiner”); } });
- 12. Apache Flink flink.incubator.apache.org DataSet<Beer> grouped = beers .groupBy(“name”) .sortGroup(“rating”, Order.DESCENDING) .reduceGroup( new GroupReduceFunction<Beer, Beer>() { public Beer reduceGroup(Iterable<Beer> in, Collector<Beer> out) { out.collect(in.iterator().next()); } });
- 13. Apache Flink DataSet<Tuple2<String, Integer>> aggregated = input.groupBy(0).sum(1); flink.incubator.apache.org
- 14. Apache Flink flink.incubator.apache.org result.print();
- 15. Apache Flink result.writeAsText(“/ciao/for/now”); flink.incubator.apache.org
- 16. github.com/aljoscha/beer-analysis www.filedropper.com/beerdatacsv www.filedropper.com/beerdata (large) flink.incubator.apache.org github.com/apache/incubator-flink meetup.com/Apache-Flink-Meetup
Editor's Notes
- #4: Data processing engine which let you write programs in a functional style and executes them automatically in parallel