Download as PDF, PPTX
More Related Content
![[오픈소스컨설팅] EFK Stack 소개와 설치 방법](https://cdn.slidesharecdn.com/ss_thumbnails/elasticstack-210712042246-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Apache NiFi User Guide
Apache NiFi User Guide
- 1.
- 2. Agenda • NiFi Overview •NiFi Architecture • NiFi User Guide • Pipeline Builder Status
- 3. NiFi Overview • Whatis NiFi? • What can NiFi do? • Who uses NiFi? • When do we need NiFi?
- 4. What is ApacheNiFi • Apache NiFi is a dataflow system based on the concepts of flow- based programming
- 5.
- 6.
- 7.
- 8.
- 9. When do weneed NiFi? • Utilize dataflow management system to solve complicated data acquisition, ingest, transform, routing process global enterprise level data flow.
- 10. NiFi Terms 1. Realtime Command and Control 2. Processors – Workers 3. Connections -buffers 4. FLOWFILES – Data Packets Attribute Type SourceSite Feed CONTENT Payload, Data, (bytes -> GBs) 5. Expression Language Operate FlowFiles by Attributes
- 11. NiFi Terms NiFi TermFBP Term Describe FlowFile Information Packet unit of flow data with attributes and content Processor Black Box transformation, route, invoke Connection Bounded Buffer linkage of processors with queue Flow Controller Scheduler broker for processors and connections Process Group Subnet set of processors and connections
- 12. NiFi Architecture ATTRIBUTE ForLineage/Routing Type incremental SourceSite MSSQL Feed Data_Feed CONTENT For Data the data Figure 1: Visual Representation of a Flow File CONNECTION PROCESSOR FLOWFILE PROCESSOR
- 13.
- 14. NiFi Data Flow Processors Workers Threads Routers Connections DataBuffers Back Pressure Prioritization FlowFile Attribute Content 1. Data Flow Design Tips Processor utilization Usage and tests Routing strategy Route on attributes & content NiFi Expression Language Dynamically operate FlowFiles Data Provenance Debug and verify
- 15.
- 16. NiFi User Guide •What is NiFi? • What can NiFi do? • Who uses NiFi? • When do we need NiFi? • ‘How to use it’
- 17. NiFi User Guide •‘How to use it’: • How to operate? • How to debug? • How to design? • What’s more? • What are learning steps?
- 18.
- 19. How To Operate •Tips: Know what processor fits your requirement
- 20.
- 21. How To Operate- Example
- 22. How To Operate- Example
- 23. How To Operate- Example
- 24. How To Operate- Example
- 25. NiFi Preliminary Knowledge •Controller Service : configure connection service • Expression language : most important skill in NiFi data flow design • Templates : deploy data flow templates to other NiFi instance
- 26.
- 27. How To Debug •Bulletin indicator • NiFi log: $NIFI_HOME/nifi-logs/nifi-api.log
- 28. How To Test •GenerateFlowFiles + UpdateAttributes + ReplaceText • Generate data flow. Add tags Add content
- 29. NiFi Data FlowDesign Tips • 1) Processor Utilization Usage and architect • 2) Routing Strategy Route on attributes or content • 3) NiFi Expression Language Dynamically operate FlowFiles • 4) Auto Testing Template Debug and verify
- 30. What’s More? • Surveymany template to understand data flow design • Hortonworks : A Collection of NiFi Examples https://community.hortonworks.com/articles/961/a-collection-of-nifi- examples.html • NiFi Confluence Wiki: Example Data Flow Templates https://cwiki.apache.org/confluence/display/NIFI/ Example+Dataflow+Templates • Awesome NiFi https://github.com/jfrazee/awesome-nifi
- 31. NiFi learning steps •1) Meet NiFi SlideShare-NiFi introduction —> NiFi-Overview • 2) Operate NiFi NiFi-Getting Started —> NiFi-User Guide —> Debug • 3) Data Flow design in NiFi NiFi-Expression language —> NiFi Templates Examples • 4) NiFi In Depth NiFi-Developer Guide —> NiFi Admin Guide —> Contribute