Technical requirements
In this chapter, you will be using the Databricks Community Edition to run your code. This can be found at https://community.cloud.databricks.com:
- Sign-up instructions can be found at https://databricks.com/try-databricks.
- The code and data used in this chapter can be downloaded from https://github.com/PacktPublishing/Essential-PySpark-for-Scalable-Data-Analytics/tree/main/Chapter04.
Before we dive deeper into implementing real-time stream processing data pipelines with Apache Spark, first, we need to understand the general architecture of a real-time analytics pipeline and its various components, as described in the following section.
