Module – 5 Apache Spark

1. Explain Spark Operations with an example?

Solution:
Apache Spark is an open-source, distributed computing framework that provides a fast and
general-purpose cluster computing system for big data processing. Spark operates on
distributed datasets and performs various operations to process and transform data. These
operations can be categorized into two types: transformations and actions.

TRANSFORMATIONS:
Transformations are operations that create a new dataset from an existing one. They are
executed lazily, which means that they are not computed immediately but instead build a
lineage of transformations that will be executed when an action is called. Here are some
common transformations:

map: Applies a function to each element of the dataset, producing a new dataset. For
example, if you have a dataset of numbers and you want to square each number:
input_data = [1, 2, 3, 4, 5]
rdd = SparkContext.parallelize(input_data)
squared_rdd = rdd.map(lambda x: x**2)

flatMap: Similar to map, but each input item can be mapped to zero or more output items.
For example, splitting lines of text into words:
lines = ["Hello world", "Spark is great", "Big Data"]
rdd = SparkContext.parallelize(lines)
words_rdd = rdd.flatMap(lambda line: line.split(" "))

distinct: Returns a new dataset with distinct elements.

input_data = [1, 2, 2, 3, 3, 4, 4, 5]
rdd = SparkContext.parallelize(input_data)
distinct_rdd = rdd.distinct()

ACTIONS:
Actions are operations that trigger the execution of transformations and return a value to
the driver program or write data to an external storage system. Actions are the operations
that actually perform computation. Here are some common actions:

collect: Retrieves all the elements of the dataset and returns them to the driver program.
Use this with caution as it can be memory-intensive for large datasets.
input_data = [1, 2, 3, 4, 5]
rdd = SparkContext.parallelize(input_data)
result = rdd.collect()

count: Returns the number of elements in the dataset.

input_data = [1, 2, 3, 4, 5]
rdd = SparkContext.parallelize(input_data)
 count = rdd.count()

first: Returns the first element of the dataset.

input_data = [1, 2, 3, 4, 5]
rdd = SparkContext.parallelize(input_data)
first_element = rdd.first()

2. Perform the below RDD

a. How to read multiple text files into RDD

SOLUTION:

from pyspark import SparkContext

# Create a SparkContext

sc = SparkContext("local", "ReadTextFilesExample")

# Specify the directory containing the text files

input_directory = "dbfs:/FileStore/txtfile/*.txt"

# Use textFile to read the text files

text_files_rdd = sc.textFile(input_directory)

# Print the entire contents of the RDD using collect()

all_elements = text_files_rdd.collect()

# Print each element in the RDD

for element in all_elements:

print(element)

# stop the SparkContext

sc.stop()
b. Read CSV file into RDD
Solution:
from pyspark.sql import SparkSession

# Create a SparkSession
# spark =
SparkSession.builder.appName("ReadCSVIntoDataFrameExample").getOrCreate()

# Specify the path to the CSV file

csv_file_path = "dbfs:/FileStore/Address.csv"

# Read the CSV file into a DataFrame

df = spark.read.csv(csv_file_path, header=True, inferSchema=True)

# Show the DataFrame

df.show()

# Stop the SparkSession

# spark.stop()

c. Ways to create an RDD

Solution:
There are 3 ways to create a rdd

1. Parallelizing an Existing Collection:

2. Reading from External Storage:

3. Creating an RDD from Another RDD:

d. Create an empty RDD

Solutions:
We can create an empty rdd using parallelize method

from pyspark import SparkContext

# Create a SparkContext
sc = SparkContext("local", "EmptyRDDExample")

# Create an empty RDD

empty_rdd = sc.parallelize([])

# Perform operations on the empty RDD (for example, count the elements)
count = empty_rdd.count()
 # Print the result
print(f"Number of elements in the empty RDD: {count}")

# Stop the SparkContext

sc.stop()

e. RDD Pair Functions

Solutions:

 reduceByKey(func)

Combines values for each key using a specified reduction function.

data = [("apple", 3), ("banana", 2), ("apple", 5), ("banana", 1)]

rdd = sc.parallelize(data)

reduced_rdd = rdd.reduceByKey(lambda x, y: x + y)

reduced_rdd.collect()

 sortByKey(ascending=True)

Sorts the RDD by keys

data = [("apple", 3), ("banana", 2), ("orange", 5), ("grape", 1)]

rdd = sc.parallelize(data)

sorted_rdd = rdd.sortByKey()

sorted_rdd.collect()

 mapValues(func)

Applies a function to each value in the RDD without changing the keys.

data = [("apple", 3), ("banana", 2), ("orange", 5)]

rdd = sc.parallelize(data)

mapped_rdd = rdd.mapValues(lambda x: x * 2)

mapped_rdd.collect()
 f. Generate Data Frame from RDD

solution:

from pyspark.sql import SparkSession

# Create a SparkSession

spark = SparkSession.builder.appName("RDDToDataFrameExample").getOrCreate()

# Create an RDD

data = [("apple", 3), ("banana", 2), ("apple", 5), ("banana", 1)]

rdd = spark.sparkContext.parallelize(data)

# Define the schema for the DataFrame

schema = ["fruit", "quantity"]

# Convert the RDD to a DataFrame

df = rdd.toDF(schema)

# Show the DataFrame

df.show()

# Stop the SparkSession

spark.stop()

3. How does spark shuffle works?

Spark shuffle is a crucial operation in Apache Spark that redistributes data across partitions to
facilitate certain transformations and operations. It plays a vital role in enabling efficient data
processing and execution of complex workloads.

Understanding Spark Shuffle

Spark shuffle typically occurs when data needs to be grouped or aggregated based on certain
keys. It involves several steps:
1. Map Phase: The input data is divided into partitions, and each partition is assigned to a Spark
executor. Each executor applies a mapper function to each record, generating key-value
pairs.

2. Sort Phase: The key-value pairs are sorted within each partition using a sorting algorithm.
This ensures that all records with the same key are grouped together.

3. Hash Partitioning: The sorted key-value pairs are hashed and distributed to a specified
number of shuffle partitions. This ensures that records with the same key end up in the same
partition, regardless of their original partition.

4. Reduce Phase: The shuffled key-value pairs are aggregated or reduced within each shuffle
partition using a reducer function. This consolidates the data associated with each key.

5. Write to Disk: The aggregated data is written to disk, typically in temporary files. This is
necessary when the data doesn't fit in memory or when multiple stages of a Spark job
require the same shuffled data.