Driver failure recovery
We can achieve driver failure recovery with the help of StreamingContext.getOrCreate(). As previously mentioned, this will either initialize a StreamingContext from a checkpoint that already exists, or create a new one.Â
We will not implement a function called createStreamContext0, which creates a StreamingContext and sets up DStreams to interpret tweets and generate the top five most-used hashtags, using a window every 15 seconds. Instead of invoking createStreamContext() and then calling ssc.start(), we will invoke getOrCreate(), so that if a checkpoint exists, then the StreamingContext will be recreated from the data in the checkpoint Directory. If there is no such directory, or if the application is on its first run, then createStreamContext() will be invoked:
val ssc = StreamingContext.getOrCreate(checkpointDirectory, createStreamContext _)
The following code shows how the function is defined, and how getOrCreate() can be invoked:
val checkpointDirectory = "checkpoints...
