(Day 243) Finishing a PySpark book

Ivan Ivanov · August 31, 2024

Hello :) Today is Day 243!

A quick summary of today:

finished the book - Databricks Certified Associate Developer for Apache Spark Using Python

Chapter 5: Advanced Operations and Optimizations in Spark

1. Different Options to Group Data in Spark DataFrames

Spark DataFrames provide several ways to group data for aggregation and analysis:

GroupBy: Groups data by one or more columns and allows you to perform aggregations on the grouped data.
Rollup: Similar to GroupBy, but it also creates subtotals along with the aggregations, which is useful for hierarchical data.
Cube: Extends Rollup by generating all possible combinations of columns, making it suitable for multidimensional data analysis.
Pivot: Transforms rows into columns, effectively reshaping the DataFrame by aggregating data across multiple categories.

2. Various Types of Joins in Spark

Inner Join: Returns only the rows that have matching keys in both DataFrames. It’s the default join type.
Left Join (Left Outer Join): Returns all rows from the left DataFrame, and matched rows from the right DataFrame. Unmatched rows in the right DataFrame return nulls.
Right Join (Right Outer Join): Returns all rows from the right DataFrame, and matched rows from the left DataFrame. Unmatched rows in the left DataFrame return nulls.
Outer Join (Full Outer Join): Returns all rows when there is a match in either DataFrame, filling nulls where there is no match.
Cross Join (Cartesian Join): Returns the Cartesian product of two DataFrames, i.e., every row from the left is paired with every row from the right. This is a costly operation with large datasets.
Broadcast Join: Optimizes joins by broadcasting (distributing) a small DataFrame to all executors, eliminating the need for shuffling data.
Shuffle Join: Involves shuffling data across the network to group data from different partitions before performing the join. Typically used for larger datasets where broadcasting isn’t feasible.

3. Shuffle and Broadcast Joins

Broadcast Hash Join: Suitable when one of the DataFrames is small enough to fit in memory. Spark broadcasts the smaller DataFrame to all executors, avoiding shuffling and improving performance.
Shuffle Sort-Merge Join: Applied when both DataFrames are large. Data is shuffled across the cluster, and then sorted and merged on the join key. This join is expensive due to network I/O and sorting but is necessary for large datasets where broadcasting isn’t possible.
Optimization Strategies: Use broadcast joins when one DataFrame is small, and shuffle joins for large datasets. You can control the join behavior using Spark’s broadcast() function and configuring settings like spark.sql.autoBroadcastJoinThreshold.

4. Reading and Writing Data to Disk in Spark Using Different Data Formats

CSV: Commonly used text format, easy to read and write, but lacks efficient compression and schema enforcement.
Parquet: Columnar storage format with efficient compression, schema enforcement, and is optimized for analytical queries.
ORC: Similar to Parquet, but with better compression and performance for certain types of queries.
Avro: Row-based format, useful for serializing data for storage and data exchange.
JSON: Flexible text format, good for semi-structured data, but less efficient than Parquet or ORC.
Delta: Provides ACID transactions, schema enforcement, and time travel capabilities, built on top of Parquet.

5. Using Spark SQL for Different Operations

Querying Data: Use SQL-like syntax to query structured data. Spark SQL supports complex queries with joins, aggregations, and subqueries.
Data Manipulation: Perform operations like filtering, grouping, sorting, and joining data using SQL queries.
Data Integration: Spark SQL allows integration with various data sources, such as Hive, JDBC, and others, enabling seamless data retrieval and storage.
Window Functions: Implement advanced analytics using functions like ROW_NUMBER, RANK, and LEAD/LAG for operations over specific data partitions.

6. Catalyst Optimizer

Rule-based Optimization: Applies a set of predefined rules to simplify and optimize the logical plan of a query (e.g., predicate pushdown, constant folding).
Cost-based Optimization (CBO): Uses statistics and cost models to choose the most efficient physical plan by estimating the cost of different execution strategies.
Logical Plan Optimization: Transforms the initial logical plan into an optimized version by eliminating unnecessary operations.
Physical Plan Optimization: Converts the optimized logical plan into a physical plan that can be executed efficiently on the Spark cluster.

7. Narrow and Wide Transformations in Spark

Narrow Transformations: Operations where each input partition contributes to at most one output partition. Examples include map, filter, and flatMap. These transformations are efficient because they do not involve shuffling data across the network.
Wide Transformations: Operations where input partitions contribute to multiple output partitions, requiring data to be shuffled across the network. Examples include groupBy, reduceByKey, and join. Wide transformations are more expensive due to the shuffling step.
When to Use: Prefer narrow transformations when possible to reduce network overhead. Use wide transformations when operations inherently require data shuffling.

8. Data Persistence and Caching Techniques

Caching: Use cache() or persist() to store DataFrames in memory, reducing recomputation when the same data is accessed multiple times. By default, data is cached in memory, but you can choose other storage levels (e.g., disk, memory, or a combination).
Memory Management: Monitor the memory usage and cache only the necessary data. Use the unpersist() method to free up memory when the cached data is no longer needed.
Best Practices: Cache data only when you are going to reuse it, and choose the appropriate storage level based on your resource availability and use case.

9. Data Partitioning Through Repartition and Coalesce

Repartition: Increases or decreases the number of partitions by shuffling data across the network. It is useful when you need to balance the workload across the cluster or improve parallelism for large datasets.
Coalesce: Reduces the number of partitions without a full shuffle. It’s more efficient than repartition() when decreasing the number of partitions, as it tries to minimize data movement.
When to Use: Use repartition() when you need to increase the number of partitions for better parallelism or when dealing with skewed data. Use coalesce() when reducing partitions without requiring a full shuffle.

10. User-Defined Functions (UDFs) and Custom Functions

User-Defined Functions (UDFs): Allow you to implement custom logic that is not available in Spark’s built-in functions. UDFs can be applied to DataFrames for complex data transformations.
Python UDFs (Pandas UDFs): More efficient UDFs that leverage Apache Arrow for optimized data transfer between JVM and Python processes.
When to Use: Use UDFs when you need to perform operations that are not natively supported by Spark. However, prefer built-in functions whenever possible, as UDFs can be less performant due to serialization overhead.

11. Performing Advanced Optimizations in Spark

Catalyst Optimizer: Leverage the Catalyst optimizer’s rule-based and cost-based optimizations to automatically improve query performance.
Adaptive Query Execution (AQE): Adjusts the execution plan at runtime based on actual data statistics, optimizing tasks like join strategies, reducing skew, and managing resources dynamically.
Techniques: Tune configurations (e.g., spark.sql.shuffle.partitions), use efficient data formats like Parquet, and apply optimizations like predicate pushdown and partition pruning.

12. Data-Based Optimization Techniques

Predicate Pushdown: Filters data at the source, reducing the amount of data loaded into Spark.
Partition Pruning: Only reads the relevant partitions of data based on query filters, improving I/O efficiency.
Skew Mitigation: Handle skewed data by techniques like salting, which redistributes skewed keys, or using AQE to automatically handle skew.
Efficient File Formats: Use columnar formats like Parquet and ORC for better compression and query performance.
Bucketing: Pre-partition data into buckets on a specified column to optimize join and aggregation operations.

Chapter 6: SQL Queries in Spark

What is Spark SQL?

Advantages of Spark SQL:
- Unified Data Access: Spark SQL allows you to query structured and semi-structured data using SQL, making it easier to work with data from different sources.
- Integration with Spark Core: Spark SQL seamlessly integrates with Spark’s core APIs, allowing you to mix SQL queries with DataFrame and RDD transformations.
- Optimized Performance: Spark SQL benefits from the Catalyst optimizer, which automatically optimizes query execution plans, resulting in faster data processing.
- Support for Multiple Data Formats: You can easily work with different data formats such as JSON, Parquet, and Hive tables using Spark SQL.
- Compatibility: It supports a wide range of SQL-based data processing operations, making it accessible for users familiar with traditional SQL databases.
Integration with Apache Spark:
- DataFrames and Datasets: Spark SQL is built on top of DataFrames and Datasets, which provide a high-level abstraction for data manipulation. These structures integrate seamlessly with Spark’s core APIs, enabling complex data transformations and machine learning pipelines.
- Interoperability: Spark SQL queries can be written in conjunction with other Spark operations, allowing for a smooth transition between SQL queries and Spark’s functional programming API.
- Performance Gains: The integration ensures that operations such as filtering, grouping, and joining can benefit from Spark’s in-memory computing capabilities.
Key Concepts – DataFrames and Datasets:
- DataFrames: A DataFrame is a distributed collection of data organized into named columns. It is equivalent to a table in a relational database and provides an easy-to-use API for performing data operations.
- Datasets: Datasets are a strongly-typed version of DataFrames, offering the benefits of both RDDs (type-safety) and DataFrames (query optimization via the Catalyst optimizer).

Getting Started with Spark SQL

Loading and Saving Data:
- Reading Data: Spark SQL supports reading data from various sources, including CSV, JSON, Parquet, Avro, ORC, and Hive tables. You can load data into DataFrames using the read API.
- Saving Data: Spark SQL also allows you to save DataFrames back to storage in different formats, offering options for compression, partitioning, and file format choices. The write API enables this functionality.
- Examples: For example, you can load a CSV file using spark.read.csv("file.csv") and save a DataFrame as Parquet using df.write.parquet("output.parquet").
Utilizing Spark SQL to Filter and Select Data Based on Specific Criteria:
- Filtering: Spark SQL allows you to filter data using SQL-like WHERE clauses or DataFrame filter()/where() methods. This is useful for narrowing down datasets based on specific conditions.
- Selecting Data: You can select specific columns or expressions using SQL’s SELECT statement or DataFrame methods like select() and selectExpr().
- Examples: Using SQL: SELECT * FROM table WHERE age > 30, or with DataFrames: df.filter(df("age") > 30).select("name", "age").
Exploring Sorting and Aggregation Operations Using Spark SQL:
- Sorting Data: Spark SQL supports sorting data using SQL’s ORDER BY clause or DataFrame orderBy()/sort() methods. This is useful for organizing data based on specific columns.
- Aggregation: You can perform aggregation operations such as SUM(), AVG(), COUNT(), and GROUP BY in SQL or use DataFrame functions like groupBy() and agg().
- Examples: SQL example: SELECT department, COUNT(*) FROM employees GROUP BY department, or with DataFrames: df.groupBy("department").count().
Grouping and Aggregating Data – Grouping Data Based on Specific Columns and Performing Aggregate Functions:
- Grouping: Grouping data is a common operation in data analysis. Spark SQL supports grouping by specific columns using GROUP BY in SQL or the groupBy() method in DataFrames.
- Aggregate Functions: After grouping, you can apply aggregate functions like SUM(), MAX(), MIN(), and custom aggregations to summarize the data.
- Examples: SQL example: SELECT department, AVG(salary) FROM employees GROUP BY department, or with DataFrames: df.groupBy("department").agg(avg("salary")).

Advanced Spark SQL Operations

Leveraging Window Functions to Perform Advanced Analytical Operations on DataFrames:
- Window Functions: Window functions allow you to perform calculations across a set of rows related to the current row, without collapsing the rows into a single output. Examples include RANK(), ROW_NUMBER(), and LEAD()/LAG().
- Use Cases: These functions are useful for tasks like calculating moving averages, ranking data, and comparing data across rows.
- Examples: SQL example: SELECT name, salary, RANK() OVER (ORDER BY salary DESC) AS rank FROM employees, or with DataFrames: df.withColumn("rank", rank().over(Window.orderBy("salary"))).
User-Defined Functions (UDFs):
- Custom Logic: UDFs allow you to define custom functions that can be applied to DataFrame columns. They are useful when built-in Spark functions do not cover specific use cases.
- Performance Considerations: While UDFs provide flexibility, they may introduce performance overhead due to serialization and deserialization. Use them judiciously, and prefer built-in functions when possible.
- Examples: Define a UDF in Python: from pyspark.sql.functions import udf; @udf def square(x): return x * x; df.withColumn("squared", square(df("value"))).
Working with Complex Data Types – Pivot and Unpivot:
- Pivoting: Pivoting transforms rows into columns, which is useful for reshaping data. Spark SQL provides the pivot() function for this purpose.
- Unpivoting: Unpivoting, or reversing a pivot, converts columns back into rows. Although Spark SQL doesn’t have a direct unpivot function, you can achieve this by using a combination of functions like stack().
- Examples: Pivot example: df.groupBy("category").pivot("year").sum("revenue") to transform sales data across years into separate columns.

Chapter 7: Structured Streaming in Spark

Real-time Data Processing

Real-time data processing involves continuous ingestion, processing, and analysis of streaming data. Key characteristics include low latency, scalability, fault tolerance, and a streaming data model. Advantages include rapid response, personalization, operational efficiency, and situational awareness.

Streaming Architectures

Streaming architectures consist of streaming sources, streaming processing engines, and streaming sinks. Different architectures include event-driven, lambda, and unified streaming architectures. Each architecture has its strengths and trade-offs, making them suitable for various scenarios.

Spark Streaming

Spark Streaming is a real-time data processing framework built on Apache Spark. It extends Spark’s capabilities to support high-throughput, fault-tolerant, and scalable stream processing. Key concepts include:

DStreams: As mentioned earlier, DStreams represent the continuous stream of data in Spark Streaming. They are a sequence of RDDs, where each RDD contains data from a specific time interval. DStreams support various transformations and actions, enabling complex computations on the stream.
Window operations: Window operations allow you to apply transformations on a sliding window of data in the stream. It enables computations over a fixed window size or based on time durations, enabling tasks such as windowed aggregations or time-based joins.
Stateful operations: Spark Streaming gives you the ability to maintain stateful information across batches. It enables operations that require maintaining and updating state, such as cumulative counts.
Checkpointing: Checkpointing is a crucial mechanism in Spark Streaming to ensure fault tolerance and recovery. It periodically saves the metadata about the streaming application, including the configuration, DStream operations, and the processed data. It enables the recovery of the application if there are failures.

Advantages include a unified processing model, high-level abstractions, fault tolerance, and scalability.

Structured Streaming

Structured Streaming is a newer and more expressive streaming API in Apache Spark. It introduces a continuous processing model, treating streaming data as an unbounded table or DataFrame. Key features include:

Expressive API
Fault tolerance
Scalability
Unified batch and streaming
Ecosystem integration

Advantages include low-latency, near-real-time results, and support for windowed computations.

Streaming Fundamentals

Stateless streaming processes events independently, while stateful streaming maintains and utilizes contextual information across events.
Event time and processing time are two important notions in streaming.
Watermarking and late data handling are mechanisms to deal with event time and handle delayed or late-arriving data.

Streaming Sources and Sinks

Streaming sources retrieve data from external systems, while streaming sinks receive processed data and write it to external destinations.
Built-in streaming sources include file, Kafka, socket, and Structured Streaming sources.
Built-in streaming sinks include console, file, Kafka, and foreach sinks.

Advanced Techniques

Handling fault tolerance, schema evolution, and joins are advanced techniques in Structured Streaming. Stream-stream joins and stream-static joins are two types of joins in Structured Streaming.

Future Developments

Future developments in Structured Streaming may include enhanced support for streaming sources and sinks, advanced event time handling, performance optimization, integration with AI and machine learning, and seamless integration with streaming data warehouses.

Overall, this text provides a comprehensive overview of real-time data processing using Apache Spark, highlighting the key concepts, advantages, and techniques in Spark Streaming and Structured Streaming.

Chapter 8: ML with Spark ML

Advantages of Apache Spark for Large-Scale ML

Speed and Performance: Spark’s in-memory computing and optimized data processing accelerate large-scale data processing, reducing time for iterative ML algorithms.
Distributed Computing: Spark distributes data and computations across multiple nodes, making it scalable and suitable for processing massive datasets.
Fault Tolerance: Spark’s built-in fault tolerance recovers from failures by tracking lineage with Resilient Distributed Datasets (RDDs).
Versatility and Flexibility: Spark’s APIs and libraries support various data processing tasks, with MLlib offering a wide range of distributed ML algorithms and tools.
Real-Time and Streaming Capabilities: Spark Streaming enables real-time ML applications, valuable in scenarios like fraud detection and sentiment analysis.
Ecosystem and Community Support: Spark has a vibrant community and a rich ecosystem of tools, ensuring continuous development and extensive resources for learning.

Spark MLlib vs Spark ML

Spark MLlib

RDD-based API: Utilizes the RDD abstraction for distributed data processing, suitable for batch processing and iterative algorithms.
Diverse Algorithms: Offers a wide range of distributed ML algorithms, from classification to clustering.
Feature Engineering: Provides tools for feature extraction, transformation, and selection.
Model Persistence: Supports saving and loading trained models.

Spark ML

DataFrame-based API: Uses DataFrames for a more intuitive and structured data representation, simplifying integration with Spark SQL.
Pipelines: Introduces pipelines for chaining data transformations and model training stages, streamlining complex ML workflows.
Integrated Feature Transformers: Includes built-in feature transformers for easier feature engineering.
Unified API: Provides a consistent API for various ML tasks like classification, regression, and clustering.

When to Use Spark MLlib

Working with older Spark versions without Spark ML support.
Need for low-level control with RDDs.
Access to specific algorithms not available in Spark ML.

When to Use Spark ML

Using Spark 2.0 or later versions.
Preference for a higher-level API with DataFrame support.
Building end-to-end ML pipelines with integrated transformers and pipelines.

Both libraries offer robust ML capabilities, with Spark MLlib focusing on low-level control and distributed algorithms, while Spark ML provides a user-friendly API with DataFrame integration.

Remaining chapter 9 and 10

These two chaters are two mock tests for the Databricks certificate. The certificate registration is $200 so I am not sure if I will do it. Regardless, the book was great ^^

I think next, I will read Streaming Databases

That is all for today!

See you tomorrow :)