How to Use PySpark hint() for Join Optimization – Broadcast, Shuffle, Merge | PySpark Tutorial

PySpark hint() Function Tutorial – Optimize Joins with Broadcast and Merge

Introduction

The hint() function in PySpark allows developers to influence the query optimizer by suggesting specific join strategies. This tutorial demonstrates how to use hints like broadcast, merge, and shuffle to improve join performance.

1. Create Spark Session

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("HintFunctionDemo").getOrCreate()

2. Sample DataFrames

Create two small DataFrames for employees and departments.

data_employees = [
    (1, "Aamir Shahzad", "Sales"),
    (2, "Ali Raza", "Marketing"),
    (3, "Bob", "Sales"),
    (4, "Lisa", "Engineering"),
    (5, "Charlie", "Marketing"),
    (6, "David", "Sales")
]

columns_employees = ["employee_id", "employee_name", "department"]

employees_df = spark.createDataFrame(data_employees, columns_employees)

data_departments = [
    ("Sales", "New York"),
    ("Marketing", "San Francisco"),
    ("Engineering", "Seattle"),
    ("Finance", "Chicago")
]

columns_departments = ["dept_name", "location"]

departments_df = spark.createDataFrame(data_departments, columns_departments)

employees_df.show()

Output:

+-----------+--------------+-----------+
|employee_id|employee_name |department |
+-----------+--------------+-----------+
|1          |Aamir Shahzad |Sales      |
|2          |Ali Raza      |Marketing  |
|3          |Bob           |Sales      |
|4          |Lisa          |Engineering|
|5          |Charlie       |Marketing  |
|6          |David         |Sales      |
+-----------+--------------+-----------+

3. Example 1: Using the broadcast hint

from pyspark.sql.functions import col

joined_df_broadcast = employees_df.join(
    departments_df.hint("broadcast"),
    employees_df["department"] == departments_df["dept_name"]
)

joined_df_broadcast.explain()
joined_df_broadcast.show()

4. Example 2: Using the merge hint (for sort-merge join)

joined_df_merge = employees_df.join(
    departments_df.hint("merge"),
    employees_df["department"] == departments_df["dept_name"]
)

joined_df_merge.explain()
joined_df_merge.show()

5. Example 3: Using the shuffle hint

joined_df_shuffle = employees_df.join(
    departments_df.hint("shuffle"),
    employees_df["department"] == departments_df["dept_name"]
)

joined_df_shuffle.explain()
joined_df_shuffle.show()

6. Conclusion

Use hint() in PySpark wisely to control join strategies and optimize query performance. Always validate your hint with explain() to ensure Spark chooses the desired execution plan. These hints are especially useful for large datasets where default join strategies may not be optimal.

🎥 Watch the Full Video Tutorial

© 2025 Aamir Shahzad. All rights reserved.