How to Use sample() to Randomly Select Data from DataFrames | PySpark Tutorial #pyspark

How to Use sample() in PySpark | Randomly Select Data from DataFrames

In this tutorial, you will learn how to use the sample() function in PySpark to retrieve a random subset of rows from a DataFrame. This is useful for testing, performance tuning, or dealing with large datasets.

Step 1: Create Sample Data

data = [
    ("Aamir Shahzad", "Pakistan", 25),
    ("Ali Raza", "USA", 30),
    ("Bob", "UK", 45),
    ("Lisa", "Canada", 35),
    ("Aamir Shahzad", "Pakistan", 50),
    ("Aamir Shahzad", "Pakistan", 50)
]

Step 2: Create a DataFrame

df = spark.createDataFrame(data, ["Name", "Country", "Age"])

print("Original DataFrame:")
df.show()

Original DataFrame:
+--------------+---------+---+
| Name| Country|Age|
+--------------+---------+---+
|Aamir Shahzad | Pakistan| 25|
| Ali Raza| USA | 30|
| Bob| UK | 45|
| Lisa| Canada | 35|
|Aamir Shahzad | Pakistan| 50|
|Aamir Shahzad | Pakistan| 50|
+--------------+---------+---+

Step 3: Use sample() to Randomly Select Rows

# Sample 50% of the data, without replacement and fixed seed
sampled_df = df.sample(withReplacement=False, fraction=0.5, seed=70)

print("Sampled DataFrame (50% of data):")
sampled_df.show()

Sampled DataFrame (output will vary):
+--------------+---------+---+
| Name| Country|Age|
+--------------+---------+---+
| Bob| UK | 45|
| Lisa| Canada | 35|
+--------------+---------+---+

📺 Watch the Full Tutorial Video

▶️ Watch on YouTube

Author: Aamir Shahzad

© 2024 PySpark Tutorials. All rights reserved.