PySpark Tutorial : Easy Ways to Create RDD in PySpark | Beginner Guide with Real Examples

Different Ways to Create RDD in PySpark

This tutorial walks you through multiple practical ways to create RDDs (Resilient Distributed Datasets) in PySpark, a fundamental concept in Apache Spark. Whether you're building data pipelines or preparing for Spark interviews, these examples will help you get started confidently.

1. Using parallelize()

This is the simplest way to create an RDD from an in-memory Python list, ideal for testing or small datasets:

numbers = [1, 2, 3, 4, 5]
rdd_parallel = spark.sparkContext.parallelize(numbers)
print(rdd_parallel.collect())

2. Using textFile()

Loads a text file into an RDD, where each line becomes a single record:

rdd_text = spark.sparkContext.textFile("path/to/textfile.txt")
print(rdd_text.take(5))

3. Using wholeTextFiles()

Reads an entire directory of small text files, each returning a tuple of (filename, content):

rdd_whole = spark.sparkContext.wholeTextFiles("path/to/folder")
print(rdd_whole.take(1))

📺 Watch the Video Tutorial

Some of the contents in this website were created with assistance from ChatGPT and Gemini.