How to Read CSV File into DataFrame from Azure Blob Storage | PySpark Tutorial

In this PySpark tutorial, you'll learn how to read a CSV file from Azure Blob Storage into a Spark DataFrame. Follow this step-by-step guide to integrate Azure storage with PySpark for efficient data processing.

Step 1: Configure Spark to Use SAS Token for Authentication

In Azure Blob Storage, SAS (Shared Access Signature) provides secure delegated access to your storage resources. Below is an example SAS token and how you configure Spark to use it.

# SAS token example (for illustration only)
sas_token = "sp=r&st=2025-03-06T17:28:38Z&se=2026-03-07T01:28:38Z&spr=https&sv=2022-11-02&sr=c&sig=VAI..."
        

Step 2: Define the File Path Using WASBS (Azure Blob Storage)

# Define file path
file_path = "wasbs://<container_name>@<storage_account_name>.blob.core.windows.net/<path_to_your_file>.csv"
        

Step 3: Configure Spark with SAS Token

# Spark configuration for accessing the blob
spark.conf.set(
    "fs.azure.sas.<container_name>.<storage_account_name>.blob.core.windows.net",
    sas_token
)
        

Step 4: Read the CSV File into a DataFrame

# Read CSV file into DataFrame
df = spark.read.format("csv") \
    .option("header", "true") \
    .option("inferSchema", "true") \
    .load(file_path)
        

Step 5: Show the Data and Print Schema

# Display the DataFrame contents
df.show()

# Print the DataFrame schema
df.printSchema()
        

Conclusion

Using the above steps, you can securely connect to Azure Blob Storage using SAS tokens and read CSV files directly into PySpark DataFrames. This method is essential for data processing workflows in big data and cloud environments.

📺 Watch the Full Tutorial Video

For a complete step-by-step video guide, watch the tutorial below:

▶️ Watch on YouTube

Author: Aamir Shahzad

© 2024 PySpark Tutorials. All rights reserved.