Spark Environment Setup Guide (Part 2)

Introduction

Before You Begin

Version Reference

Checking Package Versions and Setting Up pip

1. Upgrade the system’s default Python version (the built-in version can’t support the compute requirements of Spark 4.2.0, so an upgrade is required):

   1 2	sudo dnf install -y python3.11 python3.11-devel sudo alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 1

   

2. Check the installed version:

   1	python3 --version # Python 3.11.13

3. Install the pip package manager:

   1	sudo dnf install python3-pip # enter y to confirm

   

Creating a Virtual Environment (venv)

4. Create the virtual environment:

   1	python3 -m venv .venv # creates a hidden .venv folder in the current directory

5. Activate it:

   1	source .venv/bin/activate

   Once activated, you’ll see (.venv) prefixed to the latest line in your terminal.

At this point, the virtual environment is ready — next, let’s install PySpark and Jupyter-Lab.

Installing PySpark and Jupyter-Lab

1. Install the core packages:

   1 2	pip install pyspark pip install jupyter-lab # lighter-weight than the full jupyter package

2. Generate a Jupyter-Lab config file for external connections (watch your indentation here):

   1	jupyter server --generate-config # by default creates ~/.jupyter/jupyter_server_config.py

3. Open the config file at ~/.jupyter/jupyter_server_config.py and set it to listen for external connections:

   1 2	c.ServerApp.ip = '0.0.0.0' # listen on all interfaces c.ServerApp.allow_remote_access = True # allow remote access

   Warning

   This setting effectively opens Jupyter up to the outside network. It's fine if you're only running this on an internal network or a personal test machine, but if the box has a public IP, make sure you keep Jupyter's built-in token authentication enabled (don't turn it off manually), and consider restricting access by source IP through the firewall. Leaving this wide open — letting anyone connect in and execute code — is a surprisingly common security hole in real-world setups.
   

4. Set the environment variable:

   1 2 3

   vim ~/.bashrc # save and exit with :wq export PYSPARK_DRIVER_PYTHON=jupyter-lab source ~/.bashrc # reload

5. Open up the firewall:

   1 2	sudo firewall-cmd --add-port=8888/tcp --permanent # use --permanent based on your situation sudo firewall-cmd --reload

Starting the Service

1. Launch it:

   1	pyspark # copy the returned URL and token

   

2. Replace localhost (or 127.0.0.1) in the returned URL with the host’s actual IP address, then paste it into your browser.

Testing the Connection

Run a test snippet in Jupyter-Lab and check the output

Run the following code — if you get output similar to the screenshot below, the installation succeeded and everything’s working:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

from pyspark.sql import SparkSession

# Manually initialize the SparkSession
spark = SparkSession.builder \
    .appName("JupyterPySparkTest") \
    .getOrCreate()

# Get the SparkContext (if needed)
sc = spark.sparkContext

# Run a quick test
print("Spark Version:", spark.version)

df = spark.createDataFrame([(1, "PySpark"), (2, "JupyterLab")], ["id", "name"])
df.show()

Closing Thoughts

Congratulations — you’ve extended your Spark environment into something much closer to a real-world development setup. From here on, you can write and debug your logic directly in Jupyter, which is noticeably more interactive and efficient than working purely from a terminal, especially during data exploration and step-by-step validation.