# Spark Environment Setup Guide (Part 2)

<!--more-->

## Introduction

{{< admonition info >}}
This post picks up where the previous one left off — **[Spark Environment Setup Guide](https://as183789043.github.io/theme-document-spark-install/)** — and extends it further. This time we're connecting Jupyter into the mix. With it, writing Spark logic gets a lot easier, especially for interactive debugging and exploring data, which is far more convenient than working purely from a terminal.
{{< /admonition >}}

## Before You Begin

{{< admonition tip >}}
Make sure you already have a working VM and Spark environment in place (single-node or cluster, either is fine) — this post picks up directly from the environment we set up in the previous one.
{{< /admonition >}}

### Version Reference

|  Software   | Version  |
|  ----  | ----  |
| Virtual Machine   | 7.1.4 (or later) |
| Redhat ISO   |  9.8 |
| Java openjdk   |  17.0 |
| Spark  |  4.2.0 |
| Python3  |  3.11 |

## Checking Package Versions and Setting Up pip

1. Upgrade the system's default Python version (the built-in version can't support the compute requirements of Spark 4.2.0, so an upgrade is required):
    ~~~bash
    sudo dnf install -y python3.11 python3.11-devel
    sudo alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 1
    ~~~
    ![point](./point.png)

2. Check the installed version:
    ~~~bash
    python3 --version   # Python 3.11.13
    ~~~

3. Install the pip package manager:
    ~~~bash
    sudo dnf install python3-pip   # enter y to confirm
    ~~~
    ![pip](./pip.png)

### Creating a Virtual Environment (venv)

4. Create the virtual environment:
    ~~~bash
    python3 -m venv .venv   # creates a hidden .venv folder in the current directory
    ~~~

5. Activate it:
    ~~~bash
    source .venv/bin/activate
    ~~~
    Once activated, you'll see `(.venv)` prefixed to the latest line in your terminal.
    ![venv](./venv.png)

At this point, the virtual environment is ready — next, let's install PySpark and Jupyter-Lab.

## Installing PySpark and Jupyter-Lab

1. Install the core packages:
    ~~~bash
    pip install pyspark
    pip install jupyter-lab   # lighter-weight than the full jupyter package
    ~~~

2. Generate a Jupyter-Lab config file for external connections (watch your indentation here):
    ~~~bash
    jupyter server --generate-config   # by default creates ~/.jupyter/jupyter_server_config.py
    ~~~

3. Open the config file at `~/.jupyter/jupyter_server_config.py` and set it to listen for external connections:
    ~~~python
    c.ServerApp.ip = '0.0.0.0'              # listen on all interfaces
    c.ServerApp.allow_remote_access = True   # allow remote access
    ~~~

    {{< admonition warning >}}
    This setting effectively opens Jupyter up to the outside network. It's fine if you're only running this on an internal network or a personal test machine, but if the box has a public IP, make sure you keep Jupyter's built-in token authentication enabled (don't turn it off manually), and consider restricting access by source IP through the firewall. Leaving this wide open — letting anyone connect in and execute code — is a surprisingly common security hole in real-world setups.
    {{< /admonition >}}

4. Set the environment variable:
    ~~~bash
    vim ~/.bashrc   # save and exit with :wq
        export PYSPARK_DRIVER_PYTHON=jupyter-lab
    source ~/.bashrc   # reload
    ~~~

5. Open up the firewall:
    ~~~bash
    sudo firewall-cmd --add-port=8888/tcp --permanent   # use --permanent based on your situation
    sudo firewall-cmd --reload
    ~~~

## Starting the Service

1. Launch it:
    ~~~bash
    pyspark   # copy the returned URL and token
    ~~~
    ![token](./token.png)

2. Replace `localhost` (or `127.0.0.1`) in the returned URL with the host's actual IP address, then paste it into your browser.
   ![UI](./ui.png)

## Testing the Connection

#### Run a test snippet in Jupyter-Lab and check the output

Run the following code — if you get output similar to the screenshot below, the installation succeeded and everything's working:

~~~python
from pyspark.sql import SparkSession

# Manually initialize the SparkSession
spark = SparkSession.builder \
    .appName("JupyterPySparkTest") \
    .getOrCreate()

# Get the SparkContext (if needed)
sc = spark.sparkContext

# Run a quick test
print("Spark Version:", spark.version)

df = spark.createDataFrame([(1, "PySpark"), (2, "JupyterLab")], ["id", "name"])
df.show()
~~~

![result](./result.png)

## Closing Thoughts

Congratulations — you've extended your Spark environment into something much closer to a real-world development setup. From here on, you can write and debug your logic directly in Jupyter, which is noticeably more interactive and efficient than working purely from a terminal, especially during data exploration and step-by-step validation.

---

> Author: Rick  
> URL: https://as183789043.github.io/theme-document-spark-install-extend/