Connect to phome

If your operating system is Linux or MacOS, open a command-line terminal and type the following command (replace your-username with the username that you chose).

ssh phome.metz.supelec.fr -l your-username

After executing the command, you’ll be prompted to enter the password.

If your operating system is Windows:

1. Launch PuTTY.

2. In the session panel, specify phome.metz.supelec.fr as the host name. Select ssh (port 22) as connection type.

3. In the connection panel, set Enable TCP keepalives and set 30s between keepalives.

4. Click on the button Open and click on the button Yes if you receive a warning informing you the key of the destination server is not cached yet.

5. A command-line terminal should pop up, prompting you to enter your username and password.

Connect to slurm1

In the command-line terminal type the following command:

ssh slurm1

Allocate resources for the job

Once you’re connected to slurm1, you can allocate resources for the job by typing the following command (in place of resa-code you will type a code that will be communicated by the teacher during the tutorial.)

srun --reservation=resa-code -N 1 --ntasks-per-node=4 --pty bash

Source file edition

The Python source files that you’ll be editing in this tutorial are stored in the remote machines under your home directory /usr/users/cpuasi1/your-username. In order to edit them, you have two options:

1. Use a remote text editor, such as vim or nano (Linux users, I’m talking to you!).

or,

2. Download the file to your local machine, edit it with your usual code editor and upload it to the remote machine (Windows and MacOS users, I’m talking to you!).

The first option should only be chosen by users who are already familiar with command-line editors.

As for the other users, keep reading this section.

Creating your working directory in HDFS

In this section, you’ll be walked through the procedure to create a directory in HDFS that you’ll use as your working directory in the lab sessions.

You user account is: cpuasi1_X, where X is between 1 and 28.

In order to create your working directory in HDFS, type the following command in the terminal:

hdfs dfs -mkdir hdfs://sar01:9000/cpuasi1/cpuasi1_X

You can verify that the directory is there by listing the content of the folder hdfs://sar01:9000/cpuasi1/ with the following command:

hdfs dfs -ls hdfs://sar01:9000/cpuasi1/

Preliminary exercise

The datasets that you’ll be using in this tutorial are available under the folder hdfs://sar01:9000/data/ stored in HDFS. In order to see the content of the directory you can type the following command:

hdfs dfs -ls -h hdfs://sar01:9000/data

In order to get some familiarity with the commands necessary to run Spark programs in the cluster, let’s look at an already implemented example.

• Copy the file ~vialle/DCE-Spark/template_wc.py to your home directory by typing the following command:

  cp ~vialle/DCE-Spark/template_wc.py ./wc.py

• If you type the command ls, you should see a file named wc.py in your home directory. This file contains the Python code to count the number of occurrences of words in a text file.

• Open the file wc.py by either using a text editor on the remote machine or by downloading it on your local machine, as explained in the section above.

• Locate the following instruction:

  text_file = sc.textFile("hdfs://...")

  and replace it with the following instruction:

  text_file = sc.textFile("hdfs://sar01:9000/data/sherlock.txt")

  This will create an RDD named text_file with the content of the specified file.

• Similarly, locate the following instruction:

  counts.saveAsTextFile("hdfs://...")

  and replace it with the following instruction (replace cpuasi1_X with your username!):

  counts.saveAsTextFile("hdfs://sar01:9000/cpuasi1/cpuasi1_X/sherlock.out")

  This will create an output directory sherlock.out that will contain the files with the output of the program.

• Run the Python script wc.py with the following command:

  spark-submit --master spark://sar01:7077 wc.py

• When the execution is over, the output will be available under the directory sherlock.out. To verify it, run the following command:

  hdfs dfs -ls -h hdfs://sar01:9000/cpuasi1/cpuasi1_X/sherlock.out

  As usual, remember to replace cpuasi1_X with your username.

• In order to see the result, run the following command:

  hdfs dfs -cat hdfs://sar01:9000/cpuasi1/cpuasi1_X/sherlock.out/*