1 Computing averages

We consider a collection of CSV files containing temperature measurements in the following format:

year,month,day,hours,minutes,seconds,temperature

you can find the files under the directory hdfs://sar01:9000/data/temperatures/

Here are the details for each file:

• File temperatures_86400.csv contains one measurement per day in the years 1980 - 2018.
• File temperatures_2880.csv contains one measurement every 2880 seconds in the years 1980 - 2018.
• File temperatures_86.csv contains one measurement every 86 seconds for the year 1980 alone.
• File temperatures_10.csv contains one measurement every 10 seconds for the years 1980 - 2018.

We intend to implement a Spark algorithm to generate pairs \((y, t_{avg})\), where \(y\) is the year and \(t_{avg}\) is the average temperature in the year.

cp ~vialle/DCE-Spark/template_temperatures.py ./avg_temperatures_first.py

def avg_temperature(theTextFile): 
    temperatures = theTextFile \ 
                        .map(lambda line: line.split(",")) \ 
                        .map(lambda term: (term[0],   [float(term[6])])) \ 
                        .reduceByKey(lambda x, y: x+y) \ 
                        .mapValues(lambda lv: sum(lv)/len(lv)) 
    return temperatures 

input_path = "hdfs://sar01:9000/data/temperatures/"
output_path = "hdfs://sar01:9000/cpuecm1/cpuecm1_XX/"

spark-submit --master spark://sar01:7077 avg_temperatures_first.py temperatures_86400.csv

INFO DAGScheduler: Job 0 finished: runJob at SparkHadoopWriter.scala:78, took 3.478220 s 

cp ~vialle/DCE-Spark/template_temperatures.py ./avg_temperatures_second.py

2 Common friends in a social network

Consider a social network described by a graph encoded in a text file. Each line of the file is a list of identifiers separated by commas. For instance, the line \(A,B,C,D\) means that \(A\) is friend with \(B\), \(C\) and \(D\). An excerpt of the file looks like as follows:

A,B,C,D
B,A,D
C,A,D
D,A,B,C
...

We suppose that the friendship relation is symmetric: \((A, B)\) implies \((B, A)\).

We want to obtain the list of the common friends for each pair of individuals:

(A, B), [D]
(A, C), [D] 
(A, D), [B, C] 
(B, C), [D] 
(B, D), [A] 
(C, D), [A]

As an additional constraint, we want to represent a couple only once and avoid to represent the symmetric couple. In other words, if we output \((A, B)\), we don’t want to output \((B, A)\).

We use the following input files available in folder hdfs://sar01:9000/data/sn/:

• sn_tiny.csv. Small social network, that you can use to test your implementation.

• sn_10k_100k.csv. Social network with \(10^4\) individuals and \(10^5\) links.

• sn_100k_100k.csv. Social network with \(10^5\) individuals and \(10^5\) links.

• sn_1k_100k.csv. Social network with \(10^3\) individuals and \(10^5\) links.

• sn_1m_1m.csv. Social network with \(10^6\) individuals and \(10^6\) links.

3 Creating an inverted index

In folder hdfs://sar01:9000/data/bbc/ you’ll find a collection of 50 articles obtained from the BBC website (2004-2005) organized into five subfolders: business, entertainment, politics, sport and technology.

We want to create an inverted index, which associates each word with the list of the files in which the word occurs. More specifically, for each word, the inverted index will have a list of the names of the files (path relative to the folder /data/bbc) that contain the word.

The inverted index:

• must not contain the same word twice;

• must not contain any stopwords (the list of stopwords is provided in the hdfs://sar01:9000/data/stopwords.txt file);

Moreover:

• Words in the inverted index must only contain letters.

• Words in the inverted index must be lowercase.