Distributed architectures for big data processing and analytics (2022/2023)
Courses

Distributed architectures for big data processing and analytics (2022/2023)

General Information

SSD: ING-INF/05

CFU: 8

Professor: Paolo Garza

Teaching Assistant: Luca Colomba

Announcements

03-03-2023: Lab activities
— Team 1: Students from A to J – Tuesday from 11:30 to 13:00 (First lab activity – March 7, 2023) @ LABINF
— Team 2: Students from K to Z – Friday from 11:30 to 13:00 (First lab activity – Match 10, 2023) @ LABINF

18-02-2023: The first lecture is scheduled for February 27, 2023, at 8:30 in Classroom 27

Teaching Material

Introduction
  • Introduction to the course content and exam rules (pdf)
  • Introduction to Big Data (pdf)
  • Big Data Architectures (pdf)
Hadoop and MapReduce
  • Introduction to Apache Hadoop and the MapReduce programming paradigm (pdf)
    • Interaction with HDFS and Hadoop by means of the command line (pdf)
  • Hadoop implementation of MapReduce (pdf)
    • Source code of the Word Count Ecplise project (WordCount.zip) – Use the import maven project option to import it in Eclipse
    • PDF version of the code (i.e., PDF version of the java files) (WordCountPDF.zip)
    • BigData@Polito environment + Jupyter – How to submit MapReduce jobs on BigData@Polito (pdf)
  • MapReduce – Design patterns – Part 1 (pdf)
  • MapReduce and Hadoop – Advanced Topics: Multiple inputs, Multiple outputs, Distributed cache (pdf)
  • MapReduce – Design patterns – Part 2 (pdf)
  • MapReduce – Relational Algebra/SQL operators (pdf)
Spark
  • Introduction to Apache Spark (pdf)
    • How to submit Spark applications (pdf)
    • How to use Jupyter Notebooks for your Spark applications (pdf)
    • You can install PySpark and JupyterLab using Conda/Miniconda/pip (instructions here)
  • RDD-based programs
    • RDDs: creation, basic transformations and actions (pdf)
    • Key-value RDDs: transformations and actions on key-value RDDs (pdf)
    • DoubleRDDs (pdf)
    • Advanced Topics: Cache, accumulators, broadcast variables, custom partitioners, broadcast join (pdf)
  • Spark SQL and DataFrames
  • Data mining and Machine learning algorithms with Spark MLlib
  • GraphX/GraphFrames
    • Introduction to GraphX and GraphFrames (pdf)
    • Graph Algorithms with GraphFrames (pdf)
      • Simple example – Jupyter notebook (GraphFrameExamples.zip)
      • Select kernel GraphFrames (Yarn) to run it on jupyter.polito.it
      • Run “pyspark –packages graphframes:graphframes:0.8.1-spark3.0-s_2.12 –repositories https://repos.spark-packages.org” to run it locally on your PC – Use package graphframes:graphframes:0.8.0-spark2.4-s_2.11 if you locally installed Spark 2 instead of Spark 3
  • Streaming data analytics

Exercises

If you use your PC to write and run your code, import the projects based on Maven (those projects can be run locally).
If you use the PC available in the LAB, import the Eclipse projects with libraries (those projects cannot be run locally but only on the cluster exporting the project jar file).

MapReduce
Spark

Laboratory Material

Team 1: Students from A to J – Tuesday from 11:30 to 13:00 (First lab activity – March 7, 2023) @ LABINF
Team 2: Students from K to Z – Friday from 11:30 to 13:00 (First lab activity – March 10, 2023) @ LABINF

Problem specification and input dataSolution
Lab 1: Hadoop and Map Reduce
Problem specification (pdf)
Basic project with small example dataset (Lab1_DBD_with_libraries.zip)
Basic project based on Maven – Use this version to run the MapReduce application locally on your own PC (DO NOT USE THIS ON LABINF PCs)
— Import it using Import -> Maven -> Existing Maven project
— Linux and macOS (Lab1_DBD_mvn.zip)
— Windows (Lab1_DBD_Windows_mvn.zip)
Bigger dataset: finefoods_text.txt (zip)
— You can use this dataset to test your application locally if you are using Maven		Solution bonus track: Lab1_SolBonusMvn.zip – The project is based on mvn
Lab 2: Filter with Hadoop MapReduce
Problem specification (pdf)
Skeleton Eclipse project – MapReduce (Lab2_DBD_with_libraries.zip)
Basic project based on Maven – Use this version to run the MapReduce application locally on your own PC (DO NOT USE THIS ON LABINF PCs)
— Import it using Import -> Maven -> Existing Maven project
— Linux and macOS (Lab2_DBD_mvn.zip)
— Windows (Lab2_DBD_Windows_mvn.zip)
Outputs of the first lab
— OutputFolderLab1.zip
— OutputFolderLab1BonusTrack.zip
— You can use them to test your application locally on your own PC if you are using Maven		Solution: Lab2_DBD_Sol.zip – This project is based on mvn
Solution Bonus track: Lab2_SolBonus.zip – This project is based on mvn
Lab 3: Frequently bought/reviewed together application with Hadoop MapReduce
Problem specification (pdf)
Skeleton Eclipse project – MapReduce (Lab3_DBD_with_libraries.zip)
Basic project based on Maven – Use this version to run the MapReduce application locally on your own PC (DO NOT USE THIS ON LABINF PCs)Import it using
— Import -> Maven -> Existing Maven project
— Linux and macOS (Lab3_DBD_mvn.zip)
— Windows (Lab3_DBD_Windows_mvn.zip)
Sample file (AmazonTransposedDataset_Sample.txt)
— You can use them to test your application locally on your own PC if you are using Maven		Solution: Lab3_DBD_Sol.zip – This project is based on mvn
Comments on the three uploaded solutions (pdf)
Lab 4: Normalized ratings for product recommendations with Hadoop MapReduce
Problem specification (pdf)
Skeleton Eclipse project Hadoop – MapReduce (Lab4_DBD_with_libraries.zip)
Basic project based on Maven – Use this version to run the MapReduce application locally on your own PC (DO NOT USE THIS ON LABINF PCs)
— Import it using Import -> Maven -> Existing Maven project
— Linux and macOS (Lab4_DBD_mvn.zip)
— Windows (Lab4_DBD_Windows_mvn.zip)
Sample file (ReviewsSample.csv)		Solution: Lab4_DBD_Sol.zip – This project is based on mvn
Lab 5: Filter data and compute basic statistics with Apache Spark
Problem specification (pdf)
Sample file (SampleLocalFile.csv)		Solution: Lab5_DBD_Sol.zip – Jupyter notebook (Lab5_Sol.ipynb) and Python script (Lab5_Sol.py)
Lab 6: Frequently bought/reviewed together application with Apache Spark
Problem specification (pdf)
Sample dataset (ReviewsSample.csv)		Solution: Lab6_DBD_Sol.zip – Jupyter notebook (Lab6_Sol.ipynb) and Python script (Lab6_Sol.py)
Lab 7: Bike sharing data analysis
Problem specification (pdf)
Sample data (zip)
Example KML file (zip)
KML file containing the result of the analysis setting the threshold to 0.6 and running the program on the HDFS file (zip)		Solution: Lab7_DBD_Sol.zip – Jupyter notebook (Lab7_Sol.ipynb) and Python script (Lab7_Sol.py)
Lab 8: Bike sharing data analysis based on Spark SQL
Problem specification (pdf)
Sample data (zip)		Solution Lab8_DBD_Sol.zip – Jupyter notebooks (Lab8_Sol.ipynb and Lab8_SolSQL.ipynb) and Python scripts (Lab8_Sol.py and Lab8_SolSQL.py)
Lab 9: A classification pipeline with MLlib + SparkSQL
Problem specification (pdf)
Sample data (zip)		Solution Lab9_DBD_Sol.zip – Jupyter notebooks
Lab10: GraphFrame
Problem specification (pdf)
Data (zip)		Solution Lab10_DBD_Sol.zip – Jupyter notebooks
Lab11: Tweet analysis – Spark streaming
Problem specification (pdf)
Example files – tweets (zip)		Solution Lab11_DBD_Sol.zip – Jupyter notebooks
Lab12: Classification with MLlib + Spark Streaming
Problem specification (pdf)
Template (zip)
Streaming only data (zip)
All data – train, test and streaming (zip)		Solution Lab12_DBD_Sol.zip – Jupyter notebooks

Previous exam examples

ExamsSolutions
Exam July 19, 2023 (pdf)Question 1: (a), Question 2: (b)
MapReduce and Spark (DBD_Exam20230719Sol.zip)
Exam June 26, 2023 (pdf)Question 1: (b), Question 2: (c)
MapReduce and Spark (DBD_Exam20230626Sol.zip)
Exam September 1, 2022 (pdf)Question 1: (b), Question 2: (d)
MapReduce and Spark (DBD_Exam20220901Sol.zip)
Exam July 18, 2022 (pdf)Question 1: (b), Question 2: (b)
MapReduce and Spark (DBD_Exam20220718Sol.zip)
Exam June 27, 2022 (pdf)Question 1: (c), Question 2: (a)
MapReduce and Spark (DBD_Exam20220607Sol.zip)
Exam February 10, 2022 (pdf)Question 1: (a), Question 2: (b)
MapReduce and Spark (DBD_Exam20220210Sol.zip)
Exam September 17, 2021 (pdf)Question 1: (b), Question 2: (a)
MapReduce and Spark (DBD_Exam20210917.zip)
Exam July 5, 2021 (pdf)Question 1: (c), Question 2: (a)
MapReduce and Spark (DBD_Exam20210705Sol.zip)
Exam June 21, 2021 (pdf)Question 1: (b), Question 2: (a)
MapReduce and Spark (DBD_Exam20210621Sol.zip)
Exam July 20, 2020 (pdf)Question 1: (d), Question 2: (b)
Question 2 – Note that there are three actions. Hence, the input file is read three times.
MapReduce and Spark (DBD_Exam20200720Sol.zip)
Exam June 27, 2020 (pdf)Question 1: (b), Question 2: (a)
MapReduce and Spark (DBD_Exam20200627Sol.zip)
More examples of multiple choice questions (pdf)
Question 1: (c)
Question 2: (d)
Question 3: (d)
Question 4: (d)
Question 5: (b)
Question 6: (d)
GraphFrame – Examples of multiple choice questions (pdf)Question 1: (d)
Question 2: (c)

Additional material

Slides and screencasts about Java (kindly provided by Prof. Torchiano) (link)
Focus on the following subset of slides/lectures (for students who have never used Java):
— OO Paradigm and UML (The UML part is not mandatory)
— The Java Environment
— Java Basic Features
— Java Inheritance