Big Data Processing and Analytics (2023/24)

Big Data Processing and Analytics (2023/24)

General Information

SSD: ING-INF/05

CFU: 6

Professor: Paolo Garza

Teaching Assistant: Luca Colomba

Announcements

  • 26-09-23: The first lecture is scheduled for October 2, 2023, at 16:00 in Classroom 4P
  • 26-09-23: No lab activities during the first week of the course

Teaching Material

INTRODUCTION
  • Introduction to the course content and exam rules (slides)
  • Introduction to Big Data (slides)
  • Big Data Architectures (slides)
HADOOP AND MAPREDUCE
  • Introduction to Apache Hadoop and the MapReduce programming paradigm (slides)
    • Interaction with HDFS and Hadoop by means of the command line (slides)
  • Hadoop implementation of MapReduce (slides)
    • BigData@Polito environment + Jupyter – How to submit MapReduce jobs on BigData@Polito (slides)
  • MapReduce and Hadoop – Advanced Topics: Multiple inputs, Multiple outputs, Distributed cache (slides)
  • MapReduce – Design patterns – Part 1 (slides)
  • MapReduce – Design patterns – Part 2 (slides)
  • MapReduce – Relational Algebra/SQL operators (slides)
SPARK
  • Introduction to Apache Spark (slides)
    • How to submit Spark applications (slides)
  • RDD-based programs RDDs
    • Creation, basic transformations, and actions (slides)
    • Key-value pair RDDs: transformations and actions on PairRDDs (slides)
    • DoubleRDDs (slides)
    • Advanced Topics: Cache, accumulators, broadcast variables (slides)
  • Spark SQL, Datasets, and DataFrames (slides)
  • Data Mining
    • Recap data mining tasks (slides) – From the “Data Science And Database Technology” course
  • Spark MLlib
    • Introduction and Classification of structured data (slides)
      • Logistic Regression example code (zip)
      • Decision Trees example code (zip)
      • Decision Trees and Categorical class label example code (zip)
    • Classification of textual data (slides)
      • Textual data classification example code (zip)
    • Classification and Parameter tuning (slides)
      • Parameter tuning example code (zip)
    • Clustering of structured data (slides)
      • Clustering example code (zip)
    • Itemset and Association rule mining (slides)
      • Itemset and Association rule mining example code (zip)
    • Linear regression (slides)
      • Linear regression example code (zip)
    • Spark Streaming (slides)
      • Examples: Word Count – Streaming versions (zip)

Exercises

MAP REDUCE
  • MapReduce exercises (slides)
  • How to Write and Compile your Java Application using VSCode (pdf)
  • Linux or Mac: Basic project for MapReduce applications (based on maven) (MapReduceBasicProject.zip)
  • Windows: Basic project for MapReduce applications (based on maven) (MapReduceBasicProjectWindows.zip)
    • You must also install JDK 1.8 and select it for the imported project inside the IDE. If you already installed the JDK environment but the version is greater than JDK 1.8 you must also install JDK 1.8.
    • Winutils executable (winutils.zip)
  • If you use your PC to write and run your code locally, use the projects based on Maven (those projects can be run locally).
  • If you use the PC available in the LAB, import the projects with libraries as reported in the first lab (those projects cannot be run locally but only on the cluster exporting the project jar file).
SPARK

Laboratory Material

No lab activities during the first week of the course

Team 1: Students from A to K – Friday from 14:30 to 16:00 – LAIB1

Team 2: Students from L to Z – Friday from 16:00 to 17:30 – LAIB1

  • How to Write and Compile your Java Application using VSCode (pdf)
Problem specification and input dataSolution (Maven-based)
Lab 1: Hadoop and MapReduce
Problem specification (pdf)
Basic project and small example dataset (Lab1_BigData_with_libraries_vscode.zip)
Basic project based on Maven – Use this version to run the MapReduce application locally on your own PC (DO NOT USE IT AT LAIB1)
— Linux and macOS (Lab1.zip)
— Windows (Lab1_Windows.zip)
Bigger dataset: finefoods_text.txt (zip)
Solution: Bonus track Lab1_SolBonusMvn.zip
Lab 2: Filter with Hadoop MapReduce
Problem specification (pdf)
Skeleton project Hadoop — MapReduce (Lab2_Skeleton_with_libraries_vscode.zip)
Basic project based on Maven — Use this version of the project to run the MapReduce application locally on your own PC (DO NOT USE IT AT LAIB1)
— Linux and macOS (Lab2_Skeleton.zip)
— Windows (Lab2Windows_Skeleton.zip)
Outputs of the first lab (OutputFolderLab1.zip) (OutputFolderLab1BonusTrack.zip). You can use them to test your application locally on your own PC if you are using Maven
Solution: Lab2_Sol.zip
Solution Bonus track: Lab2_SolBonus.zip
Lab 3: Frequently bought/reviewed together with Hadoop and MapReduce
Problem specification (pdf)
Skeleton project Hadoop — MapReduce (Lab3_Skeleton_with_libraries_vscode.zip)
Sample data (AmazonTransposedDataset_Sample.txt)
Basic project based on Maven — Use this version of the project to run the MapReduce application locally on your own PC (DO NOT USE IT AT LAIB1)
— Linux and macOS (Lab3_Skeleton.zip)
— Windows (Lab3Windows_Skeleton.zip)
Solution: Lab3_Sol.zipThe second solution MUST NOT BE USED because it is highly inefficient. The second solution has been uploaded to show an inefficient solution that someone implemented in the past.
Comments on the three uploaded solutions (slides)
Lab 4: Normalized ratings for product recommendations with Hadoop MapReduce
Problem specification (pdf)
Skeleton project Hadoop — MapReduce (Lab4_Skeleton_with_libraries_vscode.zip)
Sample file (ReviewsSample.csv)
Basic project based on Maven — Use this version of the project to run the MapReduce application locally on your own PC (DO NOT USE IT AT LAIB1)
— Linux and macOS (Lab4_Skeleton.zip)
— Windows (Lab4Windows_Skeleton.zip)
Solution: Lab4_Sol.zip
Lab 5: Filter data and compute basic statistics with Apache Spark
Problem specification (pdf)
Skeleton project Spark (Lab5_Skeleton_with_libraries.zip)
Sample file (SampleLocalFile.csv)
Basic project based on Maven — Use this version of the project to run the MapReduce application locally on your own PC (DO NOT USE IT AT LAIB1)
— Linux, macOS, Windows (Lab5_Skeleton.zip)
Solution: Lab5_Sol.zip
Lab6: Frequently bought/reviewed together with Apache Spark
Problem specification (pdf)
Skeleton project Spark (Lab6_Skeleton_with_libraries.zip)
Sample file (ReviewsSample.csv)
Basic project based on Maven — Use this version of the project to run the MapReduce application locally on your own PC (DO NOT USE IT AT LAIB1)
— Linux, macOS, Windows (Lab6_Skeleton.zip)

Expected output – Task 1 (expected output if the input is the HDFS file Reviews.csv) (outputTask1Lab6.zip)
Solution: Lab6_Sol.zip
Lab7: Bike sharing data analysis
Problem specification (pdf)
Skeleton Eclipse project Spark (Lab7_Skeleton_with_libraries.zip)
Sample data (sampleData.zip)
Example KML file (exampleKML.zip)
Basic project based on Maven – Use this version of the project to run the MapReduce application locally on your own PC (DO NOT USE IT AT LAIB1)
— Linux, macOS, Windows (Lab7_Skeleton.zip)

Expected output
— Execution on sample data (sampleData/registerSample.csv and sampleData/stations.csv) and minimum criticality threshold = 0.4 (part-00000)
— Execution on complete data (/data/students/bigdata-01QYD/Lab7/register.csv and /data/students/bigdata-01QYD/Lab7/stations.csv) and minimum criticality threshold = 0.6 (part-00000)
Solution: Lab7_Sol.zip

Previous exam examples

ExamsSolutions
February 15, 2023: pdfQuestion 1: (b)
Question 2: (b)
Source code: zip
February 2, 2023: pdfQuestion 1: (a)
Question 2: (d)
Source code: zip
September 6, 2022: pdfQuestion 1: (c)
Question 2: (d)
Source code: zip
July 4, 2022: pdfQuestion 1: (c)
Question 2: (d)
Source code: zip
February 21, 2022: pdfQuestion 1: (d)
Question 2: (b)
Source code: zip
February 2, 2022: pdfQuestion 1: (b)
Question 2: (d)
Source code: zip
June 30, 2021: pdfQuestion 1: (a)
Question 2: (c)
Source code: zip
February 5, 2021: pdfQuestion 1: (b)
Question 2: (c)
Source code: zip

Additional material

  • Slides and screencasts about Java (kindly provided by Prof. Torchiano) (link)
    • Suggested slides/lectures for those students who have never used Java
    • OO Paradigm and UML (The UML part is not mandatory)
    • The Java Environment
    • Java Basic Features
    • Java Inheritance