Big Data Processing and Analytics (2022/23)
Courses

Big Data Processing and Analytics (2022/23)

General Information

SSD: ING-INF/05

CFU: 6

Professor: Paolo Garza

Teaching Assistant: Luca Colomba

Announcements

  • 21-09-22: The first lecture is scheduled for September 27, 2022, at 10:00 in Classroom 3P
  • 21-09-22: No lab activities during the first week of the course

Teaching Material

  • Introduction to the course content and exam rules (slides)
  • Introduction to Big Data (slides)
  • Big Data Architectures (slides)
  • Hadoop and MapReduce
    • Introduction to Apache Hadoop and the MapReduce programming paradigm (slides)
      • Interaction with HDFS and Hadoop by means of the command line (slides)
    • Hadoop implementation of MapReduce (slides)
      • BigData@Polito environment + Jupyter – How to submit MapReduce jobs on BigData@Polito (slides)
    • MapReduce and Hadoop – Advanced Topics: Multiple inputs, Multiple outputs, Distributed cache (slides)
    • MapReduce – Design patterns – Part 1 (slides)
    • MapReduce – Design patterns – Part 2 (slides)
    • MapReduce – Relational Algebra/SQL operators (slides)
  • Spark
    • Introduction to Apache Spark (slides)
      • How to submit Spark applications (slides)
    • RDD-based programs RDDs
      • Creation, basic transformations, and actions (slides)
      • Key-value pair RDDs: transformations and actions on PairRDDs (slides)
      • DoubleRDDs (slides)
      • Advanced Topics: Cache, accumulators, broadcast variables (slides)
    • Spark SQL, Datasets, and DataFrames (slides)
    • Data Mining
      • Recap data mining tasks (slides) – From the “Data Science And Database Technology” course
    • Spark MLlib
      • Introduction and Classification of structured data (slides)
        • Logistic Regression example code (zip)
        • Decision Trees example code (zip)
        • Decision Trees and Categorical class label example code (zip)
      • Classification of textual data (slides)
        • Textual data classification example code (zip)
      • Classification and Parameter tuning (slides)
        • Parameter tuning example code (zip)
      • Clustering of structured data (slides)
        • Clustering example code (zip)
      • Itemset and Association rule mining (slides)
        • Itemset and Association rule mining example code (zip)
      • Linear regression (slides)
        • Linear regression example code (zip)
    • Spark Streaming (slides)
      • Examples: Word Count – Streaming versions (zip)

Exercise

If you use your PC to write and run your code, import the projects based on Maven (those projects can be run locally).
If you use the PC available in the LAB, import the Eclipse projects with libraries (those projects cannot be run locally but only on the cluster exporting the jar file of the project).

  • MapReduce
    • Basic project
      • Linux and MacOs
        • Basic Eclipse project for MapReduce applications (with libraries) (MapReduceBasicProjectWithLibraries.zip) – Import using Import/General/Existing Projects into Workspace
        • Basic Eclipse project for MapReduce applications (based on maven) (MapReduceBasicProject.zip) – Import it using Import/Maven/Existing Maven Projects
      • Windows
        • Basic Eclipse project for MapReduce applications (with libraries) (MapReduceBasicProjectWithLibraries.zip) – Import using Import/General/Existing Projects into Workspace
        • Setup instructions for running MapReduce applications locally inside Eclipse (ConfigureWindowsEnviroment.pdf)
          • You must install also JDK 1.8 and select it for the imported project inside Eclipse. If you already installed the JDK environment but the version is greater than JDK 1.8 you must install also JDK 1.8.
          • Winutils executable (winutils.zip)
          • Basic Eclipse project for MapReduce applications (based on maven) (MapReduceBasicProjectWindows.zip)

Laboratory Material

No lab activities during the first week of the course

Team 1: Students from A to L – Friday from 8:30 to 10:00 – LAIB5

Team 2: Students from M to Z – Friday from 10:00 to 11:30 – LAIB5

  • Lab1: Hadoop and MapReduce
    • Problem specification (pdf)
    • Basic project and small example data set (Lab1_BigData_with_libraries.zip)
    • Basic project based on Maven – Use this version of the project to run the MapReduce application locally on your own PC (DO NOT USE IT AT LAIB5)
      • Import it using Import/Maven/Existing Maven Projects
      • Bigger data set: finefoods_text.txt (zip)
    • Solution Bonus track Lab1_SolBonusMvn.zip – The project is based on mvn
  • Lab2: Filter with Hadoop MapReduce
  • Lab3: Frequently bought/reviewed together with Hadoop MapReduce
  • Lab4: Normalized ratings for product recommendations with Hadoop MapReduce
  • Lab5: Filter data and compute basic statistics with Apache Spark
    • Problem specification (pdf)
    • Skeleton Eclipse project Spark (Lab5_Skeleton_with_libraries.zip)
    • Sample file (SampleLocalFile.csv)
    • Basic project based on Maven – Use this version of the project to run the MapReduce application locally on your own PC (DO NOT USE IT AT LAIB5)
      • Import it using Import/Maven/Existing Maven Projects
  • Lab6: Frequently bought/reviewed together with Apache Spark
    • Problem specification (pdf)
    • Skeleton Eclipse project Spark (Lab6_Skeleton_with_libraries.zip)
    • Sample file (ReviewsSample.csv)
    • Basic project based on Maven – Use this version of the project to run the MapReduce application locally on your own PC (DO NOT USE IT AT LAIB5)
      • Import it using Import/Maven/Existing Maven Projects
  • Lab7: Bike sharing data analysis
    • Problem specification (pdf)
    • Skeleton Eclipse project Spark (Lab7_Skeleton_with_libraries.zip)
    • Sample data (sampleData.zip)
    • Example KML file (exampleKML.zip)
    • Basic project based on Maven – Use this version of the project to run the MapReduce application locally on your own PC (DO NOT USE IT AT LAIB5)
      • Import it using Import/Maven/Existing Maven Projects
    • Expected output
      • Execution on sample data (sampleData/registerSample.csv and sampleData/stations.csv) and minimum criticality threshold = 0.4 (part-00000)
      • Execution on complete data (/data/students/bigdata-01QYD/Lab7/register.csv and /data/students/bigdata-01QYD/Lab7/stations.csv) and minimum criticality threshold = 0.6 (part-00000)
    • Solution
  • Lab8: Bike sharing data analysis based on Spark SQL
  • Lab9: A classification pipeline with MLlib + SparkSQL
    • Problem specification (pdf)
    • Skeleton Eclipse project – Spark (Lab9_Skeleton_with_libraries.zip)
    • Basic project based on Maven – Use this version of the project to run the Spark application locally on your own PC (DO NOT USE IT AT LAIB5)
      • Import it using Import/Maven/Existing Maven Projects
    • Sample file with 100 reviews (ReviewsSample.csv)
    • Solution
      • Logistic regression (zip)
      • DecisionTree (zip)
      • Logistic regression based on text analysis (zip)
      • DecisionTree based on text analysis (zip)
  • Lab 10: Tweet analysis – Spark streaming

Exam examples

  • This academic year the exam in OPEN BOOK

  • Spark Streaming – Examples of multiple choice questions (pdf)
    • Answers
      • Question 1: (c)
      • Question 2: (d)
      • Question 3: (b)
  • Exam June 30, 2017
    • Exam (pdf)
      • Draft of the solution
        • Question 1: (b)
        • Question 2: (c)
        • Source code/Eclipse projects (zip)
  • Exam July 14, 2017
    • Exam (pdf)
      • Draft of the solution
        • Question 1: (d)
        • Question 2: (c)
        • Source code/Eclipse projects (zip)
  • Exam September 14, 2017
    • Exam (pdf)
      • Draft of the solution
        • Question 1: (a)
        • Question 2: (b)
        • Source code/Eclipse projects (zip)
  • Exam June 26, 2018
    • Exam (pdf)
      • Draft of the solution
        • Question 1: (c)
        • Question 2: (c)
        • Source code/Eclipse projects (zip)
  • Exam July 16, 2018
    • Exam (pdf)
      • Draft of the solution
        • Question 1: (d)
        • Question 2: (a)
        • Source code/Eclipse projects (zip)
  • Exam September 3, 2018
    • Exam (pdf)
      • Draft of the solution
        • Question 1: (d)
        • Question 2: (c)
        • Source code/Eclipse projects (zip)
  • Exam February 15, 2019
    • Exam (pdf)
      • Draft of the solution
        • Question 1: (d)
        • Question 2: (c)
        • Source code/Eclipse projects (zip)
  • Exam July 2, 2019
    • Exam (pdf)
      • Draft of the solution
        • Question 1: (a)
        • Question 2: (b)
        • Source code/Eclipse projects (zip)
  • Exam July 18, 2019
    • Exam (pdf)
      • Draft of the solution
        • Question 1: (b)
        • Question 2: (b)
        • Source code/Eclipse projects (zip)
  • Exam July 2, 2020
    • Exam (pdf)
      • Draft of the solution
        • Question 1: (b)
        • Question 2: (a)
        • Source code/Eclipse projects (zip)
  • Exam July 16, 2020
    • Exam (pdf)
      • Draft of the solution
        • Question 1: (b)
        • Question 2: (b) – Note that there are two actions and hence the input file is read two times.
        • Source code/Eclipse projects (zip)
  • Exam September 17, 2020
    • Exam (pdf)
      • Draft of the solution
        • Question 1: (d)
        • Question 2: (c)
        • Source code/Eclipse projects (zip)
  • Exam February 5, 2021
    • Exam (pdf)
      • Draft of the solution
        • Question 1: (b)
        • Question 2: (c)
        • Source code/Eclipse projects (zip)
  • Exam June 30, 2021
    • Exam (pdf)
      • Draft of the solution
        • Question 1: (a)
        • Question 2: (c)
        • Source code/Eclipse projects (zip)
  • Exam February 2, 2022
    • Exam – Version #1 (pdf)
      • Draft of the solution
        • Question 1: (b)
        • Question 2: (d)
        • Source code/Eclipse projects (zip)
  • Exam February 21, 2022
    • Exam – Version #1 (pdf)
      • Draft of the solution
        • Question 1: (d)
        • Question 2: (b)
        • Source code/Eclipse projects (zip)
  • Exam July 4, 2022
    • Exam (pdf)
      • Draft of the solution
        • Question 1: (c)
        • Question 2: (d)
        • Source code/Eclipse projects (zip)
  • Exam September 6, 2022
    • Exam (pdf)
      • Draft of the solution
        • Question 1: (c)
        • Question 2: (d)
        • Source code/Eclipse projects (zip)
  • Exam February 2, 2023
    • Exam (pdf)
      • Draft of the solution
        • Question 1: (d)
        • Question 2: (d)
        • Source code/Eclipse projects (zip)
  • Exam February 15, 2023
    • Exam (pdf)
      • Draft of the solution
        • Question 1: (c)
        • Question 2: (b)
        • Source code/Eclipse projects (zip)

Additional material

  • Slides and screencasts about Java (kindly provided by prof. Torchiano) (link)
    • Suggested slides/lectures for those students who have never used Java
      • OO Paradigm and UML (The UML part is not mandatory)
      • The Java Environment
      • Java Basic Features
      • Java Inheritance