Big Data Processing and Analytics (2024/25)
Courses

Big Data Processing and Analytics (2024/25)

General Information

SSD: ING-INF/05

CFU: 6

Professor: Paolo Garza

Teaching Assistant: Luca Colomba

Announcements

Lecture schedule: October 14 – October 18

Monday, October 14, 14:30-17:30On-site lecture (Room R1) + streaming virtual classroom
Friday, October 18 – 14:30-16:00Team 1 – LAIB3 + streaming virtual classroom
Friday, October 18 – 16:00-17:30Team 2 – LAIB3 + streaming virtual classroom

Team 1: Students from A to K – Friday from 14:30 to 16:00 – LAIB3

Team 2: Students from L to Z – Friday from 16:00 to 17:30 – LAIB3

Teaching Material

INTRODUCTION
  • Introduction to the course content and exam rules (slides) – Updated on September 23, 2024
  • Introduction to Big Data (slides) – Updated on September 23, 2024
  • Big Data Architectures (slides)
HADOOP AND MAPREDUCE
  • Introduction to Apache Hadoop and the MapReduce programming paradigm (slides) – Updated on September 23, 2024
    • Interaction with HDFS and Hadoop by means of the command line (slides)
  • Hadoop implementation of MapReduce (slides)
    • BigData@Polito environment + Jupyter – How to submit MapReduce jobs on BigData@Polito (slides)
  • MapReduce and Hadoop – Advanced Topics: Multiple inputs, Multiple outputs, Distributed cache (slides) – Counters are not covered this year.
  • MapReduce – Design patterns – Part 1 (slides)
  • MapReduce – Design patterns – Part 2 (slides)
  • MapReduce – Relational Algebra/SQL operators (slides)
SPARK
  • Introduction to Apache Spark (slides)
    • How to submit Spark applications (slides)
    • How to use Jupyter Notebooks for your Spark applications (pdf)
    • You can install PySpark and JupyterLab using Conda/Miniconda/pip (instructions here)
  • RDD-based programs RDDs
    • Creation, basic transformations, and actions (slides)
    • Key-value pair RDDs: transformations and actions on PairRDDs (slides)
      • Inner join, left outer join, right outer join, full outer join, and “NOT IN” with PairRDDs: Examples – Notebook (JoinsRDD.zip)
    • DoubleRDDs (slides)
    • Advanced Topics: Cache, accumulators, broadcast variables (slides)
  • Spark SQL, Datasets, and DataFrames (slides)
  • Spark MLlib
    • Introduction to MLlib (slides)
    • Classification of structured data and textual data (slides)
      • Classification example code (zip)
    • Linear regression (slides)
      • Linear regression example code (zip)
    • Classification and Parameter tuning (slides)
      • Parameter tuning example code (zip)
    • Clustering of structured data (slides)
      • Clustering example code (zip)
    • Itemset and Association rule mining (slides)
      • Itemset and Association rule mining example code (zip)
  • Spark Streaming (slides)

Exercises

MAP REDUCE
  • MapReduce exercises (slides)
  • How to Write and Compile your Java Application using VSCode (pdf)
  • Linux or Mac: Basic project for MapReduce applications (based on maven) (MapReduceBasicProject.zip)
  • Windows: Basic project for MapReduce applications (based on maven) (MapReduceBasicProjectWindows.zip)
    • You must also install JDK 1.8 and select it for the imported project inside the IDE. If you have already installed the JDK environment but the version is greater than JDK 1.8, you must also install JDK 1.8.
    • Winutils executable (winutils.zip)
  • If you use your PC to write and run your code locally, use the projects based on Maven (those projects can be run locally).
  • If you use the PC available in the LAB, import the projects with libraries as reported in the first lab (those projects cannot be run locally but only on the cluster exporting the project jar file).
SPARK

Laboratory Material

Team 1: Students from A to K – Friday from 14:30 to 16:00 – LAIB3

Team 2: Students from L to Z – Friday from 16:00 to 17:30 – LAIB3

  • How to Write and Compile your Java Application using VSCode (pdf)
Problem specification and input dataSolution (Maven-based for Java)
Lab 1: Hadoop and Map Reduce
Problem specification (pdf)
Basic project and small example dataset (Lab1_BigData_with_libraries_vscode.zip)
Basic project based on Maven – Use this version to run the MapReduce application locally on your own PC (DO NOT USE IT AT LAIB3)
— Linux and macOS (Lab1.zip)
— Windows (Lab1_Windows.zip)
Bigger dataset: finefoods_text.txt (zip)		Solution: Bonus track Lab1_SolBonusMvn.zip

Previous exam examples

Additional material

  • Slides and screencasts about Java (kindly provided by Prof. Torchiano) (link)
    • Suggested slides/lectures for those students who have never used Java
      • OO Paradigm and UML (The UML part can be skipped)
      • The Java Environment
      • Java Basic Features
      • Java Inheritance