Big Data Processing and Analytics (2025/26)
Courses

Big Data Processing and Analytics (2025/26)

General Information

SSD: ING-INF/05

CFU: 6

Professor: Paolo Garza

Teaching Assistants: Lorenzo Vaiani and Etibar Vazirov

Announcements

The first lecture is scheduled for Monday, September 22, 2025, at 14:30 in Room R3 (+ streaming of the virtual classroom)

Teaching Material

INTRODUCTION
  • Introduction to the course content and exam rules (slides)
  • Introduction to Big Data (slides)
  • Big Data Architectures (slides)
HADOOP AND MAPREDUCE
  • Introduction to Apache Hadoop and the MapReduce programming paradigm (slides)
    • Interaction with HDFS and Hadoop by means of the command line (slides)
  • Hadoop implementation of MapReduce (slides)
    • BigData@Polito environment + Jupyter – How to submit MapReduce jobs on BigData@Polito (slides)
  • MapReduce and Hadoop – Advanced Topics: Multiple inputs, Multiple outputs, Distributed cache (slides)
  • MapReduce – Design patterns – Part 1 (slides)
  • MapReduce – Design patterns – Part 2 (slides)
  • MapReduce – Relational Algebra/SQL operators (slides)
SPARK
  • Introduction to Apache Spark (slides)
    • How to submit Spark applications (slides)
    • How to use Jupyter Notebooks for your Spark applications (pdf)
    • You can install PySpark and JupyterLab using Conda/Miniconda/pip (instructions here)
  • RDD-based programs RDDs
    • Creation, basic transformations, and actions (slides) – Notebook with some examples from the slides (FirstExamplesNotebook.zip)
    • Key-value pair RDDs: transformations and actions on PairRDDs (slides)
      • Inner join, left outer join, right outer join, full outer join, and “NOT IN” with PairRDDs: Examples – Notebook (JoinsRDD.zip)
    • DoubleRDDs (slides)
    • Advanced Topics: Cache, accumulators, broadcast variables (slides) – Notebooks with some examples (ExamplesAccumulatorPython.zip)
  • Spark SQL and DataFrames (slides)
  • Spark MLlib
    • Introduction to MLlib (slides)
    • Classification of structured data and textual data (slides)
      • Classification example code (zip)
    • Regression (slides) – Not covered this academic year
      • Linear regression example code (zip) – Not covered this academic year
    • Clustering of structured data (slides) – Not covered this academic year
      • Clustering example code (zip) – Not covered this academic year
    • Itemset and Association rule mining (slides) – Not covered this academic year
      • Itemset and Association rule mining example code (zip) – Not covered this academic year
  • Spark Streaming (slides)

Exercises

MAP REDUCE
  • MapReduce exercises (slides)
  • How to Write and Compile your Java Application using VSCode (pdf)
  • Linux or Mac: Basic project for MapReduce applications (based on maven) (MapReduceBasicProject.zip)
  • Windows: Basic project for MapReduce applications (based on maven) (MapReduceBasicProjectWindows.zip)
    • You must also install JDK 1.8 and select it for the imported project inside the IDE. If you have already installed the JDK environment but the version is greater than JDK 1.8, you must also install JDK 1.8.
    • Winutils executable (winutils.zip)
  • If you use your PC to write and run your code locally, use the projects based on Maven (those projects can be run locally).
  • If you use the PC available in the LAB, import the projects with libraries as reported in the first lab (those projects cannot be run locally, but only on the cluster by exporting the project jar file).
SPARK

Laboratory Material

No lab activities during the first week of the course

Previous exam examples

The Spark solutions of some past exams are still based on Java. However, except for the syntax, the solutions are based on the same Spark methods and workflows. The solution workflow is programming language-independent.

ExamsSolutions
Spark Streaming – Examples of multiple choice questions (pdf) Question 1: (c)
Question 2: (d)
Question 3: (b)
June 12, 2025: pdfQuestion 1: (b)
Question 2: (c)
Source code: zip
February 21, 2025: pdfQuestion 1: (b)
Question 2: (a)
Source code: zip
February 4, 2025: pdfQuestion 1: (b)
Question 2: (a)
Source code: zip
September 12, 2024: pdfQuestion 1: (c) – Each file is read 3 times
Question 2: (a)
Source code: zip – Spark – Python-based solution
June 20, 2024: pdfQuestion 1: (b)
Question 2: (a)
Source code: zip – Spark – Python-based solution
February 19, 2024: pdfQuestion 1: (a)
Question 2: (b)
Source code: zip – Spark – Python-based solution
A solution based on Spark SQL is available for this exam
February 5, 2024: pdfQuestion 1: (a)
Question 2: (b)
Source code: zip – Spark – Python-based solution
A solution based on Spark SQL is available for this exam
September 21, 2023: pdfQuestion 1: (a)
Question 2: (b)
Source code: zip – Spark – Python-based solution
June 21, 2023: pdfQuestion 1: (c)
Question 2: (d)
Source code: zip – Spark – Python-based solution
February 15, 2023: pdfQuestion 1: (c)
Question 2: (b)
Source code: zip – Spark – Python-based solution
A solution based on Spark SQL is available for this exam
February 2, 2023: pdfQuestion 1: (d)
Question 2: (d)
Source code: zip – Spark – Python-based solution
A solution based on Spark SQL is available for this exam
September 6, 2022: pdfQuestion 1: (c)
Question 2: (d)
Source code: zip – Spark – Python-based solution
A solution based on Spark SQL is available for this exam
July 4, 2022: pdfQuestion 1: (c)
Question 2: (d)
Source code: zip – Spark – Python-based solution
February 21, 2022: pdfQuestion 1: (d)
Question 2: (b)
Source code: zip – Spark – Python-based solution
A solution based on Spark SQL is available for this exam
February 2, 2022: pdfQuestion 1: (b)
Question 2: (d)
Source code: zip – Spark – Python-based solution
June 30, 2021: pdfQuestion 1: (a)
Question 2: (c)
Source code: zip
February 5, 2021: pdfQuestion 1: (b)
Question 2: (c)
Source code: zip
September 17, 2020: pdfQuestion 1: (d)
Question 2: (c)
Source code: zip
July 16, 2020: pdfQuestion 1: (b)
Question 2: (b) – Note that there are two actions; hence, the input file is read twice.
Source code: zip
July 2, 2020: pdfQuestion 1: (b)
Question 2: (a)
Source code: zip
July 18, 2019: pdfQuestion 1: (b)
Question 2: (b)
Source code: zip
July 2, 2019: pdfQuestion 1: (a)
Question 2: (b)
Source code: zip
February 15, 2019: pdfQuestion 1: (d)
Question 2: (c)
Source code: zip
September 3, 2018: pdfQuestion 1: (d)
Question 2: (c)
Source code: zip – Spark – Python-based solution
A solution based on Spark SQL is available for this exam
Example to show how to create and manage “static windows” with only Spark SQL APIs
July 16, 2018: pdf
Question 1: (d)
Question 2: (a)
Source code: zip
June 26, 2018: pdfQuestion 1: (c)
Question 2: (c)
Source code: zip

Additional material

  • Slides and screencasts about Java (kindly provided by Prof. Torchiano) (link)
    • Suggested slides/lectures for those students who have never used Java
      • OO Paradigm and UML (The UML part can be skipped)
      • The Java Environment
      • Java Basic Features
      • Java Inheritance