Distributed architectures for big data processing and analytics (2025/2026)
Courses

Distributed architectures for big data processing and analytics (2025/2026)

General Information

SSD: ING-INF/05

CFU: 8

Professor: Paolo Garza

Teaching Assistants: Simone Papicchio

Teaching material

Introduction

  • Introduction to the course content and exam rules (pdf) – This slide deck contains the exam rules, which are also provided in the course description (link)
  • Introduction to Big Data (pdf)
  • Big Data Architectures (pdf)

Hadoop and MapReduce

  • Introduction to Apache Hadoop and the MapReduce programming paradigm (pdf)
  • Hadoop implementation of MapReduce (pdf)
    • BigData@Polito environment + Jupyter – How to submit MapReduce jobs on BigData@Polito (slides)
  • MapReduce – Design patterns – Part 1 (pdf)
  • MapReduce and Hadoop – Advanced Topics: Multiple inputs, Multiple outputs, Distributed cache (pdf)
  • MapReduce – Design patterns – Part 2 (pdf)
  • MapReduce – Relational Algebra/SQL operators (pdf)

Spark

  • Introduction to Apache Spark (pdf)
    • How to submit Spark applications (pdf)
    • How to use Jupyter Notebooks for your Spark applications (pdf)
    • You can install PySpark and JupyterLab using Conda/Miniconda/pip (instructions here)
  • RDD-based programs
    • RDDs: creation, basic transformations, and actions (pdf)
      • Some examples (partially selected from the slides): Examples – Notebook (ExamplesSlides.zip)
    • Key-value RDDs: transformations and actions on key-value RDDs (pdf)
      • Inner join, left outer join, right outer join, full outer join, and “NOT IN” with PairRDDs: Examples – Notebook (JoinsRDD.zip)
    • DoubleRDDs (pdf)
    • Advanced Topics: Cache, accumulators, broadcast variables, custom partitioners, broadcast join (pdf)
  • Spark SQL and DataFrames
  • Data mining and Machine learning algorithms with Spark MLlib
  • GraphX/GraphFrames
    • Introduction to GraphX and GraphFrames (pdf)
    • Graph Algorithms with GraphFrames (pdf)
      • Simple example – Jupyter notebook (GraphFrameExamples.zip)
      • Select kernel GraphFrames (Yarn) to run it on jupyter.polito.it
      • Run “pyspark –packages graphframes:graphframes:0.8.1-spark3.0-s_2.12 –repositories https://repos.spark-packages.org” to run it locally on your PC – Use package graphframes:graphframes:0.8.0-spark2.4-s_2.11 if you locally installed Spark 2 instead of Spark 3
  • Streaming data analytics

Exercises

MapReduce

  • MapReduce Exercises (slides)
  • How to Write and Compile your Java Application using VSCode (pdf)
  • Linux or Mac: Basic project for MapReduce applications (based on Maven) (MapReduceBasicProject.zip) (pdf)
  • Windows: Basic project for MapReduce applications (based on Maven) (MapReduceBasicProjectWindows.zip)
    • How to configure the Windows environment to run MapReduce applications locally on your PC(ConfigureWindowsEnviroment.pdf)
    • You must also install JDK 1.8 and select it for the imported project inside the IDE. If you have already installed the JDK environment but the version is greater than JDK 1.8, you must also install JDK 1.8.
    • Winutils executable (winutils.zip)
  • If you use your PC to write and run your code locally, use the projects based on Maven (those projects can be run locally).
  • If you use the PC available in the LAB, import the projects with libraries as reported in the first lab (those projects cannot be run locally, but only on the cluster by exporting the project jar file).

Spark

Laboratory Material

No lab activities during the first week.

Team 1: Students from A to D – Tuesday from 11:30 to 13:00 (First lab activity – March 3, 2026) @ LABINF
Team 2: Students from E to M – Friday from 11:30 to 13:00 (First lab activity – March 6, 2026) @ LABINF
Team 3: Students from N to Z – Friday from 16:00 to 17:30 (First lab activity – March 6, 2026) @ LABINF

ScheduleProblem specification and input dataSolution (Maven-based for Java)
Team 1: March 3, 2026 – 11:30-13:00
Team 2: March 6, 2026 – 11:30-13:00
Team 3: March 6, 2026 – 16:00-17:30		Lab 1: Hadoop and MapReduce
Problem specification (pdf)
– Basic project with libraries and a small example dataset (Lab1_BigData_with_libraries_vscode.zip)
– Basic project based on Maven – Use this version to run the MapReduce application locally on your own PC (DO NOT USE IT AT LABINF)
— Linux and macOS Maven-based project (Lab1.zip)
— Windows Maven-based project (Lab1_Windows.zip)
Bigger dataset: finefoods_text.txt (zip)		Solution: Bonus track Lab1_SolBonusMvn.zip
Team 1: March 10, 2026 – 11:30-13:00
Team 2: March 13, 2026 – 11:30-13:00
Team 3: March 13, 2026 – 16:00-17:30		Lab 2: Filter with Hadoop MapReduce
Problem specification (pdf)
– Skeleton project Hadoop — MapReduce with libraries (lib)
– Basic Maven project (DO NOT USE IT AT LABINF)
— Linux and macOS Maven-based project (Lab2_Skeleton.zip)
— Windows Maven-based project (Lab2Windows_Skeleton.zip)
Outputs of the first lab (OutputFolderLab1.zip) (OutputFolderLab1BonusTrack.zip). You can use them to test your application locally on your PC.		Solution: Lab2_Sol.zip
Solution Bonus track: Lab2_SolBonus.zip
Team 1: March 17, 2026 – 11:30-13:00
Team 2: March 20, 2026 – 11:30-13:00
Team 3: March 20, 2026 – 16:00-17:30		Lab 3: Frequently bought/reviewed together with Hadoop and MapReduce
Problem specification (pdf)
– Skeleton project Hadoop — MapReduce with libraries (lib)
– Basic Maven project (DO NOT USE IT AT LABINF)
— Linux and macOS Maven-based project (Lab3_Skeleton.zip)
— Windows Maven-based project (Lab3Windows_Skeleton.zip)
Sample data (AmazonTransposedDataset_Sample.txt)		Solution: Lab3_Sol.zip
— Comments on the three uploaded solutions (pdf)
— The second solution MUST NOT BE USED – It is highly inefficient
Team 2 and Team 3: March 23, 2026 – 14:30-16:00 – LAIB 1
Team 3: March 23, 2026 – 16:00-17:30 – LABINF		Lab 4: Normalized ratings for product recommendations with Hadoop MapReduce
Problem specification (pdf)
– Skeleton project Hadoop – MapReduce with libraries (Lab4_Skeleton_with_libraries_vscode.zip)
– Basic Maven project (DO NOT USE IT AT LABINF)
— Linux and macOS Maven-based project (Lab4_Skeleton.zip)
— Windows Maven-based project (Lab4Windows_Skeleton.zip)
Sample file (ReviewsSample.csv)
Large file (Reviews_cleaned.csv)		Solution: Lab4_Sol.zip

Previous exam examples

ExamsSolutions
Exam July 11, 2025 (pdf)Question 1: (c)
Question 2: (c)
MapReduce and Spark (DBD_Exam20250711Sol.zip)
Exam June 27, 2025 (pdf)Question 1: (a)
Question 2: (b)
MapReduce and Spark (DBD_Exam20250627Sol.zip)
Exam February 10, 2025 (pdf)Question 1: (b)
Question 2: (a)
Exam September 6, 2024 (pdf)Question 1: (a) – The three codes are equivalent. They are based on commutative functions/methods.
Question 2: (a) – There are 3 distinct keys emitted by the map phase. Hence, the reduce method is invoked 3 times. It follows that the sum of the values of the three instances of numCitiesD is 3.
MapReduce and Spark (DBD_Exam20240906Sol.zip)
Exam July 19, 2024 (pdf)Question 1: (b) – 2 times – Three actions are based on the content of the input file, but highTempRDD is cached. Hence, the input file is read once to compute the value of the count action applied to tempRDD and then one more time to compute the content of highTempRDD, which is then used to calculate the results of the actions count and reduce applied to highTempRDD. Globally, due to the cache of highTempRDD, the input file is read twice.
Question 2: (d) – 6 – There are 6 input lines => the map method is invoked, overall, 6 times.
MapReduce and Spark (DBD_Exam20240719Sol.zip)
Exam July 5, 2024 (pdf)Question 1: (c) – Application B is not equivalent to A and C because .reduce(lambda v1,v2: min(v1, v2) ).filter(lambda value : value>5) is not equivalent to .filter(lambda value : value>5).reduce(lambda v1,v2: min(v1, v2) ). The two functions are not commutative.
Question 2: (a) – Considering all instances of the reducer class, the reduce method is invoked 3 times overall (2 + 1 + 0).
MapReduce and Spark (DBD_Exam20240705Sol.zip) – A more efficient solution based on one single job has been uploaded – June 3, 2025
Sketch of a solution based on SQL (SQLBasedSolution.pdf)
Exam February 20, 2024 (pdf)Question 1: (a), Question 2: (b)
MapReduce and Spark (DBD_Exam20240220Sol.zip)
Exam September 18, 2023 (pdf)Question 1: (c), Question 2: (c)
MapReduce and Spark (Paper-based sketch of the solution – No code_ Exam20230918.pdf)
Exam July 19, 2023 (pdf)Question 1: (a), Question 2: (b)
MapReduce and Spark (DBD_Exam20230719Sol.zip) – with an SQL-based solution and some example data – Updated on June 2, 2025
Exam June 26, 2023 (pdf)Question 1: (b), Question 2: (c)
MapReduce and Spark (DBD_Exam20230626Sol.zip) – with an SQL-based solution and some example data
Exam September 1, 2022 (pdf)Question 1: (b), Question 2: (d)
MapReduce and Spark (DBD_Exam20220901Sol.zip)
Exam July 18, 2022 (pdf)Question 1: (b), Question 2: (b)
MapReduce and Spark (DBD_Exam20220718Sol.zip) – with an SQL-based solution – Example related to “static windows” and how to manage them either RDD or Spark SQL APIs
Exam June 27, 2022 (pdf)Question 1: (c), Question 2: (a)
MapReduce and Spark (DBD_Exam20220607Sol.zip) – with an SQL-based solution and some example data
Exam February 10, 2022 (pdf)Question 1: (a), Question 2: (b)
MapReduce and Spark (DBD_Exam20220210Sol.zip)
Exam September 17, 2021 (pdf)Question 1: (b), Question 2: (a)
MapReduce and Spark (DBD_Exam20210917.zip)
Exam July 5, 2021 (pdf)Question 1: (c), Question 2: (a)
MapReduce and Spark (DBD_Exam20210705Sol.zip) – with an SQL-based solution
Exam June 21, 2021 (pdf)Question 1: (b), Question 2: (a)
MapReduce and Spark (DBD_Exam20210621Sol.zip)
Exam July 20, 2020 (pdf)Question 1: (d), Question 2: (b)
Question 2 – Note that there are three actions. Hence, the input file is read three times.
MapReduce and Spark (DBD_Exam20200720Sol.zip)
Exam June 27, 2020 (pdf)Question 1: (b), Question 2: (a)
MapReduce and Spark (DBD_Exam20200627Sol.zip)
More examples of multiple choice questions (pdf)
		Question 1: (c)
Question 2: (d)
Question 3: (d)
Question 4: (d)
Question 5: (b)
Question 6: (d)
GraphFrame – Examples of multiple choice questions (pdf)Question 1: (d)
Question 2: (c)

Additional material

Slides and screencasts about Java (kindly provided by Prof. Torchiano) (link)
Focus on the following subset of slides/lectures (for students who have never used Java):
— OO Paradigm and UML (The UML part is not mandatory)
— The Java Environment
— Java Basic Features
— Java Inheritance

Other material about Java (link)