General Information
SSD: ING-INF/05
CFU: 6
Professor: Paolo Garza
Teaching Assistant: Luca Colomba
Announcements
Lecture schedule: December 2 – December 6
Monday, December 2, 14:30-17:30 | On-site lecture (Room R1) + streaming virtual classroom |
Friday, December 6 – 14:30-16:00 | Team 1 – LAIB3 + streaming virtual classroom |
Friday, December 6 – 16:00-17:30 | Team 2 – LAIB3 + streaming virtual classroom |
Team 1: Students from A to K – Friday from 14:30 to 16:00 – LAIB3
Team 2: Students from L to Z – Friday from 16:00 to 17:30 – LAIB3
Teaching Material
INTRODUCTION
- Introduction to the course content and exam rules (slides) – Updated on September 23, 2024
- Introduction to Big Data (slides) – Updated on September 23, 2024
- Big Data Architectures (slides)
HADOOP AND MAPREDUCE
- Introduction to Apache Hadoop and the MapReduce programming paradigm (slides) – Updated on September 23, 2024
- Interaction with HDFS and Hadoop by means of the command line (slides)
- Hadoop implementation of MapReduce (slides)
- BigData@Polito environment + Jupyter – How to submit MapReduce jobs on BigData@Polito (slides)
- MapReduce and Hadoop – Advanced Topics: Multiple inputs, Multiple outputs, Distributed cache (slides) – Counters are not covered this year.
- MapReduce – Design patterns – Part 1 (slides)
- MapReduce – Design patterns – Part 2 (slides)
- MapReduce – Relational Algebra/SQL operators (slides)
SPARK
- Introduction to Apache Spark (slides)
- How to submit Spark applications (slides)
- How to use Jupyter Notebooks for your Spark applications (pdf)
- You can install PySpark and JupyterLab using Conda/Miniconda/pip (instructions here)
- RDD-based programs RDDs
- Creation, basic transformations, and actions (slides) – Notebook with some examples from the slides (FirstExamplesNotebook.zip) – Examples uploaded on October 28
- Key-value pair RDDs: transformations and actions on PairRDDs (slides)
- Inner join, left outer join, right outer join, full outer join, and “NOT IN” with PairRDDs: Examples – Notebook (JoinsRDD.zip)
- DoubleRDDs (slides)
- Advanced Topics: Cache, accumulators, broadcast variables (slides) – Notebooks with some examples (ExamplesAccumulatorPython.zip) – Examples uploaded on November 18
- Spark SQL and DataFrames (slides)
- Simple examples – Jupyter notebook (SparkSimpleExamples.zip)
- Spark SQL – Join examples (ExamplesSparkSQLJoins.zip)
- Spark MLlib
- Introduction to MLlib (slides)
- Classification of structured data and textual data (slides)
- Classification example code (zip)
- Regression (slides)
- Linear regression example code (zip)
- Clustering of structured data (slides)
- Clustering example code (zip)
- Itemset and Association rule mining (slides)
- Itemset and Association rule mining example code (zip)
- Spark Streaming (slides)
- Simple examples – Jupyter notebooks (SparkSteamingExamples.zip)
Exercises
MAP REDUCE
- MapReduce exercises (slides)
- Solutions of Exercises 1-29 (SolutionsExMapReduce.zip)
- How to Write and Compile your Java Application using VSCode (pdf)
- Linux or Mac: Basic project for MapReduce applications (based on maven) (MapReduceBasicProject.zip)
- Windows: Basic project for MapReduce applications (based on maven) (MapReduceBasicProjectWindows.zip)
- How to configure the Windows environment to run MapReduce applications locally on your PC(ConfigureWindowsEnviroment.pdf)
- You must also install JDK 1.8 and select it for the imported project inside the IDE. If you have already installed the JDK environment but the version is greater than JDK 1.8, you must also install JDK 1.8.
- Winutils executable (winutils.zip)
- If you use your PC to write and run your code locally, use the projects based on Maven (those projects can be run locally).
- If you use the PC available in the LAB, import the projects with libraries as reported in the first lab (those projects cannot be run locally but only on the cluster exporting the project jar file).
SPARK
- Spark RDD-based exercises (pdf)
- Example data – One folder with (few) data for each exercise (ExSparkData.zip)
- RDD-based solutions of Exercises 30-46 – Jupyter notebooks (SparkNotebooksSol30_46.zip)
- Solution of Exercise 44 based on Left Outer Join (ex44LeftOuterJoin.zip)
- Solution of Exercise 46 based on Spark SQL APIs + RDD.groupByKey() – Example to show how to create and manage “static windows” with almost only Spark SQL APIs (ex46_DF.zip) – Updated on December 1, 2024
- PySpark Installation Guide
- How to run PySpark applications on your PC or Google Colab: You can install PySpark and JupyterLab using Conda/Miniconda/pip (instructions here)
- Spark SQL exercises (pdf)
- Example data – One folder with (few) data for each exercise (ExSparkSQLData.zip)
- Solutions of Exercises 47-50 – Jupyter notebooks (SparkNotebooksSol47_50.zip)
- Spark MLlib exercises (pdf)
- Example data – One folder with (few) data for each exercise (ExampleMLlibData.zip)
- Solutions of Exercise 51 (SparkNotebooksSol51.zip)
- Spark streaming exercises (pdf)
- Example data – One folder with (few) data for each exercise (ExampleSparkStreamingData.zip)
- Solutions of Exercises 58-65 – Jupyter notebooks (SparkNotebooksSol58_65.zip)
Laboratory Material
Team 1: Students from A to K – Friday from 14:30 to 16:00 – LAIB3
Team 2: Students from L to Z – Friday from 16:00 to 17:30 – LAIB3
- How to Write and Compile your Java Application using VSCode (pdf)
Problem specification and input data | Solution (Maven-based for Java) |
Lab 1: Hadoop and Map Reduce Problem specification (pdf) Basic project and small example dataset (Lab1_BigData_with_libraries_vscode.zip) Basic project based on Maven – Use this version to run the MapReduce application locally on your own PC (DO NOT USE IT AT LAIB3) — Linux and macOS (Lab1.zip) — Windows (Lab1_Windows.zip) Bigger dataset: finefoods_text.txt (zip) | Solution: Bonus track Lab1_SolBonusMvn.zip |
Lab 2: Filter with Hadoop MapReduce and Frequency Count Problem specification (pdf) Skeletion project Hadoop — MapReduce (Lab2_Skeleton_with_libraries_vscode.zip) Basic project based on Maven — Use this version of the project to run the MapReduce application locally on your own PC (DO NOT USE IT AT LAIB3) — Linux and macOS (Lab2_Skeleton.zip) — Windows (Lab2Windows_Skeleton.zip) Outputs of the first lab (OutputFolderLab1.zip). You can use them to test your application locally on your own PC if you are using Maven | Solution: Task 1, Task 2 |
Lab 3: Frequently bought/reviewed together with Hadoop and MapReduce Problem specification (pdf) Skeleton project Hadoop — MapReduce (Lab3_Skeleton_with_libraries_vscode.zip) Sample data (AmazonTransposedDataset_Sample.txt) Basic project based on Maven — Use this version of the project to run the MapReduce application locally on your own PC (DO NOT USE IT AT LAIB3) — Linux and macOS (Lab3_Skeleton.zip) — Windows (Lab3Windows_Skeleton.zip) | Solution: Lab3_Sol.zip – The second solution MUST NOT BE USED because it is highly inefficient. The second solution has been uploaded to show an inefficient solution that someone implemented in the past. Comments on the three uploaded solutions (slides) |
Lab 4: Normalized ratings for product recommendations with Hadoop MapReduce Problem specification (pdf) Skeleton project Hadoop — MapReduce (Lab4_Skeleton_with_libraries_vscode.zip) Sample file (ReviewsSample.csv) Basic project based on Maven — Use this version of the project to run the MapReduce application locally on your own PC (DO NOT USE IT AT LAIB3) — Linux and macOS (Lab4_Skeleton.zip) — Windows (Lab4Windows_Skeleton.zip) | Solution: Lab4_Sol.zip |
Lab 5: Filter data and compute basic statistics with Apache Spark Problem specification (pdf) Sample file (SampleLocalFile.csv) | Solution: Lab5_Sol.zip |
Lab 6: Frequently bought/reviewed together with Apache Spark Problem specification (pdf) Sample file (ReviewsSample.csv) Expected output – Task 1 (expected output if the input is the HDFS file Reviews.csv) (outputTask1Lab6.zip) | Solution: Lab6_Sol.zip |
Lab 7: Bike sharing data analysis Problem specification (pdf) Sample data (sampleData.zip) Example KML file (exampleKML.zip) Expected output — Execution on sample data (sampleData/registerSample.csv and sampleData/stations.csv) and minimum criticality threshold = 0.4 (part-00000) — Execution on complete data (/data/students/bigdata-01QYD/Lab7/register.csv and /data/students/bigdata-01QYD/Lab7/stations.csv) and minimum criticality threshold = 0.6 (part-00000) | Solution: … |
Previous exam examples
The Spark solutions of some past exams are still based on Java. They will be updated to Python in the coming weeks. However, except for the syntax, the solutions are based on the same Spark methods and workflows. The solutions are programming language-independent.
Exams | Solutions |
Spark Streaming – Examples of multiple choice questions (pdf) | Question 1: (c) Question 2: (d) Question 3: (b) |
September 12, 2024: pdf | Question 1: (b) Question 2: (a) Source code: zip – Spark – Python-based solution |
June 20, 2024: pdf | Question 1: (b) Question 2: (a) Source code: zip – Spark – Python-based solution |
February 19, 2024: pdf | Question 1: (a) Question 2: (b) Source code: zip – Spark – Python-based solution |
February 5, 2024: pdf | Question 1: (a) Question 2: (b) Source code: zip – Spark – Python-based solution A solution based on Spark SQL is available for this exam |
September 21, 2023: pdf | Question 1: (a) Question 2: (b) |
June 21, 2023: pdf | Question 1: (c) Question 2: (d) |
February 15, 2023: pdf | Question 1: (c) Question 2: (b) Source code: zip – Spark – Python-based solution A solution based on Spark SQL is available for this exam |
February 2, 2023: pdf | Question 1: (d) Question 2: (d) Source code: zip – Spark: a solution based on Spark SQL is available for this exam |
September 6, 2022: pdf | Question 1: (c) Question 2: (d) Source code: zip – Spark: a solution based on Spark SQL is available for this exam |
July 4, 2022: pdf | Question 1: (c) Question 2: (d) Source code: zip |
February 21, 2022: pdf | Question 1: (d) Question 2: (b) Source code: zip – Spark: a solution based on Spark SQL is available for this exam |
February 2, 2022: pdf | Question 1: (b) Question 2: (d) Source code: zip |
June 30, 2021: pdf | Question 1: (a) Question 2: (c) Source code: zip |
February 5, 2021: pdf | Question 1: (b) Question 2: (c) Source code: zip |
September 17, 2020: pdf | Question 1: (d) Question 2: (c) Source code: zip |
July 16, 2020: pdf | Question 1: (b) Question 2: (b) – Note that there are two actions; hence, the input file is read twice. Source code: zip |
July 2, 2020: pdf | Question 1: (b) Question 2: (a) Source code: zip |
July 18, 2019: pdf | Question 1: (b) Question 2: (b) Source code: zip |
July 2, 2019: pdf | Question 1: (a) Question 2: (b) Source code: zip |
February 15, 2019: pdf | Question 1: (d) Question 2: (c) Source code: zip |
September 3, 2018: pdf | Question 1: (d) Question 2: (c) Source code: zip |
July 16, 2018: pdf | Question 1: (d) Question 2: (a) Source code: zip |
June 26, 2018: pdf | Question 1: (c) Question 2: (c) Source code: zip |
Additional material
- Slides and screencasts about Java (kindly provided by Prof. Torchiano) (link)
- Suggested slides/lectures for those students who have never used Java
- OO Paradigm and UML (The UML part can be skipped)
- The Java Environment
- Java Basic Features
- Java Inheritance
- Suggested slides/lectures for those students who have never used Java