General Information
SSD: ING-INF/05
CFU: 6
Professor: Paolo Garza
Teaching Assistant: Luca Colomba
Announcements
- 26-09-23: The first lecture is scheduled for October 2, 2023, at 16:00 in Classroom 4P
- 26-09-23: No lab activities during the first week of the course
Teaching Material
INTRODUCTION
- Introduction to the course content and exam rules (slides)
- Introduction to Big Data (slides)
- Big Data Architectures (slides)
HADOOP AND MAPREDUCE
- Introduction to Apache Hadoop and the MapReduce programming paradigm (slides)
- Interaction with HDFS and Hadoop by means of the command line (slides)
- Hadoop implementation of MapReduce (slides)
- BigData@Polito environment + Jupyter – How to submit MapReduce jobs on BigData@Polito (slides)
- MapReduce and Hadoop – Advanced Topics: Multiple inputs, Multiple outputs, Distributed cache (slides)
- MapReduce – Design patterns – Part 1 (slides)
- MapReduce – Design patterns – Part 2 (slides)
- MapReduce – Relational Algebra/SQL operators (slides)
SPARK
- Introduction to Apache Spark (slides)
- How to submit Spark applications (slides)
- RDD-based programs RDDs
- Spark SQL, Datasets, and DataFrames (slides)
- Spark SQL – Join examples (ExamplesSparkSQLJoins.zip)
- Data Mining
- Recap data mining tasks (slides) – From the “Data Science And Database Technology” course
- Spark MLlib
- Introduction and Classification of structured data (slides)
- Classification of textual data (slides)
- Textual data classification example code (zip)
- Classification and Parameter tuning (slides)
- Parameter tuning example code (zip)
- Clustering of structured data (slides)
- Clustering example code (zip)
- Itemset and Association rule mining (slides)
- Itemset and Association rule mining example code (zip)
- Linear regression (slides)
- Linear regression example code (zip)
- Spark Streaming (slides)
- Examples: Word Count – Streaming versions (zip)
Exercises
MAP REDUCE
- MapReduce exercises (slides)
- Solutions of Exercises 1-29 (SolutionsExMapReduce.zip)
- How to Write and Compile your Java Application using VSCode (pdf)
- Linux or Mac: Basic project for MapReduce applications (based on maven) (MapReduceBasicProject.zip)
- Windows: Basic project for MapReduce applications (based on maven) (MapReduceBasicProjectWindows.zip)
- How to configure the Windows environment to run MapReduce applications locally on your PC(ConfigureWindowsEnviroment.pdf)
- You must also install JDK 1.8 and select it for the imported project inside the IDE. If you already installed the JDK environment but the version is greater than JDK 1.8 you must also install JDK 1.8.
- Winutils executable (winutils.zip)
- If you use your PC to write and run your code locally, use the projects based on Maven (those projects can be run locally).
- If you use the PC available in the LAB, import the projects with libraries as reported in the first lab (those projects cannot be run locally but only on the cluster exporting the project jar file).
SPARK
- Spark RDD-, Dataset-, DataFrame-based exercises (slides)
- Example data – One folder with (few) data for each exercise (ExampleDataSpark.zip)
- Solutions of Exercises 30-50 (SolutionsExSpark30-50.zip) – Updated December 6, 2023 – A new alternative solution for Exercise 44, based on left outer join, has been uploaded (Exercise44v2).
- Solutions of Exercises from 32 to 38 and 44 based on Spark SQL (SolSparkSQL32-38_44.zip) – Updated December 6, 2023 – One more alternative solution for Exercise 44 has been uploaded.
- Spark MLlib (slides)
- Solution (MLlibExampleExercise.zip)
- Spark streaming exercises (slides)
- Solutions of Exercises 51-53 (SolutionsSparkStreaming51_53.zip)
Laboratory Material
No lab activities during the first week of the course
Team 1: Students from A to K – Friday from 14:30 to 16:00 – LAIB1
Team 2: Students from L to Z – Friday from 16:00 to 17:30 – LAIB1
- How to Write and Compile your Java Application using VSCode (pdf)
| Problem specification and input data | Solution (Maven-based) |
| Lab 1: Hadoop and MapReduce Problem specification (pdf) Basic project and small example dataset (Lab1_BigData_with_libraries_vscode.zip) Basic project based on Maven – Use this version to run the MapReduce application locally on your own PC (DO NOT USE IT AT LAIB1) — Linux and macOS (Lab1.zip) — Windows (Lab1_Windows.zip) Bigger dataset: finefoods_text.txt (zip) | Solution: Bonus track Lab1_SolBonusMvn.zip |
| Lab 2: Filter with Hadoop MapReduce Problem specification (pdf) Skeleton project Hadoop — MapReduce (Lab2_Skeleton_with_libraries_vscode.zip) Basic project based on Maven — Use this version of the project to run the MapReduce application locally on your own PC (DO NOT USE IT AT LAIB1) — Linux and macOS (Lab2_Skeleton.zip) — Windows (Lab2Windows_Skeleton.zip) Outputs of the first lab (OutputFolderLab1.zip) (OutputFolderLab1BonusTrack.zip). You can use them to test your application locally on your own PC if you are using Maven | Solution: Lab2_Sol.zip Solution Bonus track: Lab2_SolBonus.zip |
| Lab 3: Frequently bought/reviewed together with Hadoop and MapReduce Problem specification (pdf) Skeleton project Hadoop — MapReduce (Lab3_Skeleton_with_libraries_vscode.zip) Sample data (AmazonTransposedDataset_Sample.txt) Basic project based on Maven — Use this version of the project to run the MapReduce application locally on your own PC (DO NOT USE IT AT LAIB1) — Linux and macOS (Lab3_Skeleton.zip) — Windows (Lab3Windows_Skeleton.zip) | Solution: Lab3_Sol.zip – The second solution MUST NOT BE USED because it is highly inefficient. The second solution has been uploaded to show an inefficient solution that someone implemented in the past. Comments on the three uploaded solutions (slides) |
| Lab 4: Normalized ratings for product recommendations with Hadoop MapReduce Problem specification (pdf) Skeleton project Hadoop — MapReduce (Lab4_Skeleton_with_libraries_vscode.zip) Sample file (ReviewsSample.csv) Basic project based on Maven — Use this version of the project to run the MapReduce application locally on your own PC (DO NOT USE IT AT LAIB1) — Linux and macOS (Lab4_Skeleton.zip) — Windows (Lab4Windows_Skeleton.zip) | Solution: Lab4_Sol.zip |
| Lab 5: Filter data and compute basic statistics with Apache Spark Problem specification (pdf) Skeleton project Spark (Lab5_Skeleton_with_libraries.zip) Sample file (SampleLocalFile.csv) Basic project based on Maven — Use this version of the project to run the Spark application locally on your own PC (DO NOT USE IT AT LAIB1) — Linux, macOS, Windows (Lab5_Skeleton.zip) | Solution: Lab5_Sol.zip |
| Lab 6: Frequently bought/reviewed together with Apache Spark Problem specification (pdf) Skeleton project Spark (Lab6_Skeleton_with_libraries.zip) Sample file (ReviewsSample.csv) Basic project based on Maven — Use this version of the project to run the Spark application locally on your own PC (DO NOT USE IT AT LAIB1) — Linux, macOS, Windows (Lab6_Skeleton.zip) Expected output – Task 1 (expected output if the input is the HDFS file Reviews.csv) (outputTask1Lab6.zip) | Solution: Lab6_Sol.zip |
| Lab 7: Bike sharing data analysis Problem specification (pdf) Skeleton project Spark (Lab7_Skeleton_with_libraries.zip) Sample data (sampleData.zip) Example KML file (exampleKML.zip) Basic project based on Maven – Use this version of the project to run the Spark application locally on your own PC (DO NOT USE IT AT LAIB1) — Linux, macOS, Windows (Lab7_Skeleton.zip) Expected output — Execution on sample data (sampleData/registerSample.csv and sampleData/stations.csv) and minimum criticality threshold = 0.4 (part-00000) — Execution on complete data (/data/students/bigdata-01QYD/Lab7/register.csv and /data/students/bigdata-01QYD/Lab7/stations.csv) and minimum criticality threshold = 0.6 (part-00000) | Solution: Lab7_Sol.zip |
| Lab 8: Bike sharing data analysis based on Spark SQL Problem specification (pdf) Skeleton project Spark (Lab8_Skeleton_with_libraries.zip) Basic project based on Maven – Use this version of the project to run the Spark application locally on your own PC (DO NOT USE IT AT LAIB1) — Linux, macOS, Windows (Lab8_Skeleton.zip) Sample data (sampleData.zip) | Solution: Lab8_Sol.zip |
| Lab 9: A classification pipeline with MLlib + SparkSQL Problem specification (pdf) Skeleton project Spark (Lab9_Skeleton_with_libraries.zip) Basic project based on Maven – Use this version of the project to run the Spark application locally on your own PC (DO NOT USE IT AT LAIB1) — Linux, macOS, Windows (Lab9_Skeleton.zip) Sample file with 100 reviews (ReviewsSample.csv) | Solution: — Logistic regression (zip) — Decision Tree (zip) — Logistic regression based on text analysis (zip) — Decision Tree based on text analysis (zip) |
| Lab 10: Tweet Analysis — Spark Streaming Problem specification (pdf) Skeleton project Spark (Lab10_Skeleton_with_libraries.zip) Basic project based on Maven – Use this version of the project to run the Spark application locally on your own PC (DO NOT USE IT AT LAIB1) — Linux, macOS, Windows (Lab10_Skeleton.zip) Example files — tweets (exampledata_tweets.zip) | Solution: Lab10_sol.zip |
Previous exam examples
| Exams | Solutions |
| Spark Streaming – Examples of multiple choice questions (pdf) | Question 1: (c) Question 2: (d) Question 3: (b) |
| February 19, 2024: pdf | Question 1: (a) Question 2: (b) Source code: zip |
| February 5, 2024: pdf | Question 1: (a) Question 2: (b) Source code: zip |
| September 21, 2023: pdf | Question 1: (a) Question 2: (b) |
| June 21, 2023: pdf | Question 1: (c) Question 2: (d) |
| February 15, 2023: pdf | Question 1: (c) Question 2: (b) Source code: zip – Updated January 16, 2024 – Spark: a solution based on Spark SQL has been uploaded |
| February 2, 2023: pdf | Question 1: (d) – Updated January 19, 2024 – The reported answer was wrong Question 2: (d) Source code: zip – Updated December 17, 2023 – Spark: a second possible solution based on RDDs and a solution based on Spark SQL have been uploaded |
| September 6, 2022: pdf | Question 1: (c) Question 2: (d) Source code: zip – Updated January 22, 2024 – Spark: a solution based on Spark SQL has been uploaded |
| July 4, 2022: pdf | Question 1: (c) Question 2: (d) Source code: zip |
| February 21, 2022: pdf | Question 1: (d) Question 2: (b) Source code: zip – Updated January 15, 2024 – Spark: a solution based on Spark SQL has been uploaded |
| February 2, 2022: pdf | Question 1: (b) Question 2: (d) Source code: zip |
| June 30, 2021: pdf | Question 1: (a) Question 2: (c) Source code: zip |
| February 5, 2021: pdf | Question 1: (b) Question 2: (c) Source code: zip |
| September 17, 2020: pdf | Question 1: (d) Question 2: (c) Source code: zip |
| July 16, 2020: pdf | Question 1: (b) Question 2: (b) – Note that there are two actions; hence, the input file is read twice. Source code: zip |
| July 2, 2020: pdf | Question 1: (b) Question 2: (a) Source code: zip |
| July 18, 2019: pdf | Question 1: (b) Question 2: (b) Source code: zip |
| July 2, 2019: pdf | Question 1: (a) Question 2: (b) Source code: zip |
| February 15, 2019: pdf | Question 1: (d) Question 2: (c) Source code: zip |
| September 3, 2018: pdf | Question 1: (d) Question 2: (c) Source code: zip |
| July 16, 2018: pdf | Question 1: (d) Question 2: (a) Source code: zip |
| June 26, 2018: pdf | Question 1: (c) Question 2: (c) Source code: zip |
Additional material
- Slides and screencasts about Java (kindly provided by Prof. Torchiano) (link)
- Suggested slides/lectures for those students who have never used Java
- OO Paradigm and UML (The UML part is not mandatory)
- The Java Environment
- Java Basic Features
- Java Inheritance