General Information
SSD: ING-INF/05
CFU: 8
Professor: Paolo Garza
Teaching Assistant: Simone Papicchio
Teaching Material
Introduction
- Introduction to the course content and exam rules (pdf)
- Introduction to Big Data (pdf)
- Big Data Architectures (pdf)
Hadoop and MapReduce
- Introduction to Apache Hadoop and the MapReduce programming paradigm (pdf)
- Interaction with HDFS and Hadoop using the command line (pdf)
- Hadoop implementation of MapReduce (pdf)
- Source code of the Word Count Ecplise project (WordCount.zip) – Use the import maven project option to import it into Visual Studio Code
- BigData@Polito environment + Jupyter – How to submit MapReduce jobs on BigData@Polito (pdf)
- MapReduce – Design patterns – Part 1 (pdf)
- MapReduce and Hadoop – Advanced Topics: Multiple inputs, Multiple outputs, Distributed cache (pdf)
- MapReduce – Design patterns – Part 2 (pdf)
- MapReduce – Relational Algebra/SQL operators (pdf)
Spark
- Introduction to Apache Spark (pdf)
- How to submit Spark applications (pdf)
- How to use Jupyter Notebooks for your Spark applications (pdf)
- You can install PySpark and JupyterLab using Conda/Miniconda/pip (instructions here)
- RDD-based programs
- RDDs: creation, basic transformations and actions (pdf)
- Some examples (partially selected from the slides): Examples – Notebook (ExamplesFromSlides.zip)
- Key-value RDDs: transformations and actions on key-value RDDs (pdf)
- Inner join, left outer join, right outer join, full outer join, and “NOT IN” with PairRDDs: Examples – Notebook (JoinsRDD.zip) – Uploaded on April 21, 2024
- DoubleRDDs (pdf)
- Advanced Topics: Cache, accumulators, broadcast variables, custom partitioners, broadcast join (pdf)
- RDD partition examples (RDDPartitionsExamples.zip)
- Introduction to PageRank (pdf) – Example: PageRank “naive” implementation (RDDPageRank.zip)
- RDDs: creation, basic transformations and actions (pdf)
- Spark SQL and DataFrames
- Spark SQL (pdf)
- Simple examples – Jupyter notebook (SparkSQLSimpleExamples.zip)
- Spark SQL join examples – Jupyter notebook (ExamplesSparkSQLJoins.zip)
- Spark SQL (pdf)
- Data mining and Machine learning algorithms with Spark MLlib
- Introduction and Preprocessing (pdf)
- Classification (pdf)
- Classification examples – Jupyter notebooks and sample data (ExampleClassificationMLlib.zip)
- Clustering (pdf)
- Clustering example – Jupyter notebook and sample data (ExampleClusteringMLlib.zip)
- Regression (pdf)
- Regression example – Jupyter notebook and sample data (ExampleRegressionMLlib.zip)
- Itemset and Association rule mining (pdf)
- Itemset and Association rule mining example – Jupyter notebook and sample data (ExampleItemsetMLlib.zip)
- GraphX/GraphFrames
- Introduction to GraphX and GraphFrames (pdf)
- Graph Algorithms with GraphFrames (pdf)
- Simple example – Jupyter notebook (GraphFrameExamples.zip)
- Select kernel GraphFrames (Yarn) to run it on jupyter.polito.it
- Run “pyspark –packages graphframes:graphframes:0.8.1-spark3.0-s_2.12 –repositories https://repos.spark-packages.org” to run it locally on your PC – Use package graphframes:graphframes:0.8.0-spark2.4-s_2.11 if you locally installed Spark 2 instead of Spark 3
- Streaming data analytics
- Spark Streaming Spark Streaming (DStreams) (pdf)
- Simple examples – Jupyter notebooks (SparkSteamingExamples.zip)
- Structured Streaming (pdf)
- Simple examples – Jupyter notebooks (SparkStructutedStreamingExamples.zip)
- Introduction to other big stream processing frameworks: Apache Storm, Apache Flink, .. (pdf)
- Spark Streaming Spark Streaming (DStreams) (pdf)
Exercises
MapReduce
- MapReduce Exercises (slides)
- Solutions of Exercises 1-29 (SolutionsExMapReduce.zip)
- How to configure Visual Studio Code on your personal laptop: 📘guide.
- Note that you must also install JDK 1.8 and select it for the imported project inside the IDE. If you have already installed the JDK environment but the version is greater than JDK 1.8, you must also install JDK 1.8.
Windows users only: You must configure the winutils (🗃️winutils.zip) and set up some environmental variables. Follow this 📘extra guide for the complete configuration.
- Some of you solved the problems with their Windows version by downloading winutils.exe and hadoop.dll from this alternative source: https://github.com/steveloughran/winutils/tree/master/hadoop-2.7.1/bin
There are multiple versions of the basic projects. The version with libraries is the only one you can use on the LABINF computers. Use it on your laptop if you are not interested in running the applications locally. All the other versions are Maven projects, so you can use them locally on your personal laptop to write the code and then run it locally inside Visual Studio Code or on the BigData@Polito cluster. The legend is as follows: 📚lib: Project/template with libraries, 🐧mavU: Maven project for Linux/MacOS, 🪟mavW: Maven project for Windows (Hadoop projects only).
- Basic project for MapReduce applications (📚lib, 🐧mavU, 🪟mavW)
Spark
- Spark exercises (pdf)
- Example data – One folder with (few) data for each exercise (ExSparkData.zip)
- RDD-based solutions of Exercises 30-46 – Jupyter notebooks (SparkNotebooksSol30_46.zip)
- Solution for Exercise 44 based on Left Outer Join (ex44LeftOuterJoin.zip) – Uploaded on April 24, 2024
- PySpark Installation Guide
- How to run PySpark applications on your PC or Google Colab: You can install PySpark and JupyterLab using Conda/Miniconda/pip (instructions here)
- Spark SQL exercises (pdf)
- Example data – One folder with (few) data for each exercise (ExSparkSQLData.zip)
- Solutions of Exercises 47-50 – Jupyter notebooks (SparkNotebooksSol47_50.zip)
- Spark MLlib exercises (pdf)
- Example data – One folder with (few) data for each exercise (ExampleMLlibData.zip)
- Solutions of Exercise 51 (SparkNotebooksSol51.zip)
- GraphFrame exercises (pdf)
- Example data – One folder with (few) data for each exercise (ExampleGraphFrameData.zip)
- Solutions of Exercises 52-57b – Jupyter notebooks (SparkNotebooksSol52_57b.zip)
- Spark streaming exercises (pdf)
- Example data – One folder with (few) data for each exercise (ExampleSparkStreamingData.zip)
- Solutions of Exercises 58-65 – Jupyter notebooks (SparkNotebooksSol58_65.zip)
- Spark structured streaming and MLlib exercise (pdf)
- Example data – One folder with (few) data for each exercise (ExampleSparkStructuredMLlibData.zip)
- Solution of Exercise 66 – Jupyter notebooks (SparkNotebooksSol66.zip)
Laboratory Material
Team 1: Students from A to L – Tuesday from 11:30 to 13:00 (First lab activity – March 12, 2024) @ LABINF
Team 2: Students from M to Z – Friday from 11:30 to 13:00 (First lab activity – March 15, 2024) @ LABINF
- How to configure Visual Studio Code on your personal laptop: 📘guide.
- Note that you must also install JDK 1.8 and select it for the imported project inside the IDE. If you have already installed the JDK environment but the version is greater than JDK 1.8, you must also install JDK 1.8.
- Windows users only: You must configure the winutils (🗃️winutils.zip) and set up some environmental variables. Follow this 📘extra guide for the complete configuration.
- Some of you solved the problems with their Windows version by downloading winutils.exe and hadoop.dll from this alternative source: https://github.com/steveloughran/winutils/tree/master/hadoop-2.7.1/bin
- Laboratory materials are available in multiple versions. The version with libraries is the only one you can use on the LABINF computers. Use it on your laptop if you are not interested in running the applications locally. All the other versions are Maven projects, so you can use them locally on your personal laptop to write the code and then run it locally inside Visual Studio Code or on the BigData@Polito cluster. The legend is as follows: 📚lib: Project/template with libraries, 🐧mavU: Maven project for Linux/MacOS, 🪟mavW: Maven project for Windows (Hadoop projects only).
- Basic project for MapReduce applications (📚lib, 🐧mavU, 🪟mavW)
- How to configure JDK 1.8 on MAC in case of errors with standard procedure:
- Follow the guide on Downloads for Amazon Corretto 8 – Amazon Corretto 8
- PySpark Installation Guide: How to run PySpark applications on your PC or Google Colab: You can install PySpark and JupyterLab using Conda/Miniconda/pip (instructions here)
Problem specifications/Lab solutions
Problem specification and input data | Solution (Maven-based) |
Lab 1: Hadoop and MapReduce Problem specification (pdf) Basic project and small example dataset (Lab1_BigData_with_libraries_vscode.zip) Basic project based on Maven – Use this version to run the MapReduce application locally on your own PC (DO NOT USE IT AT LABINF) — Linux and macOS (Lab1.zip) — Windows (Lab1_Windows.zip) Bigger dataset: finefoods_text.txt (zip) | Solution: Bonus track Lab1_SolBonusMvn.zip |
Lab 2: Filter with Hadoop MapReduce Problem specification (pdf) Skeleton project Hadoop — MapReduce (Lab2_Skeleton_with_libraries_vscode.zip) Basic project based on Maven — Use this version of the project to run the MapReduce application locally on your own PC (DO NOT USE IT AT LABINF) — Linux and macOS (Lab2_Skeleton.zip) — Windows (Lab2Windows_Skeleton.zip) Outputs of the first lab (OutputFolderLab1.zip) (OutputFolderLab1BonusTrack.zip). You can use them to test your application locally on your own PC if you are using Maven | Solution: Lab2_Sol.zip Solution Bonus track: Lab2_SolBonus.zip |
Lab 3: Frequently bought/reviewed together with Hadoop and MapReduce Problem specification (pdf) Skeleton project Hadoop — MapReduce (Lab3_Skeleton_with_libraries_vscode.zip) Sample data (AmazonTransposedDataset_Sample.txt) Basic project based on Maven — Use this version of the project to run the MapReduce application locally on your own PC (DO NOT USE IT AT LABINF) — Linux and macOS (Lab3_Skeleton.zip) — Windows (Lab3Windows_Skeleton.zip) | Solution: Lab3_DBD_Sol.zip – This project is based on mvn — Comments on the three uploaded solutions (pdf) — The second solution MUST NOT BE USED – It is highly inefficient |
Lab 4: Normalized ratings for product recommendations with Hadoop MapReduce Problem specification (pdf) Skeleton Eclipse project Hadoop – MapReduce (Lab4_DBD_with_libraries.zip) Basic project based on Maven – Use this version to run the MapReduce application locally on your own PC (DO NOT USE THIS ON LABINF PCs) — Linux and macOS (Lab4_DBD_mvn.zip) — Windows (Lab4_DBD_Windows_mvn.zip) Sample file (ReviewsSample.csv) | Solution: Lab4_Sol.zip |
Lab 5: Filter data and compute basic statistics with Apache Spark Problem specification (pdf) Sample file (SampleLocalFile.csv) | Solution: Lab5_DBD_Sol.zip — Jupyter notebook (Lab5_Sol.ipynb) — Python script (Lab5_Sol.py) |
Lab 6: Frequently bought/reviewed together application with Apache Spark Problem specification (pdf) Sample dataset (ReviewsSample.csv) | Solution: Lab6_DBD_Sol.zip — Jupyter notebook (Lab6_Sol.ipynb) — Python script (Lab6_Sol.py) |
Lab 7: Bike sharing data analysis Problem specification (pdf) Sample data (zip) Example KML file (zip) KML file containing the result of the analysis setting the threshold to 0.6 and running the program on the HDFS file (zip) |
Previous exam examples
Exams | Solutions |
Exam July 19, 2023 (pdf) | Question 1: (a), Question 2: (b) MapReduce and Spark (DBD_Exam20230719Sol.zip) |
Exam June 26, 2023 (pdf) | Question 1: (b), Question 2: (c) MapReduce and Spark (DBD_Exam20230626Sol.zip) |
Exam September 1, 2022 (pdf) | Question 1: (b), Question 2: (d) MapReduce and Spark (DBD_Exam20220901Sol.zip) |
Exam July 18, 2022 (pdf) | Question 1: (b), Question 2: (b) MapReduce and Spark (DBD_Exam20220718Sol.zip) |
Exam June 27, 2022 (pdf) | Question 1: (c), Question 2: (a) MapReduce and Spark (DBD_Exam20220607Sol.zip) |
Exam February 10, 2022 (pdf) | Question 1: (a), Question 2: (b) MapReduce and Spark (DBD_Exam20220210Sol.zip) |
Exam September 17, 2021 (pdf) | Question 1: (b), Question 2: (a) MapReduce and Spark (DBD_Exam20210917.zip) |
Exam July 5, 2021 (pdf) | Question 1: (c), Question 2: (a) MapReduce and Spark (DBD_Exam20210705Sol.zip) |
Exam June 21, 2021 (pdf) | Question 1: (b), Question 2: (a) MapReduce and Spark (DBD_Exam20210621Sol.zip) |
Exam July 20, 2020 (pdf) | Question 1: (d), Question 2: (b) Question 2 – Note that there are three actions. Hence, the input file is read three times. MapReduce and Spark (DBD_Exam20200720Sol.zip) |
Exam June 27, 2020 (pdf) | Question 1: (b), Question 2: (a) MapReduce and Spark (DBD_Exam20200627Sol.zip) |
More examples of multiple choice questions (pdf) | Question 1: (c) Question 2: (d) Question 3: (d) Question 4: (d) Question 5: (b) Question 6: (d) |
GraphFrame – Examples of multiple choice questions (pdf) | Question 1: (d) Question 2: (c) |
Additional material
Slides and screencasts about Java (kindly provided by Prof. Torchiano) (link)
Focus on the following subset of slides/lectures (for students who have never used Java):
— OO Paradigm and UML (The UML part is not mandatory)
— The Java Environment
— Java Basic Features
— Java Inheritance