THIS IS THE WEB PAGE OF THE FORMER YEAR
General Information
SSD: ING-INF/05
CFU: 8
Professor: Paolo Garza
Teaching Assistant: Luca Colomba
Announcements
- 04-03- 22: Lab activities
- 21-02-22: The first lecture is scheduled for February 28, 2022, at 8:30 in Classroom R2
Teaching Material
- Introduction to the course content and exam rules (pdf)
- Introduction to Big Data (pdf)
- Big Data Architectures (pdf)
- Hadoop and MapReduce
- Introduction to Apache Hadoop and the MapReduce programming paradigm (pdf)
- Interaction with HDFS and Hadoop by means of the command line (pdf)
- Hadoop implementation of MapReduce (pdf)
- Source code of the Word Count Ecplise project (WordCount.zip) – Use the import maven project option to import it in Eclipse
- PDF version of the code (i.e., PDF version of the java files) (WordCountPDF.zip)
- BigData@Polito environment + Jupyter – How to submit MapReduce jobs on BigData@Polito (pdf)
- MapReduce – Design patterns – Part 1 (pdf)
- MapReduce and Hadoop – Advanced Topics: Multiple inputs, Multiple outputs, Distributed cache (pdf)
- MapReduce – Design patterns – Part 2 (pdf)
- MapReduce – Relational Algebra/SQL operators (pdf)
- Introduction to Apache Hadoop and the MapReduce programming paradigm (pdf)
- Spark
- Introduction to Apache Spark (pdf)
- How to submit Spark applications (pdf)
- How to use Jupyter notebooks for your Spark applications (pdf)
- A useful online tutorial for those who want to install and run Spark locally on their PCs (tested for Linux)
- How to use PySpark on your computer” by Favio Vázquez (link)
- Download Spark from https://spark.apache.org/
- A useful online tutorial for those who want to install and run Spark locally on their PCs (tested for Linux)
- RDD-based programs
- RDDs: creation, basic transformations and actions (pdf)
- Key-value RDDs: transformations and actions on key-value RDDs (pdf)
- DoubleRDDs (pdf)
- Advanced Topics: Cache, accumulators, broadcast variables, custom partitioners, broadcast join (pdf)
- RDD partition examples (RDDPartitionsExamples.zip)
- Example: PageRank “naive” implementation (RDDPageRank.zip)
- Introduction to PageRank (pdf)
- Spark SQL and DataFrames
- Spark SQL (pdf)
- Simple examples – Jupyter notebook (SparkSQLSimpleExamples.zip)
- Spark SQL join examples – Jupyter notebook (ExamplesSparkSQLJoins.zip)
- Spark SQL (pdf)
- Data mining and Machine learning algorithms with Spark MLlib
- Introduction and Preprocessing (pdf)
- Classification (pdf)
- Classification examples – Jupyter notebooks and sample data (ExampleClassificationMLlib.zip)
- Clustering (pdf)
- Clustering example – Jupyter notebook and sample data (ExampleClusteringMLlib.zip)
- Regression (pdf)
- Regression example – Jupyter notebook and sample data (ExampleRegressionMLlib.zip)
- Itemset and Association rule mining (pdf)
- Itemset and Association rule mining example – Jupyter notebook and sample data (ExampleItemsetMLlib.zip)
- GraphX/GraphFrames
- Introduction to GraphX and GraphFrames (pdf)
- Graph Algorithms with GraphFrames (pdf)
- Simple example – Jupyter notebook (GraphFrameExamples.zip)
- Select kernel GraphFrames (Yarn) to run it on jupyter.polito.it
- Run “pyspark –packages graphframes:graphframes:0.8.1-spark3.0-s_2.12 –repositories https://repos.spark-packages.org” to run it locally on your PC
- Use package graphframes:graphframes:0.8.0-spark2.4-s_2.11 if you locally installed Spark 2 instead of Spark 3
- Introduction to Apache Spark (pdf)
- Streaming data analytics
- Spark Streaming Spark Streaming (DStreams) (pdf)
- Simple examples – Jupyter notebooks (SparkSteamingExamples.zip)
- Structured Streaming (pdf)
- Simple examples – Jupyter notebooks (SparkStructutedStreamingExamples.zip)
- Introduction to other big stream processing frameworks: Apache Storm, Apache Flink, .. (pdf)
- Spark Streaming Spark Streaming (DStreams) (pdf)
Exercise
If you use your PC to write and run your code, import the projects based on Maven (those projects can be run locally).
If you use the PC available in the LAB, import the Eclipse projects with libraries (those projects cannot be run locally but only on the cluster exporting the jar file of the project).
- MapReduce exercises (slides)
- Solutions of Exercises 1-29 (SolutionsExMapReduce.zip)
- Basic MapReduce project
- Linux and MacOs
- Basic Eclipse project for MapReduce applications (with libraries) (MapReduceBasicProjectWithLibraries.zip) – Import using Import/General/Existing Projects into Workspace
- Basic Eclipse project for MapReduce applications (based on maven) (MapReduceBasicProject.zip) – Import it using Import/Maven/Existing Maven Projects
- Windows
- Basic Eclipse project for MapReduce applications (with libraries) (MapReduceBasicProjectWithLibraries.zip) – Import using Import/General/Existing Projects into Workspace
- Setup instructions for running MapReduce applications locally inside Eclipse (ConfigureWindowsEnviroment.pdf)
- You must install also JDK 1.8 and select it for the imported project inside Eclipse. If you already installed the JDK environment but the version is greater than JDK 1.8 you must install also JDK 1.8.
- Winutils executable (winutils.zip)
- Basic Eclipse project for MapReduce applications (based on maven) (MapReduceBasicProjectWindows.zip)
- Linux and MacOs
- Spark
- Spark exercises (pdf)
- Example data – One folder with (few) data for each exercise (ExSparkData.zip)
- RDD-based solutions of Exercises 30-46 – Jupyter notebooks (SparkNotebooksSol30_46.zip)
- Spark SQL exercises (pdf)
- Example data – One folder with (few) data for each exercise (ExSparkSQLData.zip)
- Solutions of Exercises 47-50 – Jupyter notebooks (SparkNotebooksSol47_50.zip)
- Spark MLlib exercises (pdf)
- Example data – One folder with (few) data for each exercise (ExampleMLlibData.zip)
- Solutions of Exercise 51 (SparkNotebooksSol51.zip)
- GraphFrame exercises (pdf)
- Example data – One folder with (few) data for each exercise (ExampleGraphFrameData.zip)
- Solutions of Exercises 52-57b – Jupyter notebooks (SparkNotebooksSol52_57b.zip)
- Spark streaming exercises (pdf)
- Example data – One folder with (few) data for each exercise (ExampleSparkStreamingData.zip)
- Solutions of Exercises 58-65 – Jupyter notebooks (SparkNotebooksSol58_65.zip)
- Spark structured streaming and MLlib exercise (pdf)
- Example data – One folder with (few) data for each exercise (ExampleSparkStructuredMLlibData.zip)
- Solution of Exercise 66 – Jupyter notebooks (SparkNotebooksSol66.zip)
- Spark exercises (pdf)
Laboratory Material
No lab activities during the first week of the course
Team 1: Students from A to G – Tuesday from 11:30 to 13:00 (First lab activity – March 8, 2022) – LABINF
Team 2: Students from H to Z – Friday from 11:30 to 13:00 (First lab activity – March 11, 2022) – LABINF
- Lab1: Hadoop and MapReduce
- Problem specification (pdf)
- Basic project and small example data set (Lab1_DBD_with_libraries.zip)
- Basic project based on Maven – Use this version of the project to run the MapReduce application locally on your own PC (DO NOT USE IT ON THE LABINF PCs)
- Import it using Import/Maven/Existing Maven Projects
- Linux and macOS (Lab1_DBD_mvn.zip)
- Windows (Lab1_DBD_Windows_mvn.zip)
- Import it using Import/Maven/Existing Maven Projects
- Bigger data set: finefoods_text.txt (zip)
- You can use it to test your application locally on your own PC if you are using Maven
- Solution Bonus track
- Lab1_SolBonusMvn.zip – The project is based on mvn
- Lab2: Filter with Hadoop MapReduce
- Problem specification (pdf)
- Skeleton Eclipse project Hadoop – MapReduce (Lab2_DBD_with_libraries.zip)
- Basic project based on Maven – Use this version of the project to run the MapReduce application locally on your own PC (DO NOT USE IT ON THE LABINF PCs)
- Import it using Import/Maven/Existing Maven Projects
- Linux and macOS (Lab2_DBD_mvn.zip)
- Windows (Lab2_DBD_Windows_mvn.zip)
- Import it using Import/Maven/Existing Maven Projects
- Outputs of the first lab
- OutputFolderLab1.zip
- OutputFolderLab1BonusTrack.zip
- You can use them to test your application locally on your own PC if you are using Maven
- Solution
- Lab2_DBD_Sol.zip – This project is based on mvn
- Solution Bonus track
- Lab2_SolBonus.zip – This project is based on mvn
- Lab3: Frequently bought/reviewed together application with Hadoop MapReduce
- Problem specification (pdf)
- Skeleton Eclipse project Hadoop – MapReduce (Lab3_DBD_with_libraries.zip)
- Basic project based on Maven – Use this version of the project to run the MapReduce application locally on your own PC (DO NOT USE IT ON THE LABINF PCs)
- Import it using Import/Maven/Existing Maven Projects
- Linux and macOS (Lab3_DBD_mvn.zip)
- Windows (Lab3_DBD_Windows_mvn.zip)
- Import it using Import/Maven/Existing Maven Projects
- Sample file (AmazonTransposedDataset_Sample.txt)
- You can use them to test your application locally on your own PC if you are using Maven
- Solution
- Lab3_DBD_Sol.zip – The project is based on mvn
- Comments on the three uploaded solutions (slides)
- Lab4: Normalized ratings for product recommendations with Hadoop MapReduce
- Problem specification (pdf)
- Skeleton Eclipse project Hadoop – MapReduce (Lab4_DBD_with_libraries.zip)
- Basic project based on Maven – Use this version of the project to run the MapReduce application locally on your own PC (DO NOT USE IT ON THE LABINF PCs)
- Import it using Import/Maven/Existing Maven Projects
- Linux and macOS (Lab4_DBD_mvn.zip)
- Windows (Lab4_DBD_Windows_mvn.zip)
- Import it using Import/Maven/Existing Maven Projects
- Sample file (ReviewsSample.csv)
- Solution
- Lab4_DBD_Sol.zip – This project is based on mvn
- Lab5: Filter data and compute basic statistics with Apache Spark
- Problem specification (pdf)
- Sample file (SampleLocalFile.csv)
- Solution
- Lab5_DBD_Sol.zip – Jupyter notebook (Lab5_Sol.ipynb) and Python script (Lab5_Sol.py)
- Lab6: Frequently bought/reviewed together application with Apache Spark
- Problem specification (pdf)
- Sample dataset (ReviewsSample.csv)
- Solution
- Lab6_DBD_Sol.zip – Jupyter notebook (Lab6_Sol.ipynb) and Python script (Lab6_Sol.py)
- Lab7: Bike sharing data analysis
- Problem specification (pdf)
- Sample data (zip)
- Example KML file (zip)
- KML file containing the result of the analysis setting the threshold to 0.6 and running the program on the HDFS files (zip)
- Solution
- Lab7_DBD_Sol.zip – Jupyter notebook (Lab7_Sol.ipynb) and Python script (Lab7_Sol.py)
- Lab8: Bike sharing data analysis based on Spark SQL
- Problem specification (pdf)
- Sample data (zip)
- Solution
- Lab8_DBD_Sol.zip – Jupyter notebooks and Python scripts
- Lab9: A classification pipeline with MLlib + SparkSQL
- Problem specification (pdf)
- Lab9_template.zip (zip)
- Solution
- Lab9_DBD_Sol.zip – Jupyter notebooks
- Lab10: GraphFrame
- Problem specification (pdf)
- Data (Lab10Data.zip)
- Solution
- Lab10_DBD_Sol.zip – Jupyter notebook
- Lab11: Tweet analysis – Spark streaming
- Problem specification (pdf)
- Example files – tweets (Lab11Data.zip)
- Solution
- Lab11_DBD_Sol.zip – Jupyter notebooks
- Lab12: Classification with MLlib + Spark streaming
- Problem specification (pdf)
- Template (Lab12_DBD_templates.zip)
- Streaming only data (Lab12_DBD_streaming_data.zip)
- All data – train, test, and streaming (Lab12_DBD_all_data.zip)
- Solution
- Lab12_DBD_Sol.zip – Jupyter notebooks
Previous exams
- Exam September 1, 2022 (pdf)
- Solution
- Question 1: (b)
- Question 2: (d)
- MapReduce and Spark (DBD_Exam20220901Sol.zip)
- Solution
- Exam July 18, 2022 (pdf)
- Solution
- Question 1: (b)
- Question 2: (b)
- MapReduce and Spark (DBD_Exam20220718Sol.zip)
- Solution
- Exam June 27, 2022 (pdf)
- Solution
- Question 1: (c)
- Question 2: (a)
- MapReduce and Spark (DBD_Exam20220607Sol.zip)
- Solution
- Exam February 10, 2022 (pdf)
- Solution
- Question 1: (a)
- Question 2: (b)
- MapReduce and Spark (DBD_Exam20220210Sol.zip)
- Solution
- Exam September 17, 2021 (pdf)
- Solution
- Question 1: (b)
- Question 2: (a)
- MapReduce and Spark (DBD_Exam20210917.zip)
- Solution
- Exam July 5, 2021 (pdf)
- Solution
- Question 1: (c)
- Question 2: (a)
- MapReduce and Spark (DBD_Exam20210705Sol.zip)
- Solution
- Exam June 21, 2021 (pdf)
- Solution
- Question 1: (b)
- Question 2: (a)
- MapReduce and Spark (DBD_Exam20210621Sol.zip)
- Solution
- Exam January 22, 2021 (pdf)
- Solution
- Question 1: (c)
- Question 2: (c)
- MapReduce and Spark (DBD_Exam20210122Sol.zip)
- Solution
- Exam September 14, 2020 (pdf)
- Solution
- Question 1: (d)
- Question 2: (c)
- MapReduce and Spark (DBD_Exam20200914Sol.zip)
- Solution
- Exam July 20, 2020 (pdf)
- Solution
- Question 1: (d)
- Question 2: (b) – Note that there are three actions and hence the input file is read three times.
- MapReduce and Spark (DBD_Exam20200720Sol.zip)
- Solution
- Exam June 27, 2020 (pdf)
- Solution
- Question 1: (b)
- Question 2: (a)
- MapReduce and Spark (DBD_Exam20200627Sol.zip)
- Solution
Exam examples and multiple choice questions
- Some more examples of multiple choice questions (pdf)
- Solution
- Question 1: (c)
- Question 2: (d)
- Question 3: (d)
- Question 4: (d)
- Question 5: (b)
- Question 6: (d)
- Solution
- GraphFrame – Examples of multiple choice questions (pdf)
- Solution
- Question 1: (d)
- Question 2: (c)
- Solution
- Exam Example #1 (pdf)
- Solution
- Question 1: (d)
- Question 2: (c)
- SolutionExamExample1.zip
- Solution
- Exam Example #2 (pdf)
- Solution
- Question 1: (d)
- Question 2: (c)
- SolutionExamExample2.zip
- Solution
- Exam Example #3 (pdf)
- Solution
- Question 1: (c)
- Question 2: (c)
- SolutionExamExample3.zip
- Solution
- Exam Example #4 (pdf)
- Solution
- Question 1: (d)
- Question 2: (c)
- SolutionExamExample4.zip
- Solution
- Exam Example #5 (pdf)
- Solution
- Question 1: (b)
- Question 2: (b)
- SolutionExamExample5.zip
- Solution
Additional material
- Slides and screencasts about Java (kindly provided by prof. Torchiano) (link)
- Suggested slides/lectures for those students who have never used Java
- OO Paradigm and UML (The UML part is not mandatory)
- The Java Environment
- Java Basic Features
- Java Inheritance
- Suggested slides/lectures for those students who have never used Java