DataBase and Data Mining Group

Distributed architectures for big data processing and analytics (2020/2021)

Table of content

General information
Exam rules
Slides
Exercises
Practices
Exam Examples
Additional material

Pay attention that this page is the web page for to the academic year 2020/2021

General information

ECTS: 8
Professor: Paolo Garza
Teaching assistant: Luca Colomba

Exam rules

Exam rules Academic Year 2020-2021 (exam rules)

Slides

Introduction to the course content and exam rules (pdf)
Introduction to Big Data (pdf)
Big Data Architectures (pdf)
Hadoop and MapReduce
- Introduction to Apache Hadoop and the MapReduce programming paradigm (pdf)
  - Interaction with HDFS and Hadoop by means of the command line (pdf)
- Hadoop implementation of MapReduce (pdf)
  - Source code of the Word Count Ecplise project (WordCount.zip) – Use the import maven project option to import it in Eclipse
  - PDF version of the code (i.e., PDF version of the java files) (WordCountPDF.zip)
  - BigData@Polito environment + Jupyter – How to submit MapReduce jobs on BigData@Polito (pdf)
- MapReduce – Design patterns – Part 1 (pdf)
- MapReduce and Hadoop – Advanced Topics: Multiple inputs, Multiple outputs, Distributed cache (pdf)
- MapReduce – Design patterns – Part 2 (pdf)
- MapReduce – Relational Algebra/SQL operators (pdf)
Spark
- Introduction to Apache Spark (pdf)
  - How to submit Spark applications (pdf)
  - How to use Jupyter notebooks for your Spark applications (pdf)
    - A useful online tutorial for those who want to install and run Spark locally on their PCs (tested for Linux) – How to use PySpark on your computer” by Favio Vázquez (link)
- RDD-based programs
  - RDDs: creation, basic transformations and actions (pdf)
  - Key-value RDDs: transformations and actions on key-value RDDs (pdf)
  - DoubleRDDs (pdf)
  - Advanced Topics: Cache, accumulators, broadcast variables (pdf)
  - Advanced Topics – Part II: Custom partitioners, broadcast join (pdf)
    - RDD partition examples (RDDPartitionsExamples.zip)
    - PageRank example (RDDPageRank.zip)
      Introduction to PageRank (pdf) – Uploaded on April 19, 2021
- Spark SQL and DataFrames
  - Spark SQL (pdf)
    - Simple examples – Jupyter notebook (SparkSQLSimpleExamples.zip)
    - Spark SQL join examples – Jupyter notebook (ExamplesSparkSQLJoins.zip)
  - Spark SQL – Part II (pdf)

- Data mining and Machine learning algorithms with Spark
  - MLlib
    - Introduction and Preprocessing (pdf)
    - Classification (pdf)
      - Classification examples – Jupyter notebooks and sample data (ExampleClassificationMLlib.zip)
    - Clustering (pdf)
      - Clustering example – Jupyter notebook and sample data (ExampleClusteringMLlib.zip)
    - Regression (pdf)
      - Regression example – Jupyter notebook and sample data (ExampleRegressionMLlib.zip)
    - Itemset and Association rule mining (pdf)
      - Itemset and Association rule mining example – Jupyter notebook and sample data (ExampleItemsetMLlib.zip)
- GraphX/GraphFrames
  - Introduction to GraphX and GraphFrames (pdf)
  - Graph Algorithms with GraphFrames (pdf)
    - Simple example – Jupyter notebook (GraphFrameExamples.zip)
      - Select kernel GraphFrames (Yarn) to run it on jupyter.polito.it
      - Run “pyspark –packages graphframes:graphframes:0.8.1-spark3.0-s_2.12 –repositories https://repos.spark-packages.org” to run it locally on your PC
        
        Use package graphframes:graphframes:0.8.0-spark2.4-s_2.11 if you locally installed Spark 2 instead of Spark 3
Streaming data analytics
- Spark Streaming
  - Spark Streaming (DStreams) (pdf)
    - Simple examples – Jupyter notebooks (SparkSteamingExamples.zip)
  - Structured Streaming (pdf)
    - Simple examples – Jupyter notebooks (SparkStructutedStreamingExamples.zip)
- Introduction to other big stream processing frameworks: Apache Storm, Apache Flink, .. (pdf)

Exercises

MapReduce
- MapReduce exercises (pdf)
  - Solutions of Exercises 1-12 (Solutions1_12.zip)
  - Solutions of Exercises 13-22 (Solutions13_22.zip)
  - Solutions of Exercises 23-29 (Solutions23_29.zip)
- Basic project
  - Linux and macOS
    - Basic Eclipse project for MapReduce applications (based on maven) (MapReduceBasicProject.zip)
  - Windows
    - Setup instructions (ConfigureWindowsEnviroment.pdf)
      - You must install also JDK 1.8 and select it for the imported project inside Eclipse. If you already installed the JDK environment but the version is greater than JDK 1.8 you must install also JDK 1.8.
    - Winutils executable (winutils.zip)
    - Basic Eclipse project for MapReduce applications (based on maven) (MapReduceBasicProjectWindows.zip)
Spark
- Spark RDD-, DataFrame-based exercises (pdf)
  - Example data – One folder with (few) data for each exercise (ExSparkData.zip)
  - Solutions of Exercises 30-36 – Jupyter notebooks (SparkNotebooksSol30_36.zip)
  - Solutions of Exercises 37-42 – Jupyter notebooks (SparkNotebooksSol37_42.zip)
    - Exercises 37-38 – Spark SQL-based solutions – Jupyter notebooks (SparkNotebooksSol37_38DataframeSQL.zip)
  - Solutions of Exercises 43-46 – Jupyter notebooks (SparkNotebooksSol43_46.zip)
- Spark SQL exercises (pdf)
  - Example data – One folder with (few) data for each exercise (ExSparkSQLData.zip)
  - Solutions of Exercises 47-50 – Jupyter notebooks (SparkNotebooksSol47_50.zip)
- Spark MLlib exercises (pdf)
  - Example data – One folder with (few) data for each exercise (ExampleMLlibData.zip)
  - Solutions of Exercise 51 (SparkNotebooksSol51.zip)
- GraphFrame exercises (pdf)
  - Example data – One folder with (few) data for each exercise (ExampleGraphFrameData.zip)
  - Solutions of Exercises 52-57 – Jupyter notebooks (SparkNotebooksSol52_57.zip)
- Spark streaming exercises (pdf)
  - Example data – One folder with (few) data for each exercise (ExampleSparkStreamingData.zip)
  - Solutions of Exercises 58-65 – Jupyter notebooks (SparkNotebooksSol58_65.zip)
- Spark structured streaming and MLlib exercise (pdf)
  - Example data – One folder with (few) data for each exercise (ExampleSparkStructuredMLlibData.zip)
  - Solution of Exercise 66 – Jupyter notebooks (SparkNotebooksSol66.zip)

Practices

No lab activities during the first week
TEAM 1: Students from A to L – Friday from 2.30 pm to 4 pm
TEAM 2: Students from M to Z – Friday from 4 pm to 5.30 pm

Lab1: Hadoop and MapReduce (Friday, March 12)
- Problem specification (pdf)
- How to import and run locally on your PC a MapReduce program by using Eclipse + maven (01_ImportProject_LocalRun.mp4)
- How to create a jar file and execute your application on the remote cluster BigData@Polito (02_Jar_ClusterExecution.mp4)
- Basic project and small example data set
  - Linux and macOS (Lab1.zip)
  - Windows (Lab1Windows.zip)
- Bonus task – Skeleton Eclipse project Hadoop – MapReduce
  - Linux and macOS (Lab1Bonus_Skeleton.zip)
  - Windows (Lab1WindowsBonus_Skeleton.zip)
- Solution
  - Lab1Bonus_Sol2021.zip
Lab2: Frequently bought/reviewed together application with Hadoop MapReduce (Friday, March 19)
- Problem specification (pdf)
- Skeleton Eclipse project Hadoop – MapReduce
  - Linux and macOS (Lab2_Skeleton2021.zip)
  - Windows (Lab2Windows_Skeleton2210.zip)
- Sample file (AmazonTransposedDataset_Sample.txt)
- Solution
  - Lab2_Sol2021.zip – Three alternative solutions are provided (the solutions are characterized by a different efficiency)
  - Comments on the three uploaded solutions (slides)
Lab3: Normalized ratings for product recommendations with Hadoop MapReduce (Friday, March 26)
- Problem specification (pdf)
- Sample dataset (ReviewsSample.csv)
- Skeleton Eclipse project Hadoop – MapReduce
  - Linux and macOS (Lab3_Skeleton2021.zip)
  - Windows (Lab3Windows_Skeleton2021.zip)
- Solution
  - Lab3_Sol2021zip
Lab4: Filter data and compute basic statistics with Apache Spark (Friday, April 9)
- Problem specification (pdf)
- Sample file (SampleLocalFile.csv)
- Solution
  - Lab4_Sol2021.zip – Jupyter notebook (Lab4_Sol.ipynb) and Python script (Lab4_Sol.py)
Lab5: Frequently bought/reviewed together application with Apache Spark (Friday, April 16)
- Problem specification (pdf)
- Sample dataset (ReviewsSample.csv)
- Solution
  - Lab5_Sol2021.zip – Jupyter notebook (Lab5_DBD2021Sol.ipynb) and Python script (Lab5_DBD2021Sol.py)
Lab6: Bike sharing data analysis (Friday, April 23)
- Problem specification (pdf)
- Sample data (zip)
- Example KML file (zip)
- Another KML visualizer that can be used to visualize on a map the result of your analysis: http://kmlviewer.nsspot.net
- Solution
  - Lab6_Sol2021.zip – Jupyter notebook (Lab6_DBD2021Sol.ipynb) and Python script (Lab6_DBD2021Sol.py)
Lab7: Bike sharing data analysis based on Spark SQL (Friday, April 30 – 14:30-16:00)
- Problem specification (pdf)
- Sample data (zip)
- Solution
  - Lab7_Sol1920.zip – Jupyter notebook and Python script
Lab8: A classification pipeline with MLlib + SparkSQL (Friday, May 7 – 14:30-16:00)
- Problem specification (pdf)
- Template (zip)
- Solution
  - Lab8_Sol2021.zip – Jupyter notebooks
Lab9: GraphFrame (Friday, May 14 – 14:30-16:00)
- Problem specification (pdf)
- Solution
  - Lab9_Sol2021.zip – Jupyter notebook
Lab10: Tweet analysis – Spark streaming (Friday, May 21 – 14:30-16:00)
- Problem specification (pdf)
- Example files – tweets (exampledata_tweets.zip)
- Solution
  - Lab10_Sol1920.zip – Jupyter notebook
Lab11: Classification with MLlib + Spark streaming (Friday, May 28 – 14:30-16:00)
- Problem specification (pdf)
- Template (zip)
- All data – train, test and streaming (all_data.zip)
- Streaming only data (streaming.zip)
- Solution
  - Lab11_Sol2021.zip – Jupyter notebook

Exam Examples

Exam Example #1 (pdf)
- Solution
  - Question 1: (d)
  - Question 2: (c)
  - SolutionExamExample1.zip
Exam Example #2 (pdf)
- Solution
  - Question 1: (d)
  - Question 2: (c)
  - SolutionExamExample2.zip
Exam Example #3 (pdf)
- Solution
  - Question 1: (c)
  - Question 2: (c)
  - SolutionExamExample3.zip
Exam Example #4 (pdf)
- Solution
  - Question 1: (d)
  - Question 2: (c)
  - SolutionExamExample4.zip
Exam Example #5 (pdf)
- Solution
  - Question 1: (b)
  - Question 2: (b)
  - SolutionExamExample5.zip
Exam June 27, 2020 (pdf)
- Solution
  - Question 1: (b)
  - Question 2: (a)
  - MapReduce and Spark (DBD_Exam20200627Sol.zip)
Exam July 20, 2020 (pdf)
- Solution
  - Question 1: (d)
  - Question 2: (b) – Note that there are three actions and hence the input file is read three times.
  - MapReduce and Spark (DBD_Exam20200720Sol.zip)
Exam September 14, 2020 (pdf)
- Solution
  - Question 1: (d)
  - Question 2: (c)
  - MapReduce and Spark (DBD_Exam20200914Sol.zip)
Exam January 22, 2021 (pdf)
- Solution
  - Question 1: (c)
  - Question 2: (c)
  - MapReduce and Spark (DBD_Exam20210122Sol.zip)
Some more examples of multiple choice questions (pdf)
- Solution
  - Question 1: (c)
  - Question 2: (d)
  - Question 3: (d)
  - Question 4: (d)
  - Question 5: (b)
  - Question 6: (d)
Exam June 21, 2021 (pdf)
- Solution
  - Question 1: (b)
  - Question 2: (a)
  - MapReduce and Spark (DBD_Exam20210621Sol.zip)
Exam July 5, 2021 (pdf)
- Solution
  - Question 1: (c)
  - Question 2: (a)
  - MapReduce and Spark (DBD_Exam20210705Sol.zip)

Additional material

Slides and screencasts about Java (kindly provided by prof. Torchiano) (link)
- Suggested slides/lectures for those students who have never used Java
  - OO Paradigm and UML (The UML part is not needed)
  - The Java Environment
  - Java Basic Features
  - Java Inheritance
Slides about Relational model and SQL language (link)
- Suggested parts
  - Relational data model
  - SQL language:
    - Basics
    - The SELECT statement: basics
    - Nested queries
    - Set operators

Parent page

Teaching

DataBase and Data Mining Group

Distributed architectures for big data processing and analytics (2020/2021)

Table of content

Pay attention that this page is the web page for to the academic year 2020/2021

General information

Exam rules

Slides

Exercises

Practices

Exam Examples

Additional material

Parent page

Menu

© 2024 - DataBase and Data Mining Group