Distributed architectures for big data processing and analytics (2021/2022)
Courses

Distributed architectures for big data processing and analytics (2021/2022)

THIS IS THE WEB PAGE OF THE FORMER YEAR

General Information

SSD: ING-INF/05

CFU: 8

Professor: Paolo Garza

Teaching Assistant: Luca Colomba

Announcements

  • 04-03- 22: Lab activities
    • Team 1: Students from A to G – Tuesday from 11:30 to 13:00 (First lab activity – March 8, 2022) – LABINF
    • Team 2: Students from H to Z – Friday from 11:30 to 13:00 (First lab activity – March 11, 2022) – LABINF
  • 21-02-22: The first lecture is scheduled for February 28, 2022, at 8:30 in Classroom R2

Teaching Material

  • Introduction to the course content and exam rules (pdf)
  • Introduction to Big Data (pdf)
  • Big Data Architectures (pdf)
  • Hadoop and MapReduce
    • Introduction to Apache Hadoop and the MapReduce programming paradigm (pdf)
      • Interaction with HDFS and Hadoop by means of the command line (pdf)
    • Hadoop implementation of MapReduce (pdf)
      • Source code of the Word Count Ecplise project (WordCount.zip) – Use the import maven project option to import it in Eclipse
      • PDF version of the code (i.e., PDF version of the java files) (WordCountPDF.zip)
      • BigData@Polito environment + Jupyter – How to submit MapReduce jobs on BigData@Polito (pdf)
    • MapReduce – Design patterns – Part 1 (pdf)
    • MapReduce and Hadoop – Advanced Topics: Multiple inputs, Multiple outputs, Distributed cache (pdf)
    • MapReduce – Design patterns – Part 2 (pdf)
    • MapReduce – Relational Algebra/SQL operators (pdf)
  • Spark
    • Introduction to Apache Spark (pdf)
      • How to submit Spark applications (pdf)
      • How to use Jupyter notebooks for your Spark applications (pdf)
        • A useful online tutorial for those who want to install and run Spark locally on their PCs (tested for Linux)
    • RDD-based programs
      • RDDs: creation, basic transformations and actions (pdf)
      • Key-value RDDs: transformations and actions on key-value RDDs (pdf)
      • DoubleRDDs (pdf)
      • Advanced Topics: Cache, accumulators, broadcast variables, custom partitioners, broadcast join (pdf)
    • Spark SQL and DataFrames
    • Data mining and Machine learning algorithms with Spark MLlib
    • GraphX/GraphFrames
      • Introduction to GraphX and GraphFrames (pdf)
      • Graph Algorithms with GraphFrames (pdf)
        • Simple example – Jupyter notebook (GraphFrameExamples.zip)
        • Select kernel GraphFrames (Yarn) to run it on jupyter.polito.it
        • Run “pyspark –packages graphframes:graphframes:0.8.1-spark3.0-s_2.12 –repositories https://repos.spark-packages.org” to run it locally on your PC
          • Use package graphframes:graphframes:0.8.0-spark2.4-s_2.11 if you locally installed Spark 2 instead of Spark 3
  • Streaming data analytics

Exercise

If you use your PC to write and run your code, import the projects based on Maven (those projects can be run locally).
If you use the PC available in the LAB, import the Eclipse projects with libraries (those projects cannot be run locally but only on the cluster exporting the jar file of the project).

Laboratory Material

No lab activities during the first week of the course

Team 1: Students from A to G – Tuesday from 11:30 to 13:00 (First lab activity – March 8, 2022) – LABINF

Team 2: Students from H to Z – Friday from 11:30 to 13:00 (First lab activity – March 11, 2022) – LABINF

  • Lab1: Hadoop and MapReduce
    • Problem specification (pdf)
    • Basic project and small example data set (Lab1_DBD_with_libraries.zip)
    • Basic project based on Maven – Use this version of the project to run the MapReduce application locally on your own PC (DO NOT USE IT ON THE LABINF PCs)
    • Bigger data set: finefoods_text.txt (zip)
      • You can use it to test your application locally on your own PC if you are using Maven
    • Solution Bonus track
  • Lab3: Frequently bought/reviewed together application with Hadoop MapReduce
  • Lab4: Normalized ratings for product recommendations with Hadoop MapReduce
  • Lab5: Filter data and compute basic statistics with Apache Spark
  • Lab6: Frequently bought/reviewed together application with Apache Spark
  • Lab7: Bike sharing data analysis
    • Problem specification (pdf)
    • Sample data (zip)
    • Example KML file (zip)
    • KML file containing the result of the analysis setting the threshold to 0.6 and running the program on the HDFS files (zip)
    • Solution
      • Lab7_DBD_Sol.zip – Jupyter notebook (Lab7_Sol.ipynb) and Python script (Lab7_Sol.py)
  • Lab8: Bike sharing data analysis based on Spark SQL
    • Problem specification (pdf)
    • Sample data (zip)
    • Solution
  • Lab9: A classification pipeline with MLlib + SparkSQL
  • Lab10: GraphFrame
  • Lab11: Tweet analysis – Spark streaming
  • Lab12: Classification with MLlib + Spark streaming

Previous exams

Exam examples and multiple choice questions

Additional material

  • Slides and screencasts about Java (kindly provided by prof. Torchiano) (link)
    • Suggested slides/lectures for those students who have never used Java
      • OO Paradigm and UML (The UML part is not mandatory)
      • The Java Environment
      • Java Basic Features
      • Java Inheritance