Distributed architectures for big data processing and analytics (2024/2025)
Courses

Distributed architectures for big data processing and analytics (2024/2025)

General Information

SSD: ING-INF/05

CFU: 8

Professor: Paolo Garza

Teaching Assistants: Simone Monaco and Claudio Savelli

Teaching Material

Introduction
  • Introduction to the course content and exam rules (pdf)
  • Introduction to Big Data (pdf)
  • Big Data Architectures (pdf)
Hadoop and MapReduce
  • Introduction to Apache Hadoop and the MapReduce programming paradigm (pdf)
    • Interaction with HDFS and Hadoop using the command line (pdf)
  • Hadoop implementation of MapReduce (pdf)
    • Source code of the Word Count Ecplise project (WordCount.zip) – Use the import maven project option to import it into Visual Studio Code
    • BigData@Polito environment + Jupyter – How to submit MapReduce jobs on BigData@Polito (pdf)
  • MapReduce – Design patterns – Part 1 (pdf)
  • MapReduce and Hadoop – Advanced Topics: Multiple inputs, Multiple outputs, Distributed cache (pdf)
  • MapReduce – Design patterns – Part 2 (pdf)
  • MapReduce – Relational Algebra/SQL operators (pdf)
Spark
  • Introduction to Apache Spark (pdf)
    • How to submit Spark applications (pdf)
    • How to use Jupyter Notebooks for your Spark applications (pdf)
    • You can install PySpark and JupyterLab using Conda/Miniconda/pip (instructions here)
  • RDD-based programs
    • RDDs: creation, basic transformations and actions (pdf)
    • Key-value RDDs: transformations and actions on key-value RDDs (pdf)
      • Inner join, left outer join, right outer join, full outer join, and “NOT IN” with PairRDDs: Examples – Notebook (JoinsRDD.zip)
    • DoubleRDDs (pdf)
    • Advanced Topics: Cache, accumulators, broadcast variables, custom partitioners, broadcast join (pdf)
  • Spark SQL and DataFrames
  • Data mining and Machine learning algorithms with Spark MLlib
  • GraphX/GraphFrames
    • Introduction to GraphX and GraphFrames (pdf)
    • Graph Algorithms with GraphFrames (pdf)
      • Simple example – Jupyter notebook (GraphFrameExamples.zip)
      • Select kernel GraphFrames (Yarn) to run it on jupyter.polito.it
      • Run “pyspark –packages graphframes:graphframes:0.8.1-spark3.0-s_2.12 –repositories https://repos.spark-packages.org” to run it locally on your PC – Use package graphframes:graphframes:0.8.0-spark2.4-s_2.11 if you locally installed Spark 2 instead of Spark 3
  • Streaming data analytics

Exercises

MapReduce
  • MapReduce Exercises (slides)
  • How to configure Visual Studio Code on your personal laptop: 📘guide.
    • Note that you must also install JDK 1.8 and select it for the imported project inside the IDE. If you have already installed the JDK environment but the version is greater than JDK 1.8, you must also install JDK 1.8.

    • Windows users only: You must configure the winutils (🗃️winutils.zip) and set up some environmental variables. Follow this 📘extra guide for the complete configuration.

    • There are multiple versions of the basic projects. The version with libraries is the only one you can use on the LABINF computers. Use it on your laptop if you are not interested in running the applications locally. All the other versions are Maven projects, so you can use them locally on your personal laptop to write the code and then run it locally inside Visual Studio Code or on the BigData@Polito cluster. The legend is as follows: 📚lib: Project/template with libraries, 🐧mavU: Maven project for Linux/MacOS, 🪟mavW: Maven project for Windows (Hadoop projects only).

    • Basic project for MapReduce applications (📚lib, 🐧mavU, 🪟mavW)
Spark

Laboratory Material

No lab activities during the first week.

Team 1: Students from A to K – Tuesday from 11:30 to 13:00 (First lab activity – March 4, 2025) @ LABINF
Team 2: Students from L to Z – Friday from 11:30 to 13:00 (First lab activity – March 7, 2025) @ LABINF

  • How to configure Visual Studio Code on your personal laptop: 📘guide.
    • Note that you must also install JDK 1.8 and select it for the imported project inside the IDE. If you have already installed the JDK environment but the version is greater than JDK 1.8, you must also install JDK 1.8.
    • Windows users only: You must configure the winutils (🗃️winutils.zip) and set up some environmental variables. Follow this 📘extra guide for the complete configuration.
    • Laboratory materials are available in multiple versions. The version with libraries is the only one you can use on the LABINF computers. Use it on your laptop if you are not interested in running the applications locally. All the other versions are Maven projects, so you can use them locally on your personal laptop to write the code and then run it locally inside Visual Studio Code or on the BigData@Polito cluster. The legend is as follows: 📚lib: Project/template with libraries, 🐧mavU: Maven project for Linux/MacOS, 🪟mavW: Maven project for Windows (Hadoop projects only).
    • Basic project for MapReduce applications (📚lib, 🐧mavU, 🪟mavW)
  • How to configure JDK 1.8 on MAC in case of errors with standard procedure:
  • PySpark Installation Guide: How to run PySpark applications on your PC or Google Colab: You can install PySpark and JupyterLab using Conda/Miniconda/pip (instructions here)

Previous exam examples

Additional material

Slides and screencasts about Java (kindly provided by Prof. Torchiano) (link)
Focus on the following subset of slides/lectures (for students who have never used Java):
— OO Paradigm and UML (The UML part is not mandatory)
— The Java Environment
— Java Basic Features
— Java Inheritance