Big Data: Architectures and Data Analytics (2024/2025)
Courses

Big Data: Architectures and Data Analytics (2024/2025)

General Information

SSD: ING-INF/05 – CFU: 6 – Link to the official Teaching Portal web page

Professor: Daniele Apiletti

Teaching Assistant: Luca Colomba

Q&A teaching assistance on Piazza: Piazza.com/polito.it/fall2024/01qydov

Announcements

  • Lab activities on Thursday 31/10/2024 are cancelled
  • Lab activities will start on Thursday 10/10/2024 at LAIB1T.
  • We are using Piazza for class discussion. We invite all students to join the course Piazza. Piazza is highly catered to getting help fast and efficiently from classmates and teachers. Rather than emailing questions to the teaching staff, students are invited to post their questions on Piazza.

Teaching Material

IntroductionHadoop and MapReduceSpark
  • Introduction to the course content and exam rules (slides)
  • Introduction to Big Data (slides)
  • Big Data Architectures (slides)
  • Introduction to Apache Hadoop and the MapReduce programming paradigm (slides)
    • Interaction with HDFS and Hadoop using the command line (slides)
  • Hadoop implementation of MapReduce (slides)
    • BigData@Polito environment + Jupyter – How to submit MapReduce jobs on BigData@Polito (slides)
  • MapReduce and Hadoop – Advanced Topics: Multiple inputs, Multiple outputs, Distributed cache (slides)
  • MapReduce – Design patterns – Part 1 (slides)
  • MapReduce – Design patterns – Part 2 (slides)
  • MapReduce – Relational Algebra/SQL operators (slides)
  • Introduction to Apache Spark (slides)
    • How to submit Spark applications (slides)
    • How to use Jupyter Notebooks for your Spark applications (pdf)
    • You can install PySpark and JupyterLab using Conda/Miniconda/pip (instructions here)
  • RDD-based programs RDDs
    • Creation, basic transformations, and actions (slides) – Notebook with some examples from the slides (FirstExamplesNotebook.zip)
    • Key-value pair RDDs: transformations and actions on PairRDDs (slides)
      • Inner join, left outer join, right outer join, full outer join, and “NOT IN” with PairRDDs: Examples – Notebook (JoinsRDD.zip)
    • DoubleRDDs (slides)
    • Advanced Topics: Cache, accumulators, broadcast variables (slides)
  • Spark SQL, Datasets, and DataFrames (slides)
  • Spark MLlib
    • Introduction to MLlib (slides)
    • Classification of structured data and textual data (slides)
      • Classification example code (zip)
    • Regression (slides)
      • Linear regression example code (zip)
    • Clustering of structured data (slides)
      • Clustering example code (zip)
    • Itemset and Association rule mining (slides)
      • Itemset and Association rule mining example code (zip)
  • Spark Streaming (slides)

Exercises

If you use your PC to write and run your code, import the projects based on Maven (those projects can be run locally).
If you use the PC available in the LAB, import the projects with libraries (those projects cannot be run locally but only on the cluster exporting the project’s jar file). Refer to the Laboratory Material section for more information.

MapReduceSpark

Laboratory Material

Lab activities will start on Thursday 10/10/2024 at LAIB1T.

Student GroupTimeRoom
Team A: Students from A to LThursday, 08:30-10:00LAIB 1T
Team B: Students from M to ZThursday, 10:00-11:30LAIB 1T

We suggest the use of Visual Studio Code for the lab sessions. For the configuration instructions, refer to this 📘guide.

Windows users only: You must configure the winutils (🗃️winutils.zip) and set up some environmental variables. Follow this 📘extra guide for the complete configuration.

All the Laboratory materials are available in multiple versions. The version with libraries is the only one you can use on the Lab computers (without running locally). All the other versions are Maven projects, so you can use them locally on your laptop, depending on your OS and the cluster. The legend for the buttons is the following:

📚lib: with libraries, 🐧mavU: Maven project for Linux/MacOS, 🪟mavW: Maven project for Windows (Hadoop projects only)

  • Basic project for MapReduce applications (📚lib, 🐧mavU, 🪟mavW)
  • Local PySpark installation guide here
Lab 1Lab 2Lab 3Lab 4Lab 5Lab 6Lab 7Lab 8Lab 9Lab 10

Lab1: Hadoop and MapReduce

Problem specification (📄pdf)
Basic project and small example data set (📚lib, 🐧mavU, 🪟mavW)
Bigger data set: finefoods_text.txt (🗃️zip)

Solution: (Bonus track) 🐧mavU

Lab2: Filter with Hadoop MapReduce and Frequency Count

Problem specification (📄pdf)
Skeleton Project (📚lib, 🐧mavU, 🪟mavW)
Outputs of the first lab (🗃️OutLab1.zip)

Solution (Task 1🐧mavU, Task 2 🐧mavU)

Lab3: Frequently bought/reviewed together application with Hadoop MapReduce

Problem specification (📄pdf)
Skeleton Project (📚lib, 🐧mavU, 🪟mavW)
Sample file (📅AmazonTransposedDataset_Sample.txt)

Solution (🐧mavU, Comments)

Problem specification (📄pdf)
Skeleton Project (📚lib, 🐧mavU, 🪟mavW)
Sample file (📅ReviewsSample.csv)

Solution (🐧mavU)

Lab5: Filter data and compute basic statistics with Apache Spark

Problem specification (📄pdf)
Sample file (📅SampleLocalFile.csv)

Solution (🐧mavU)

Lab6: Frequently bought/reviewed together application with Apache Spark

Problem specification (📄pdf)
Sample data (📅ReviewsSample.csv)

Expected output – Task 1 (expected output if the input is the HDFS file Reviews.csv) (🗃️outputTask1Lab6.zip)

Solution (🐧mavU)

Lab7: Bike sharing data analysis

Problem specification (📄pdf)
Sample data (🗃️SampleData.zip)
Example KML file (🗃️exampleKML.zip)

Expected output:

  • On sample data (sampleData/{registerSample.csv,stations.csv}),min criticality threshold = 0.4 (part-00000)
  • On complete data (/data/students/bigdata-01QYD/Lab7/{register.csv,stations.csv}), threshold = 0.6 (part-00000)

Solution: (🐧mavU)

Lab 8: Bike sharing data analysis based on Spark SQL

Problem specification (📄pdf)
Sample data (🗃️SampleData.zip)

Solution (🐧mavU)

Lab9: A classification pipeline with MLlib + SparkSQL

Problem specification (📄pdf)
Sample data (📅ReviewsSample.csv)
Template (🗃️zip)

Solution (🐧mavU)

Lab10: Tweet analysis – Spark streaming

Problem specification (📄pdf)
Example files – tweets (🗃️exampledata_tweets.zip)

Solution (🐧mavU)

Exam examples

Since academic year 2022/23, the exam is open book.

ExamsSolutions
Spark Streaming – Examples of multiple choice questions (pdf)
– Python-based version – Updated on December 18, 2024		Question 1: (c)
Question 2: (d)
Question 3: (b)
September 12, 2024: pdfQuestion 1: (b)
Question 2: (a)
Source code: zip – Spark – Python-based solution
June 20, 2024: pdfQuestion 1: (b)
Question 2: (a)
Source code: zip – Spark – Python-based solution
February 19, 2024: pdfQuestion 1: (a)
Question 2: (b)
Source code: zip – Spark – Python-based solution
February 5, 2024: pdfQuestion 1: (a)
Question 2: (b)
Source code: zip – Spark – Python-based solution
A solution based on Spark SQL is available for this exam
September 21, 2023: pdfQuestion 1: (a)
Question 2: (b)
June 21, 2023: pdfQuestion 1: (c)
Question 2: (d)
February 15, 2023: pdfQuestion 1: (c)
Question 2: (b)
Source code: zip – Spark – Python-based solution
A solution based on Spark SQL is available for this exam
February 2, 2023: pdfQuestion 1: (d)
Question 2: (d)
Source code: zip – Spark – Python-based solution
A solution based on Spark SQL is available for this exam
September 6, 2022: pdfQuestion 1: (c)
Question 2: (d)
Source code: zip – Spark – Python-based solution
A solution based on Spark SQL is available for this exam
July 4, 2022: pdfQuestion 1: (c)
Question 2: (d)
Source code: zip – Spark – Python-based solution
February 21, 2022: pdfQuestion 1: (d)
Question 2: (b)
Source code: zip – Spark – Python-based solution
A solution based on Spark SQL is available for this exam
February 2, 2022: pdfQuestion 1: (b)
Question 2: (d)
Source code: zip – Spark – Python-based solution
June 30, 2021: pdfQuestion 1: (a)
Question 2: (c)
Source code: zip
February 5, 2021: pdfQuestion 1: (b)
Question 2: (c)
Source code: zip
September 17, 2020: pdfQuestion 1: (d)
Question 2: (c)
Source code: zip
July 16, 2020: pdfQuestion 1: (b)
Question 2: (b) – Note that there are two actions; hence, the input file is read twice.
Source code: zip
July 2, 2020: pdfQuestion 1: (b)
Question 2: (a)
Source code: zip
July 18, 2019: pdfQuestion 1: (b)
Question 2: (b)
Source code: zip
July 2, 2019: pdfQuestion 1: (a)
Question 2: (b)
Source code: zip
February 15, 2019: pdfQuestion 1: (d)
Question 2: (c)
Source code: zip
September 3, 2018: pdfQuestion 1: (d)
Question 2: (c)
Source code: zip
July 16, 2018: pdfQuestion 1: (d)
Question 2: (a)
Source code: zip
June 26, 2018: pdfQuestion 1: (c)
Question 2: (c)
Source code: zip

Additional material

  • Slides and screencasts about Java (kindly provided by prof. Torchiano) (link)
    • Suggested slides/lectures for those students who have never used Java
      • OO Paradigm and UML (The UML part is not mandatory)
      • The Java Environment
      • Java Basic Features
      • Java Inheritance