Big Data: Architectures and Data Analytics (2023/2024)
Courses

Big Data: Architectures and Data Analytics (2023/2024)

General Information

SSD: ING-INF/05

CFU: 6

Professor: Daniele Apiletti

Teaching Assistant: Simone Monaco

Q&A teaching assistance on Piazza: piazza.com/polito.it/fall2023/01qydov

Announcements

  • 20-09-23: The first lecture is scheduled for October 4, 2023 at 11:30 in Classroom 3P
  • 27-09-23: No lab activities during the first weeks of the course. The lab will start in the upcoming weeks.
  • 27-09-23: We are using Piazza for class discussion. We invite all students to join the course Piazza. Piazza is highly catered to getting help fast and efficiently from classmates and teachers. Rather than emailing questions to the teaching staff, students are invited to post their questions on Piazza.

Teaching Material

IntroductionHadoop and MapReduceSpark
  • Introduction to the course content and exam rules (slides)
  • Introduction to Big Data (slides)
  • Big Data Architectures (slides)
  • Introduction to Apache Hadoop and the MapReduce programming paradigm (slides)
    • Interaction with HDFS and Hadoop using the command line (slides)
  • Hadoop implementation of MapReduce (slides)
    • BigData@Polito environment + Jupyter – How to submit MapReduce jobs on BigData@Polito (slides)
  • MapReduce and Hadoop – Advanced Topics: Multiple inputs, Multiple outputs, Distributed cache (slides)
  • MapReduce – Design patterns – Part 1 (slides)
  • MapReduce – Design patterns – Part 2 (slides)
  • MapReduce – Relational Algebra/SQL operators (slides)
  • Introduction to Apache Spark (slides)
    • How to submit Spark applications (slides)
  • RDD-based programs RDDs
    • Creation, basic transformations and actions (slides)
    • Key-value pair RDDs: transformations and actions on PairRDDs (slides)
    • DoubleRDDs (slides)
    • Advanced Topics: Cache, accumulators, broadcast variables (slides)
  • Spark SQL, Datasets and DataFrames (slides)
  • Data Mining
    • Recap data mining tasks (slides) – From the “Data Science And Database Technology” course
  • Spark MLlib
  • Spark Streaming (slides)
    • Examples: Word Count – Streaming versions (zip)

Exercises

If you use your PC to write and run your code, import the projects based on Maven (those projects can be run locally).
If you use the PC available in the LAB, import the projects with libraries (those projects cannot be run locally but only on the cluster exporting the jar file of the project). Refer to the Laboratory Material section for more information.

MapReduceSpark

Laboratory Material

The first lab is scheduled for Friday, October 20

Student GroupTimeRoom
Team A: Students from A to LFri, 16:00-17:30Room 3D
Team B: Students from M to ZFri, 17:30-19:00Room 3D

We suggest the use of Visual Studio Code for the lab sessions. For the configuration instruction, refer to this guide.

Windows users only: You have to configure the winutils ( winutils.zip) and set up some environmental variables. Follow this extra guide for the complete configuration.

All the Laboratory materials are available in multiple versions. The version with libraries is the only one you can use on the Lab computers (without running locally). All the other versions are Maven projects, so you can use them locally on your laptop, depending on your OS and on the cluster. The legend for the buttons is the following:

lib: with libraries, mavU: Maven project for Linux/MacOS, mavW: Maven project for Windows (Hadoop projects only)

  • Basic project for MapReduce applications (lib, mavU, mavW)
Lab 1Lab 2Lab 3Lab 4Lab 5Lab 6Lab 7

Lab1: Hadoop and MapReduce

Problem specification ( pdf)
Basic project and small example data set (lib, mavU, mavW)
Bigger data set: finefoods_text.txt ( zip)

Solution (Bonus track mavU)

Lab2: Filter with Hadoop MapReduce

Problem specification ( pdf)
Skeleton Project (lib, mavU, mavW)
Outputs of the first lab ( OutLab1.zip, OutLab1Bonus.zip)

Solution (mavU, bonusmavU)

Lab3: Frequently bought/reviewed together application with Hadoop MapReduce

Problem specification ( pdf)
Skeleton Project (lib, mavU, mavW)
Sample file ( AmazonTransposedDataset_Sample.txt)

Solution (mavU, Comments)

Lab4: Normalized ratings for product recommendations with Hadoop MapReduce

Problem specification ( pdf)
Skeleton Project (lib, mavU, mavW)
Sample file ( ReviewsSample.csv)

Solution (mavU)

Lab5: Filter data and compute basic statistics with Apache Spark
Note: Windows users can download mavU for all Spark projects.

Problem specification ( pdf)
Skeleton Project (lib, mavU)
Sample file ( SampleLocalFile.csv)

Solution (mavU)

Lab6: Frequently bought/reviewed together application with Apache Spark

Problem specification ( pdf)
Skeleton Project (lib, mavU)
Sample data ( ReviewsSample.csv)

Expected output – Task 1 (expected output if the input is the HDFS file Reviews.csv) ( outputTask1Lab6.zip)
Solution (mavU)

Lab7: Bike sharing data analysis

Problem specification ( pdf)
Skeleton Project (lib, mavU)
Sample data ( SampleData.zip)
Example KML file ( exampleKML.zip)

Expected output

  • On sample data (sampleData/{registerSample.csv,stations.csv}),min criticality threshold = 0.4 (part-00000)
  • On complete data (/data/students/bigdata-01QYD/Lab7/{register.csv,stations.csv}), threshold = 0.6 (part-00000)

Solution ( zip)

Additional material

  • Slides and screencasts about Java (kindly provided by prof. Torchiano) (link)
    • Suggested slides/lectures for those students who have never used Java
      • OO Paradigm and UML (The UML part is not mandatory)
      • The Java Environment
      • Java Basic Features
      • Java Inheritance