Big Data: Architectures and Data Analytics (2021/2022)

Big Data: Architectures and Data Analytics (2021/2022)

General Information

SSD: ING-INF/05

CFU: 6

Professor: Paolo Garza

Teaching Assistant: Luca Colomba

Announcements

  • 16-09-21: The first lecture is scheduled for September 27, 2021 at 16:00 in Classroom R1
  • 23-09-21: No lab activities during the first two weeks of the course
  • 27-09-21: I created a page for “01QYDOV – Big data: architectures and data analytics” on Piazza: piazza.com/polito.it/fall2021/01qydov
    Piazza is a Q&A system that can be used to manage questions and answers offline. You can use it, instead of the email, when you have questions that are of interest also for other students (e.g., questions on the proposed solutions, problems with the configuration of the used software, etc.). There are different “channels” (lectures, lab, others) related to different topics. You can publish both public or private questions.
    I will answer periodically to your questions (I will try to answer them daily).
  • 10-10-21: First lab activity this week
    • Monday, October 11, 17:30 – 19:00 – LAIB1 – TEAM 1
    • Wednesday, October 13, 14:30 – 16:00 – LAIB1 – TEAM 2
  • 10-10-21: The lecture scheduled for Tuesday, October 12 at 10:00 will start at 11:30 and will last only 1.5 hours (from 11:30 to 13:00)

Teaching Material

  • Introduction to the course content and exam rules (slides)
  • Introduction to Big Data (slides)
  • Big Data Architectures (slides)
  • Hadoop and MapReduce
    • Introduction to Apache Hadoop and the MapReduce programming paradigm (slides)
      • Interaction with HDFS and Hadoop by means of the command line (slides)
    • Hadoop implementation of MapReduce (slides)
      • BigData@Polito environment + Jupyter – How to submit MapReduce jobs on BigData@Polito (slides)
    • MapReduce and Hadoop – Advanced Topics: Multiple inputs, Multiple outputs, Distributed cache (slides)
    • MapReduce – Design patterns – Part 1 (slides) (slides without black background) (slides)
    • MapReduce – Design patterns – Part 2 (slides)
    • MapReduce – Relational Algebra/SQL operators (slides)
  • Spark
    • Introduction to Apache Spark (slides)
      • How to submit Spark applications (slides)
    • RDD-based programs RDDs
      • Creation, basic transformations and actions (slides)
      • Key-value pair RDDs: transformations and actions on PairRDDs (slides)
      • DoubleRDDs (slides)
      • Advanced Topics: Cache, accumulators, broadcast variables (slides)
    • Spark SQL, Datasets and DataFrames (slides)
    • Data Mining
      • Recap data mining tasks (slides)
    • Spark MLlib
      • Introduction and Classification of structured data (slides)
        • Logistic Regression example code (zip)
        • Decision Trees example code (zip)
        • Decision Trees and Categorical class label example code (zip)
      • Classification of textual data (slides)
        • Textual data classification example code (zip)
      • Classification and Parameter tuning (slides)
        • Parameter tuning example code (zip)
      • Clustering of structured data (slides)
        • Clustering example code (zip)
      • Itemset and Association rule mining (slides)
        • Itemset and Association rule mining example code (zip)
      • Linear regression (slides)
        • Linear regression example code (zip)
    • Spark Streaming (slides)
      • Word Count – Streaming version (zip)
      • Word Count – Streaming version – Read data from HDFS folder (zip)
      • Word Count – Window version (zip)
      • Word Count – Stateful version (zip)
      • Word Count – Output sort by key – Based on the transformPair() transformation (zip)

Exercise

If you use your PC to write and run your code, import the projects based on Maven (those projects can be run locally).
If you use the PC available in the LAB, import the Eclipse projects with libraries (those projects cannot be run locally but only on the cluster exporting the jar file of the project).

  • MapReduce
    • MapReduce exercises (slides)
    • Basic project
      • Linux and MacOs
        • Basic Eclipse project for MapReduce applications (with libraries) (MapReduceBasicProjectWithLibraries.zip) – Import using Import/General/Existing Projects into Workspace
        • Basic Eclipse project for MapReduce applications (based on maven) (MapReduceBasicProject.zip) – Import it using Import/Maven/Existing Maven Projects
      • Windows
        • Basic Eclipse project for MapReduce applications (with libraries) (MapReduceBasicProjectWithLibraries.zip) – Import using Import/General/Existing Projects into Workspace
        • Setup instructions for running MapReduce applications locally inside Eclipse (ConfigureWindowsEnviroment.pdf)
          • You must install also JDK 1.8 and select it for the imported project inside Eclipse. If you already installed the JDK environment  but the version is greater than JDK 1.8 you must install also JDK 1.8.
          • Winutils executable (winutils.zip)
          • Basic Eclipse project for MapReduce applications (based on maven) (MapReduceBasicProjectWindows.zip)
  • Spark
    • Spark RDD-, Dataset-, DataFrame-based exercises (slides)
      • Example data – One folder with (few) data for each exercise (ExSparkData.zip)
      • Solutions of Exercises 30-50 (SolutionsExSpark.zip)
        • Ex. 39 Bis – Comparison between two alternative solutions (slides)
    • Spark streaming exercises (slides)
      • Solutions of Exercises 51-53 (SolutionsSparkStreaming.zip)

Laboratory Material

No lab activities during the first two weeks of the course

Team 1: Students from A to L – Monday from 17:30 to 19:00 – LAIB1

Team 2: Students from M to Z – Wednesday from 14:30 to 16:00 – LAIB1

  • Lab1: Hadoop and MapReduce
    • Problem specification (pdf)
    • Basic project and small example data set (Lab1_BigData_with_libraries.zip)
    • Basic project based on Maven – Use this version of the project to run the MapReduce application locally on your own PC (DO NOT USE IT AT LAIB1)
      • Import it using Import/Maven/Existing Maven Projects
      • Bigger data set: finefoods_text.txt (zip)

Exam examples

Pay attention that from the academic year 2020/21 the exam is closed book

  • Exam ..
    • ..

Additional material

  • Slides and screencasts about Java (kindly provided by prof. Torchiano) (link)
    • Suggested slides/lectures for those students who have never used Java
      • OO Paradigm and UML (The UML part is not mandatory)
      • The Java Environment
      • Java Basic Features
      • Java Inheritance