General Information
SSD: ING-INF/05
CFU: 6
Professor: Paolo Garza
Teaching Assistant: Luca Colomba
Announcements
- 26-09-23: The first lecture is scheduled for October 2, 2023, at 16:00 in Classroom 4P
- 26-09-23: No lab activities during the first week of the course
Teaching Material
INTRODUCTION
- Introduction to the course content and exam rules (slides)
- Introduction to Big Data (slides)
- Big Data Architectures (slides)
HADOOP AND MAPREDUCE
- Introduction to Apache Hadoop and the MapReduce programming paradigm (slides)
- Interaction with HDFS and Hadoop by means of the command line (slides)
- Hadoop implementation of MapReduce (slides)
- BigData@Polito environment + Jupyter – How to submit MapReduce jobs on BigData@Polito (slides)
- MapReduce and Hadoop – Advanced Topics: Multiple inputs, Multiple outputs, Distributed cache (slides)
- MapReduce – Design patterns – Part 1 (slides)
- MapReduce – Design patterns – Part 2 (slides)
- MapReduce – Relational Algebra/SQL operators (slides)
SPARK
- Introduction to Apache Spark (slides)
- How to submit Spark applications (slides)
- RDD-based programs RDDs
- Spark SQL, Datasets, and DataFrames (slides)
- Spark SQL – Join examples (ExamplesSparkSQLJoins.zip)
- Data Mining
- Recap data mining tasks (slides) – From the “Data Science And Database Technology” course
- Spark MLlib
- Introduction and Classification of structured data (slides)
- Classification of textual data (slides)
- Textual data classification example code (zip)
- Classification and Parameter tuning (slides)
- Parameter tuning example code (zip)
- Clustering of structured data (slides)
- Clustering example code (zip)
- Itemset and Association rule mining (slides)
- Itemset and Association rule mining example code (zip)
- Linear regression (slides)
- Linear regression example code (zip)
- Spark Streaming (slides)
- Examples: Word Count – Streaming versions (zip)
Exercises
MAP REDUCE
- MapReduce exercises (slides)
- Solutions of Exercises 1-29 (SolutionsExMapReduce.zip)
- How to Write and Compile your Java Application using VSCode (pdf)
- Linux or Mac: Basic project for MapReduce applications (based on maven) (MapReduceBasicProject.zip)
- Windows: Basic project for MapReduce applications (based on maven) (MapReduceBasicProjectWindows.zip)
- How to configure the Windows environment to run MapReduce applications locally on your PC(ConfigureWindowsEnviroment.pdf)
- You must also install JDK 1.8 and select it for the imported project inside the IDE. If you already installed the JDK environment but the version is greater than JDK 1.8 you must also install JDK 1.8.
- Winutils executable (winutils.zip)
- If you use your PC to write and run your code locally, use the projects based on Maven (those projects can be run locally).
- If you use the PC available in the LAB, import the projects with libraries as reported in the first lab (those projects cannot be run locally but only on the cluster exporting the project jar file).
SPARK
- Spark RDD-, Dataset-, DataFrame-based exercises (slides)
- Example data – One folder with (few) data for each exercise (ExampleDataSpark.zip)
- Solutions of Exercises 30-50 (SolutionsExSpark30-50.zip) – Updated December 6, 2023 – A new alternative solution for Exercise 44, based on left outer join, has been uploaded (Exercise44v2).
- Solutions of Exercises from 32 to 38 and 44 based on Spark SQL (SolSparkSQL32-38_44.zip) – Updated December 6, 2023 – One more alternative solution for Exercise 44 has been uploaded.
- Spark MLlib (slides)
- Solution (MLlibExampleExercise.zip)
- Spark streaming exercises (slides)
- Solutions of Exercises 51-53 (SolutionsSparkStreaming51_53.zip)
Laboratory Material
Additional material
- Slides and screencasts about Java (kindly provided by Prof. Torchiano) (link)
- Suggested slides/lectures for those students who have never used Java
- OO Paradigm and UML (The UML part is not mandatory)
- The Java Environment
- Java Basic Features
- Java Inheritance