General Information
SSD: ING-INF/05
CFU: 6
Professor: Daniele Apiletti
Teaching Assistant: Simone Monaco
Q&A teaching assistance on Piazza: piazza.com/polito.it/fall2022/01qydov/
Announcements
- 20-09-22: The first lecture is scheduled for September 29, 2022 at 13:00 in Classroom R2
- 26-09-22: No lab activities during the first weeks of the course, Lab will start on October 11th, 2022.
- We are using Piazza for class discussion, we invite all students to join the course Piazza. Piazza is highly catered to getting help fast and efficiently from both classmates and teachers. Rather than emailing questions to the teaching staff, students are invited to post their questions on Piazza.
Teaching Material
- Introduction to the course content and exam rules (slides)
- Introduction to Big Data (slides)
- Big Data Architectures (slides)
- Hadoop and MapReduce
- Introduction to Apache Hadoop and the MapReduce programming paradigm (slides)
- Interaction with HDFS and Hadoop by means of the command line (slides)
- Hadoop implementation of MapReduce (slides)
- BigData@Polito environment + Jupyter – How to submit MapReduce jobs on BigData@Polito (slides)
- MapReduce and Hadoop – Advanced Topics: Multiple inputs, Multiple outputs, Distributed cache (slides)
- MapReduce – Design patterns – Part 1 (slides)
- MapReduce – Design patterns – Part 2 (slides)
- MapReduce – Relational Algebra/SQL operators (slides)
- Introduction to Apache Hadoop and the MapReduce programming paradigm (slides)
- Spark
- Introduction to Apache Spark (slides)
- How to submit Spark applications (slides)
- RDD-based programs RDDs
- Spark SQL, Datasets and DataFrames (slides)
- Spark SQL – Join examples (ExamplesSparkSQLJoins.zip)
- Data Mining
- Recap data mining tasks (slides) – From the “Data Science And Database Technology” course
- Spark MLlib
- Introduction and Classification of structured data (slides)
- Classification of textual data (slides, example code zip)
- Classification and Parameter tuning (slides, example code zip)
- Clustering of structured data (slides, example code zip)
- Itemset and Association rule mining (slides, example code zip)
- Linear regression (slides, example code zip)
- Spark Streaming (slides)
- Examples: Word Count – Streaming versions (zip)
- Introduction to Apache Spark (slides)
Exercises
If you use your PC to write and run your code, import the projects based on Maven (those projects can be run locally).
If you use the PC available in the LAB, import the Eclipse projects with libraries (those projects cannot be run locally but only on the cluster exporting the jar file of the project).
MapReduce
- MapReduce exercises (slides)
- Solutions of Exercises 1-29 (SolutionsExMapReduce.zip)
- Basic project
Linux and MacOS | Windows |
• Basic Eclipse project for MapReduce applications (with libraries) (MapReduceBasicProjectWithLibraries.zip) – Import using Import/General/Existing Projects into Workspace • Basic Eclipse project for MapReduce applications (based on maven) (MapReduceBasicProject.zip) – Import it using Import/Maven/Existing Maven Projects | • Basic Eclipse project for MapReduce applications (with libraries) (MapReduceBasicProjectWithLibraries.zip) – Import using Import/General/Existing Projects into Workspace • Setup instructions for running MapReduce applications locally inside Eclipse (ConfigureWindowsEnviroment.pdf) – You must install also JDK 1.8 and select it for the imported project inside Eclipse. If you already installed the JDK environment but the version is greater than JDK 1.8 you must install also JDK 1.8. – Winutils executable (winutils.zip) – Basic Eclipse project for MapReduce applications (based on maven) (MapReduceBasicProjectWindows.zip) |
Spark
- Spark RDD-, Dataset-, DataFrame-based exercises (slides)
- Example data – One folder with (few) data for each exercise (ExampleDataSpark.zip)
- Solutions of Exercises 30-50 (SolutionsExSpark30-50.zip) – Updated on November 19, 2021 – Added a second possible solution for Exercise #44 (folder Exercise44 _v2)
- Solutions of Exercises from 32 to 38 and 44 based on Spark SQL (SolSparkSQL32-38_44.zip)
- Spark streaming exercises (slides)
- Solutions of Exercises 51-53 (SolutionsSparkStreaming51_53.zip)
Laboratory Material
Student Group | Time | Room |
Team A: Students from A to L | Tue, 16:00-17:30 | Laib1 |
Team B: Students from M to Z | Tue, 17:30-19:00 | Laib1 |
- Lab1: Hadoop and MapReduce
- Problem specification (pdf)
- Basic project and small example data set (Lab1_BigData_with_libraries.zip)
- Basic project based on Maven – Use this version of the project to run the MapReduce application locally on your own PC
- Import it using Import/Maven/Existing Maven Projects
- Linux and macOS (Lab1.zip)
- Windows (Lab1Windows.zip)
- Bigger data set: finefoods_text.txt (zip)
- Import it using Import/Maven/Existing Maven Projects
- Solution Bonus track
- Lab1_SolBonusMvn.zip – The project is based on mvn
- Lab2: Filter with Hadoop MapReduce
- Problem specification (pdf)
- Skeleton Eclipse project Hadoop – MapReduce
- Version with libraries: Lab2_Skeleton_with_libraries.zip
- Maven project, Linux and macOS (Lab2_Skeleton.zip)
- Maven project, Windows (Lab2Windows_Skeleton.zip)
- Outputs of the first lab (OutputFolderLab1.zip) (OutputFolderLab1BonusTrack.zip). You can use them to test your application locally on your own PC if you are using Maven
- Solution: Lab2_Sol.zip – The project is based on mvn
- Solution Bonus track: Lab2_SolBonus.zip – The project is based on mvn
- Lab3: Frequently bought/reviewed together application with Hadoop MapReduce
- Problem specification (pdf)
- Skeleton Eclipse project Hadoop – MapReduce
- Version with libraries (Lab3_Skeleton_with_libraries.zip)
- Maven project, Linux and macOS (Lab3_Skeleton.zip)
- Maven project, Windows (Lab3Windows_Skeleton.zip)
- Sample file (AmazonTransposedDataset_Sample.txt)
- Solution
- Lab3_Sol.zip – The project is based on mvn
- Comments on the three uploaded solutions (slides)
- Lab4: Normalized ratings for product recommendations with Hadoop MapReduce
- Problem specification (pdf)
- Skeleton Eclipse project Hadoop – MapReduce
- Version with libraries (Lab4_Skeleton_with_libraries.zip)
- Maven project, Linux and macOS (Lab4_Skeleton.zip)
- Maven project, Windows (Lab4Windows_Skeleton.zip)
- Sample file (ReviewsSample.csv)
- Solution (zip – The project is based on mvn)
- Lab5: Filter data and compute basic statistics with Apache Spark
- Problem specification (pdf)
- Skeleton Eclipse project – Spark
- Version with libraries (Lab5_Skeleton_with_libraries.zip)
- Maven project Linux, macOS, Windows (Lab5_Skeleton.zip)
- Sample file (SampleLocalFile.csv)
- Solution (zip)
- Lab6: Frequently bought/reviewed together application with Apache Spark
- Problem specification (pdf)
- Skeleton Eclipse project – Spark
- Version with libraries (Lab6_Skeleton_with_libraries.zip)
- Maven project Linux, macOS, Windows (Lab6_Skeleton.zip)
- Sample file (ReviewsSample.csv)
- Expected output – Task 1 (expected output if the input is the HDFS file Reviews.csv) (outputTask1Lab6.zip)
- Solution (zip)
- Lab7: Bike sharing data analysis
- Problem specification (pdf)
- Skeleton Eclipse project – Spark
- Version with libraries (Lab7_Skeleton_with_libraries.zip)
- Maven project Linux, macOS, Windows (Lab7_Skeleton.zip)
- Sample data (sampleData.zip)
- Example KML file (exampleKML.zip)
- Expected output
- Execution on sample data (sampleData/registerSample.csv and sampleData/stations.csv) and minimum criticality threshold = 0.4 (part-00000)
- Execution on complete data (/data/students/bigdata-01QYD/Lab7/register.csv and /data/students/bigdata-01QYD/Lab7/stations.csv) and minimum criticality threshold = 0.6 (part-00000)
- Solution (zip)
- Lab8: Bike sharing data analysis based on Spark SQL
- Problem specification (pdf)
- Skeleton Eclipse project – Spark
- Version with libraries (Lab8_Skeleton_with_libraries.zip)
- Maven project Linux, macOS, Windows (Lab8_Skeleton.zip)
- Sample data (sampleData.zip)
- Solution (zip)
- Lab9: A classification pipeline with MLlib + SparkSQL
- Problem specification (pdf)
- Skeleton Eclipse project – Spark
- Version with libraries (Lab9_Skeleton_with_libraries.zip)
- Maven project (Lab9_Skeleton.zip)
- Sample file with 100 reviews (ReviewsSample.csv)
- Solution
- Lab10: Tweet analysis – Spark streaming
- Problem specification (pdf)
- Skeleton Eclipse project – Spark
- Version with libraries (Lab10_Skeleton_with_libraries.zip)
- Maven project (Lab10_Skeleton.zip)
- Example files – tweets (exampledata_tweets.zip)
- Solution
Exam examples
Pay attention that from the academic year 2022/23 the exam is open book.
Text | Solutions |
Spark Streaming – Examples of multiple choice questions (pdf) | Question 1: (c) Question 2: (d) Question 3: (b) |
Exam June 30, 2017 (pdf) | Question 1: (b) Question 2: (c) Source code/Eclipse projects (zip) |
Exam July 14, 2017 (pdf) | Question 1: (d) Question 2: (c) Source code/Eclipse projects (zip) |
Exam September 14, 2017 (pdf) | Question 1: (a) Question 2: (b) Source code/Eclipse projects (zip) |
Exam June 26, 2018 (pdf) | Question 1: (c) Question 2: (c) Source code/Eclipse projects (zip) |
Exam July 16, 2018 (pdf) | Question 1: (d) Question 2: (a) Source code/Eclipse projects (zip) |
Exam September 3, 2018 (pdf) | Question 1: (d) Question 2: (c) Source code/Eclipse projects (zip) |
Exam February 15, 2019 (pdf) | Question 1: (d) Question 2: (c) Source code/Eclipse projects (zip) |
Exam July 2, 2019 (pdf) | Question 1: (a) Question 2: (b) Source code/Eclipse projects (zip) |
Exam July 18, 2019 (pdf) | Question 1: (b) Question 2: (b) Source code/Eclipse projects (zip) |
Exam July 2, 2020 (pdf) | Question 1: (b) Question 2: (a) Source code/Eclipse projects (zip) |
Exam July 16, 2020 (pdf) | Question 1: (b) Question 2: (b) – Note that there are two actions and hence the input file is read two times. Source code/Eclipse projects (zip) |
Exam September 17, 2020 (pdf) | Question 1: (d) Question 2: (c) Source code/Eclipse projects (zip) |
Exam February 5, 2021 (pdf) | Question 1: (b) Question 2: (c) Source code/Eclipse projects (zip) |
Exam June 30, 2021Exam (pdf) | Question 1: (a) Question 2: (c) Source code/Eclipse projects (zip) |
Exam February 2, 2022 (pdf) | Question 1: (b) Question 2: (d) Source code/Eclipse projects (zip) |
Exam February 21, 2022 (pdf) | Question 1: (d) Question 2: (d) Source code/Eclipse projects (zip) |
Exam July 4, 2022 (pdf) | Question 1: (c) Question 2: (d) Source code/Eclipse projects (zip) |
Exam September 6, 2022 (pdf) | Question 1: (c) Question 2: (d) |
Additional material
- Slides and screencasts about Java (kindly provided by prof. Torchiano) (link)
- Suggested slides/lectures for those students who have never used Java
- OO Paradigm and UML (The UML part is not mandatory)
- The Java Environment
- Java Basic Features
- Java Inheritance
- Suggested slides/lectures for those students who have never used Java