General Information
SSD: ING-INF/05 – CFU: 6 – Link to the official Teaching Portal web page
Professor: Daniele Apiletti
Teaching Assistant: Luca Colomba
Q&A teaching assistance on Piazza: Piazza.com/polito.it/fall2024/01qydov
Announcements
- Lab activities will start on Thursday 10/10/2024 at LAIB1T.
- We are using Piazza for class discussion. We invite all students to join the course Piazza. Piazza is highly catered to getting help fast and efficiently from classmates and teachers. Rather than emailing questions to the teaching staff, students are invited to post their questions on Piazza.
Teaching Material
- Introduction to Apache Hadoop and the MapReduce programming paradigm (slides)
- Interaction with HDFS and Hadoop using the command line (slides)
- Hadoop implementation of MapReduce (slides)
- BigData@Polito environment + Jupyter – How to submit MapReduce jobs on BigData@Polito (slides)
- MapReduce and Hadoop – Advanced Topics: Multiple inputs, Multiple outputs, Distributed cache (slides)
- MapReduce – Design patterns – Part 1 (slides)
- MapReduce – Design patterns – Part 2 (slides)
- MapReduce – Relational Algebra/SQL operators (slides)
- Spark material will be published soon.
Exercises
If you use your PC to write and run your code, import the projects based on Maven (those projects can be run locally).
If you use the PC available in the LAB, import the projects with libraries (those projects cannot be run locally but only on the cluster exporting the project’s jar file). Refer to the Laboratory Material section for more information.
- MapReduce exercises (slides)
- Solutions of Exercises 1-29 (🗃️SolutionsExMapReduce.zip)
- Spark RDD-, Dataset-, DataFrame-based exercises (slides)
- Example data – One folder with (few) data for each exercise (🗃️ExampleDataSpark.zip)
- Solutions of Exercises 30-50 (🗃️SolutionsExSpark30-50.zip)
- Solutions of Exercises from 32 to 38 and 44 based on Spark SQL (🗃️SolSparkSQL32-38_44.zip)
- Spark streaming exercises (slides)
- Solutions of Exercises 51-53 (🗃️SolutionsSparkStreaming51_53.zip)
Laboratory Material
Lab activities will start on Thursday 10/10/2024 at LAIB1T.
| Student Group | Time | Room |
| Team A: Students from A to L | Thursday, 08:30-10:00 | LAIB 1T |
| Team B: Students from M to Z | Thursday, 10:00-11:30 | LAIB 1T |
We suggest the use of Visual Studio Code for the lab sessions. For the configuration instructions, refer to this 📘guide.
Windows users only: You must configure the winutils (🗃️winutils.zip) and set up some environmental variables. Follow this 📘extra guide for the complete configuration.
All the Laboratory materials are available in multiple versions. The version with libraries is the only one you can use on the Lab computers (without running locally). All the other versions are Maven projects, so you can use them locally on your laptop, depending on your OS and the cluster. The legend for the buttons is the following:
📚lib: with libraries, 🐧mavU: Maven project for Linux/MacOS, 🪟mavW: Maven project for Windows (Hadoop projects only)
Exam examples
Since academic year 2022/23, the exam is open book.
| Text | Solutions |
| Spark Streaming – Examples of multiple choice questions (pdf) | Question 1: (c) Question 2: (d) Question 3: (b) |
| Exam June 30, 2017 (pdf) | Question 1: (b) Question 2: (c) Source code/Eclipse projects (zip) |
| Exam July 14, 2017 (pdf) | Question 1: (d) Question 2: (c) Source code/Eclipse projects (zip) |
| Exam September 14, 2017 (pdf) | Question 1: (a) Question 2: (b) Source code/Eclipse projects (zip) |
| Exam June 26, 2018 (pdf) | Question 1: (c) Question 2: (c) Source code/Eclipse projects (zip) |
| Exam July 16, 2018 (pdf) | Question 1: (d) Question 2: (a) Source code/Eclipse projects (zip) |
| Exam September 3, 2018 (pdf) | Question 1: (d) Question 2: (c) Source code/Eclipse projects (zip) |
| Exam February 15, 2019 (pdf) | Question 1: (d) Question 2: (c) Source code/Eclipse projects (zip) |
| Exam July 2, 2019 (pdf) | Question 1: (a) Question 2: (b) Source code/Eclipse projects (zip) |
| Exam July 18, 2019 (pdf) | Question 1: (b) Question 2: (b) Source code/Eclipse projects (zip) |
| Exam July 2, 2020 (pdf) | Question 1: (b) Question 2: (a) Source code/Eclipse projects (zip) |
| Exam July 16, 2020 (pdf) | Question 1: (b) Question 2: (b) – Note that there are two actions and hence the input file is read two times. Source code/Eclipse projects (zip) |
| Exam September 17, 2020 (pdf) | Question 1: (d) Question 2: (c) Source code/Eclipse projects (zip) |
| Exam February 5, 2021 (pdf) | Question 1: (b) Question 2: (c) Source code/Eclipse projects (zip) |
| Exam June 30, 2021Exam (pdf) | Question 1: (a) Question 2: (c) Source code/Eclipse projects (zip) |
| Exam February 2, 2022 (pdf) | Question 1: (b) Question 2: (d) Source code/Eclipse projects (zip) |
| Exam February 21, 2022 (pdf) | Question 1: (d) Question 2: (d) Source code/Eclipse projects (zip) |
| Exam July 4, 2022 (pdf) | Question 1: (c) Question 2: (d) Source code/Eclipse projects (zip) |
| Exam September 6, 2022 (pdf) | Question 1: (c) Question 2: (d) |
Additional material
- Slides and screencasts about Java (kindly provided by prof. Torchiano) (link)
- Suggested slides/lectures for those students who have never used Java
- OO Paradigm and UML (The UML part is not mandatory)
- The Java Environment
- Java Basic Features
- Java Inheritance
- Suggested slides/lectures for those students who have never used Java