This web page is related to an old version of the course.
The web page of the current instance of the course is available at link.
General Information
SSD: ING-INF/05
CFU: 8
Professor: Paolo Garza
Teaching Assistant: Luca Colomba
Announcements
03-03-2023: Lab activities
— Team 1: Students from A to J – Tuesday from 11:30 to 13:00 (First lab activity – March 7, 2023) @ LABINF
— Team 2: Students from K to Z – Friday from 11:30 to 13:00 (First lab activity – Match 10, 2023) @ LABINF
18-02-2023: The first lecture is scheduled for February 27, 2023, at 8:30 in Classroom 27
Teaching Material
Introduction
- Introduction to the course content and exam rules (pdf)
- Introduction to Big Data (pdf)
- Big Data Architectures (pdf)
Hadoop and MapReduce
- Introduction to Apache Hadoop and the MapReduce programming paradigm (pdf)
- Interaction with HDFS and Hadoop by means of the command line (pdf)
- Hadoop implementation of MapReduce (pdf)
- Source code of the Word Count Ecplise project (WordCount.zip) – Use the import maven project option to import it in Eclipse
- PDF version of the code (i.e., PDF version of the java files) (WordCountPDF.zip)
- BigData@Polito environment + Jupyter – How to submit MapReduce jobs on BigData@Polito (pdf)
- MapReduce – Design patterns – Part 1 (pdf)
- MapReduce and Hadoop – Advanced Topics: Multiple inputs, Multiple outputs, Distributed cache (pdf)
- MapReduce – Design patterns – Part 2 (pdf)
- MapReduce – Relational Algebra/SQL operators (pdf)
Spark
- Introduction to Apache Spark (pdf)
- How to submit Spark applications (pdf)
- How to use Jupyter Notebooks for your Spark applications (pdf)
- You can install PySpark and JupyterLab using Conda/Miniconda/pip (instructions here)
- RDD-based programs
- RDDs: creation, basic transformations and actions (pdf)
- Key-value RDDs: transformations and actions on key-value RDDs (pdf)
- DoubleRDDs (pdf)
- Advanced Topics: Cache, accumulators, broadcast variables, custom partitioners, broadcast join (pdf)
- RDD partition examples (RDDPartitionsExamples.zip)
- Introduction to PageRank (pdf) – Example: PageRank “naive” implementation (RDDPageRank.zip)
- Spark SQL and DataFrames
- Spark SQL (pdf)
- Simple examples – Jupyter notebook (SparkSQLSimpleExamples.zip)
- Spark SQL join examples – Jupyter notebook (ExamplesSparkSQLJoins.zip)
- Spark SQL (pdf)
- Data mining and Machine learning algorithms with Spark MLlib
- Introduction and Preprocessing (pdf)
- Classification (pdf)
- Classification examples – Jupyter notebooks and sample data (ExampleClassificationMLlib.zip)
- Clustering (pdf)
- Clustering example – Jupyter notebook and sample data (ExampleClusteringMLlib.zip)
- Regression (pdf)
- Regression example – Jupyter notebook and sample data (ExampleRegressionMLlib.zip)
- Itemset and Association rule mining (pdf)
- Itemset and Association rule mining example – Jupyter notebook and sample data (ExampleItemsetMLlib.zip)
- GraphX/GraphFrames
- Introduction to GraphX and GraphFrames (pdf)
- Graph Algorithms with GraphFrames (pdf)
- Simple example – Jupyter notebook (GraphFrameExamples.zip)
- Select kernel GraphFrames (Yarn) to run it on jupyter.polito.it
- Run “pyspark –packages graphframes:graphframes:0.8.1-spark3.0-s_2.12 –repositories https://repos.spark-packages.org” to run it locally on your PC – Use package graphframes:graphframes:0.8.0-spark2.4-s_2.11 if you locally installed Spark 2 instead of Spark 3
- Streaming data analytics
- Spark Streaming Spark Streaming (DStreams) (pdf)
- Simple examples – Jupyter notebooks (SparkSteamingExamples.zip)
- Structured Streaming (pdf)
- Simple examples – Jupyter notebooks (SparkStructutedStreamingExamples.zip)
- Introduction to other big stream processing frameworks: Apache Storm, Apache Flink, .. (pdf)
- Spark Streaming Spark Streaming (DStreams) (pdf)
Exercises
If you use your PC to write and run your code, import the projects based on Maven (those projects can be run locally).
If you use the PC available in the LAB, import the Eclipse projects with libraries (those projects cannot be run locally but only on the cluster exporting the project jar file).
MapReduce
- MapReduce Exercises (slides)
- Solutions of Exercises 1-29 (SolutionsExMapReduce.zip)
- Basic MapReduce project with Linux and macOS
- Basic Eclipse project for MapReduce applications (with libraries) (MapReduceBasicProjectWithLibraries.zip) – Import using Import/General/Existing Projects into Workspace
- Basic Eclipse project for MapReduce applications (based on maven) (MapReduceBasicProject.zip) – Import this project using Import/Maven/Existing Maven Projects
- Basic MapReduce project with Windows
- Basic Eclipse project for MapReduce applications (with libraries) (MapReduceBasicProjectWithLibraries.zip) – Import using Import/General/Existing Projects into Workspace
- Setup instructions for running MapReduce applications locally inside Eclipse (ConfigureWindowsEnviroment.pdf)
- You must also install JDK 1.8 and select it for the imported project inside Eclipse. If you already installed the JDK environment, but the version is newer than JDK 1.8, you must also install JDK 1.8.
- Winutils executable (winutils.zip) – Some of you solved the problems with their Windows version by downloading winutils.exe and hadoop.dll from this alternative source: https://github.com/steveloughran/winutils/tree/master/hadoop-2.7.1/bin
- Basic Eclipse project for MapReduce applications (based on maven) (MapReduceBasicProjectWindows.zip)
Spark
- Spark exercises (pdf)
- Example data – One folder with (few) data for each exercise (ExSparkData.zip)
- RDD-based solutions of Exercises 30-46 – Jupyter notebooks (SparkNotebooksSol30_46.zip)
- Spark SQL exercises (pdf) – Spark SQL Exam exercise example 4 (pdf) – Uploaded on April 29
- Example data – One folder with (few) data for each exercise (ExSparkSQLData.zip)
- Solutions of Exercises 47-50 – Jupyter notebooks (SparkNotebooksSol47_50.zip)
- Solution of Exercise Example 4 (ExerciseExample4Spark.zip) – Uploaded on April 29
- Spark MLlib exercises (pdf)
- Example data – One folder with (few) data for each exercise (ExampleMLlibData.zip)
- Solutions of Exercise 51 (SparkNotebooksSol51.zip)
- GraphFrame exercises (pdf)
- Example data – One folder with (few) data for each exercise (ExampleGraphFrameData.zip)
- Solutions of Exercises 52-57b – Jupyter notebooks (SparkNotebooksSol52_57b.zip)
- Spark streaming exercises (pdf)
- Example data – One folder with (few) data for each exercise (ExampleSparkStreamingData.zip)
- Solutions of Exercises 58-65 – Jupyter notebooks (SparkNotebooksSol58_65.zip)
- Spark structured streaming and MLlib exercise (pdf)
- Example data – One folder with (few) data for each exercise (ExampleSparkStructuredMLlibData.zip)
- Solution of Exercise 66 – Jupyter notebooks (SparkNotebooksSol66.zip)
Additional material
Slides and screencasts about Java (kindly provided by Prof. Torchiano) (link)
Focus on the following subset of slides/lectures (for students who have never used Java):
— OO Paradigm and UML (The UML part is not mandatory)
— The Java Environment
— Java Basic Features
— Java Inheritance