Table of content
General information
- ECTS: 8
- Professor: Paolo Garza
- Teaching assistant: Luca Colomba
Exam rules
Slides
- Introduction to the course content and exam rules (pdf)
- Introduction to Big Data (pdf)
- Big Data Architectures (pdf)
- Hadoop and MapReduce
- Introduction to Apache Hadoop and the MapReduce programming paradigm (pdf)
- Interaction with HDFS and Hadoop by means of the command line (pdf)
- Hadoop implementation of MapReduce (pdf)
- Source code of the Word Count Ecplise project (WordCount.zip) – Use the import maven project option to import it in Eclipse
- PDF version of the code (i.e., PDF version of the java files) (WordCountPDF.zip)
- BigData@Polito environment + Jupyter – How to submit MapReduce jobs on BigData@Polito (pdf)
- MapReduce – Design patterns – Part 1 (pdf)
- MapReduce and Hadoop – Advanced Topics: Multiple inputs, Multiple outputs, Distributed cache (pdf)
- MapReduce – Design patterns – Part 2 (pdf)
- MapReduce – Relational Algebra/SQL operators (pdf)
- Spark
- Introduction to Apache Spark (pdf)
- How to submit Spark applications (pdf)
- How to use Jupyter notebooks for your Spark applications (pdf)
- A useful online tutorial for those who want to install and run Spark locally on their PCs (tested for Linux) – How to use PySpark on your computer” by Favio Vázquez (link)
- RDD-based programs
- RDDs: creation, basic transformations and actions (pdf)
- Key-value RDDs: transformations and actions on key-value RDDs (pdf)
- DoubleRDDs (pdf)
- Advanced Topics: Cache, accumulators, broadcast variables (pdf)
- Advanced Topics – Part II: Custom partitioners, broadcast join (pdf)
- Spark SQL and DataFrames
- Spark SQL (pdf)
- Spark SQL – Part II (pdf)
-
- Data mining and Machine learning algorithms with Spark
- MLlib
- Introduction and Preprocessing (pdf)
- Classification (pdf)
- Clustering (pdf)
- Regression (pdf)
- Itemset and Association rule mining (pdf)
- GraphX/GraphFrames
- Introduction to GraphX and GraphFrames (pdf)
- Graph Algorithms with GraphFrames (pdf)
- Simple example – Jupyter notebook (GraphFrameExamples.zip)
- Select kernel GraphFrames (Yarn) to run it on jupyter.polito.it
- Run “pyspark – -packages graphframes:graphframes:0.8.0-spark2.4-s_2.11” to run it locally on your PC
- Streaming data analytics
- Spark Streaming
- Spark Streaming (DStreams) (pdf)
- Structured Streaming (pdf)
- Introduction to other big stream processing frameworks: Apache Storm, Apache Flink, .. (pdf)
Exercises
- MapReduce
- MapReduce exercises (pdf)
- Basic project
- Linux and macOS
- Windows
- Setup instructions (ConfigureWindowsEnviroment.pdf)
- You must install also JDK 1.8 and select it for the imported project inside Eclipse. If you already installed the JDK environment but the version is greater than JDK 1.8 you must install also JDK 1.8.
- Winutils executable (winutils.zip)
- Basic Eclipse project for MapReduce applications (based on maven) (MapReduceBasicProjectWindows.zip)
- Spark
- Spark RDD-, DataFrame-based exercises (pdf)
- Spark SQL exercises (pdf)
- Spark MLlib exercises (pdf)
- GraphFrame exercises (pdf)
- Spark streaming exercises (pdf)
Practices
- No lab activities during the first week
- TEAM 1: Students from A to L – Friday from 2.30 pm to 4 pm
- TEAM 2: Students from M to Z – Friday from 4 pm to 5.30 pm
- Lab1: Hadoop and MapReduce (Friday, March 12)
- Problem specification (pdf)
- How to import and run locally on your PC a MapReduce program by using Eclipse + maven (01_ImportProject_LocalRun.mp4)
- How to create a jar file and execute your application on the remote cluster BigData@Polito (02_Jar_ClusterExecution.mp4)
- Basic project and small example data set
- Bonus task – Skeleton Eclipse project Hadoop – MapReduce
- Solution
- Lab2: Frequently bought/reviewed together application with Hadoop MapReduce (Friday, March 19)
- Problem specification (pdf)
- Skeleton Eclipse project Hadoop – MapReduce
- Sample file (AmazonTransposedDataset_Sample.txt)
- Solution
- Lab2_Sol2021.zip – Three alternative solutions are provided (the solutions are characterized by a different efficiency)
- Comments on the three uploaded solutions (slides)
- Lab3: Normalized ratings for product recommendations with Hadoop MapReduce (Friday, March 26)
- Problem specification (pdf)
- Sample dataset (ReviewsSample.csv)
- Skeleton Eclipse project Hadoop – MapReduce
- Solution
- Lab4: Filter data and compute basic statistics with Apache Spark (Friday, April 9)
- Lab5: Frequently bought/reviewed together application with Apache Spark (Friday, April 16)
Exam Examples
- Exam Example #1 (pdf)
- Exam Example #2 (pdf)
- Exam Example #3 (pdf)
- Exam Example #4 (pdf)
- Exam Example #5 (pdf)
- Exam June 27, 2020 (pdf)
- Exam July 20, 2020 (pdf)
- Solution
- Question 1: (d)
- Question 2: (b) – Note that there are three actions and hence the input file is read three times.
- Part II: MapReduce and Spark (DBD_Exam20200720Sol.zip)
- Exam September 14, 2020 (pdf)
Additional material
- Slides and screencasts about Java (kindly provided by prof. Torchiano) (link)
- Suggested slides/lectures for those students who have never used Java
- OO Paradigm and UML (The UML part is not needed)
- The Java Environment
- Java Basic Features
- Java Inheritance
- Slides about Relational model and SQL language (link)
- Suggested parts
- Relational data model
- SQL language:
- Basics
- The SELECT statement: basics
- Nested queries
- Set operators
Welcome