Spark Development Training

COURSE DETAILS & CURRICULUM

Prerequisite: SCALA for Spark

Scala Basics
· What is Scala?
· Why Scala for Spark?
· Intro to Scala REPL : Journey from Java to Scala
· Installing Scala IDE
· Basic Operations
· Defining Functions

Scala Essentials
· Control Structures in Scala
· loops – ForEach, While, Do-While
· Collections – Array, ArrayBuffer, Map, Tuples, Lists
· If Statements
· Conditional Operators
· Enumerations

OOP's and FP
Ø Class and Object Basics
Ø Scala Constructors
Ø Nested Classes
Ø Visibility Rules
Ø Overriding Methods
Ø Functional Programming
Ø Higher Order Functions
Ø Traits
· Interfaces
· Layered Traits

Prerequisite: BigData and Hadoop Framework

Ø Introduction to BigData
Ø Challenges with Bigdata
Ø Batch Vs. Realtime processing
Ø Overview- Hadoop Ecosystem
· HDFS
· Review of MapReduce
· Hive
· Sqoop
· Flume

APACHE SPARK

Introduction to Spark
Ø What is Spark?
Ø Spark Overview
Ø Setting up environment
Ø Using Spark Shell
Ø Spark Web UI

Spark Basics
Ø RDD's
Ø Spark Context
Ø Spark Ecosystem
Ø In-Memory data – Spark

Working with RDD's
Ø Creating, Loading and Saving RDD
Ø Transformations in RDD
Ø Actions in RDD
Ø Key-Value Pair RDD
Ø MapReduce and Pair RDD operations
Ø RDD Partitions

Writing and Deploying Spark Applications
Ø Spark Applications vs. Spark Shell
Ø Creating Spark Context
Ø Building a Spark Application
Ø Running a Spark Application
Ø Spark and Hadoop Integration-HDFS
Ø Handling Sequence Files

Spark RDD
ØRDD Lineage
Ø RDD Persistence Overview
Ø Distributed Persistence

Spark Streaming
Ø Spark Streaming Architecture
Ø First Spark Streaming Programming
Ø Transformations in Spark Streaming

Spark MLlib
1. What is Machine Learning?
2. ML library for Spark
3. Algorithms
· Statistics
· Classification
· Regression
· Clustering
· Collaborative Filtering

Spark SQL
Ø Overview on Hive
Ø Spark SQL Architecture
ØSQLContext in Spark SQL
Ø Working with DataFrames
Ø Example for Spark SQL
Ø Integrating Hive and Spark SQL
ØDataFrames and RDD's
Ø Knowing JSON and Parquet File Formats
Ø Loading of data
Ø Comparing Spark SQL,Impala and Hive-on-Spark

GraphX
Ø Overview of GraphX
Ø Data Visualisation in Spark
Ø Common Spark use-cases

Performance Tuning
Ø Shared Variables: Broadcast Variables
Ø Shared Variables: Accumulators
Ø Common Performance Issues
Ø Performance tuning tips

Course Deliverables
Ø Workshop style coaching
Ø Interactive approach
Ø Course material
Ø POC Implementation
Ø Hands on practice exercises for each topic
Ø Quiz at the end of each major topic
Ø Tips and techniques on Cloudera Certification Examination
Ø Linux concepts and basic commands
Ø On Demand Services
Ø Mock interviews for each individual will be conducted on need basis
Ø SQL basics on need basis
Ø Resume preparation and guidance
Ø Interview questions