BigDATA HADOOP Infrastructure & Services: Basic Concept

Hadoop Cluster & Processes

What is Hadoop Cluster?

Hadoop cluster is  the collections of one or more than one Linux Boxes. In a Hadoop cluster there should be a single Master(Linux machine/box) machine and all the rest of machines  are called Slaves(Linux machines/boxes).

Hadoop Modes to start up :

1. Single Mode
2. Pseudo distributed Mode
3. Fully distributed Mode (it should minimum 3 Linux box)

Single Mode actually acts as a single Master box and it generally used for testing purpose(mainly black box testing).

Fully distributed Mode requires minimum 3 machines/boxes. It is actually the Production mode of Hadoop cluster.

But for R&D(Research & Development) purpose Apache Software Org. has introduced another useful mode i.e Pseudo distributed Mode. In this mode we can achieve all the core functionalities of Fully distributed cluster. That means in this mode one single machine acts as Master & as well as Slave recursively. We will configure our own Hadoop cluster in this mode.

Hadoop Processes & Layers :

Hadoop has 5  different processes for different functionalities.

1. NameNode.
2. Secondary NameNode.
3. DataNode.
4. JobTracker.
5. TaskTracker.

Also Hadoop handle all the storage & analytics part in between 2 layers. These are-

1. HDFS Layer

NameNode

Secondary NameNode

DataNode

2. Application / MapReduce Layer

JobTracker

TaskTracker

Note: NameNode is called the single point of failure in Hadoop cluster.Why??? Because NameNode is the highest priority than JobTracker. Simply without data what’s the value of Application.

Hadoop Process orientation in a Cluster:

   Brief functionalities of the Hadoop processes-

1. NameNode(NN)- NN holds the total HDFS(Hadoop Distributed File System) . That means all the HDFS metadata (that send as report by each DataNode) is stored in NameNode(NN).
2. Secondary NameNode(SNN)- This is process is actually kind of a housekeeper of NameNode(NN). All the activities which generated by NameNode, SNN captured those as snapshot.
3. DataNode(DN)- DN is responsible for data writing in HDFS. Also DN periodically sends block report to NameNode(NN)
4. JobTracker(JT)- Each MapReduce job distributed by this process i.e JobTracker. JT distributes the job by parallel to all of the existing TaskTracker’s(TT) with the help of NN.
5. TaskTracker(TT)- This process is actually executing the Job’s which are distributed by JT.

We can’t edit/modify the data in HDFS. The rule is “Write Once Read Many Times”. Yes we can append the new data but can’t edit.