Introduction to Hadoop

Definition: Hadoop, an Apache open-source framework written in Java, enables the distributed processing of large datasets across computer clusters using simple programming models. Operating in an environment with distributed storage and computation across clusters, Hadoop is designed to scale seamlessly from a single server to thousands of machines, each providing local computation and storage.

Example: For example, consider a dataset of 5 terabytes. If you distribute the processing across a Hadoop cluster of 10,000 servers, each server would need to process approximately 500 megabytes of data. This allows the entire dataset to be processed much faster than traditional sequential processing.

Hadoop has two major layers namely --

• Processing/Computation layer (MapReduce)
• Storage layer (Hadoop Distributed File System).

• MapReduce
• HDFS(Hadoop Distributed File System)
• YARN(Yet Another Resource Negotiator)
• Common Utilities or Hadoop Common

Advantegs of Hadoop

• Scalability: Hadoop is highly scalable, allowing for the easy addition of nodes to a cluster to handle increased data volume and processing requirements.
• Cost-Effective:Hadoop can store and process large volumes of data on commodity hardware, making it a cost-effective solution compared to traditional database systems.
• Parallel Processing: The MapReduce programming model enables parallel processing, distributing data across multiple nodes for faster and more efficient computation.
• Fault Tolerance: Hadoop is designed to be fault-tolerant. If a node fails, data can be easily recovered as it exists in multiple copies across the cluster.
• Flexibility: Hadoop can handle structured and unstructured data, making it suitable for a variety of data types, including text, images, and videos.
• Open Source: Hadoop is an open-source framework, which means it is accessible and customizable. The community support and continuous development contribute to its robustness.
• Ecosystem: The Hadoop ecosystem includes a variety of tools and frameworks (like Hive, Pig, Spark) that extend its functionality for different use cases, such as data warehousing, machine learning, and real-time processing.

Disadvantages of Hadoop

• Complexity: Implementing and managing a Hadoop cluster can be complex. It requires expertise in system administration, configuration, and tuning for optimal performance.
• Programming Model: Developing applications in the MapReduce programming model may be challenging for those unfamiliar with it. However, higher-level abstractions like Apache Spark have addressed some of these concerns.
• Real-time Processing: Hadoop's traditional batch processing model may not be suitable for real-time or low-latency processing requirements. Other tools like Apache Spark or Flink are better suited for these scenarios.
• Hardware Dependency: While Hadoop can run on commodity hardware, optimal performance often requires a dedicated cluster of powerful machines, which may increase infrastructure costs.
• Data Security: Hadoop's security features have evolved, but ensuring robust data security and access control can still be a concern, especially in large and complex deployments.
• Data Locality: While Hadoop aims for data locality (processing data on the node where it's stored), achieving perfect data locality is not always possible, impacting performance.
• Learning Curve: There is a learning curve associated with understanding and effectively using Hadoop and its associated tools, which may require additional training for teams unfamiliar with the ecosystem.

MapReduce

How does MapReduce work?

1. Map Phase: Input data is divided into chunks, and each chunk is processed independently by a set of mappers. Mappers apply a user-defined function (map function) to transform the input data into a set of key-value pairs.
2. Shuffle and Sort Phase: The framework redistributes the output of the mappers based on the keys to group related data together. This phase ensures that all data with the same key is sent to the same reducer.
3. Reduce Phase: Reducers receive the grouped data and apply a user-defined function (reduce function) to produce the final output. The reduce function typically aggregates, filters, or performs some computation on the grouped data.
4. Output: The final results from the reducers are collected as the output of the MapReduce job.

HDFS (Hadoop Distributed File System)

What are the components of the Hadoop Distributed File System(HDFS)?

• HDFS Blocks: HDFS breaks down a file into smaller units. Each of these units is stored on different machines in the cluster. This, however, is transparent to the user working on HDFS. To them, it seems like storing all the data onto a single machine. These smaller units are the blocks in HDFS. The size of each of these blocks is 128MB by default, you can easily change it according to requirement. There are several perks to storing data in blocks rather than saving the complete file.
  • The file itself would be too large to store on any single disk alone. Therefore, it is prudent to spread it across different machines on the cluster.
  • It would also enable a proper spread of the workload and prevent the choke of a single machine by taking advantage of parallelism.
• Namenode in HDFS: HDFS operates in a master-worker architecture, this means that there are one master node and several worker nodes in the cluster. The master node is the Namenode and runs on a separate node in cluster.
  • Manages the filesystem namespace which is the filesystem tree or hierarchy of the files and directories.
  • Stores information like owners of files, file permissions, etc for all the files.
  • It is also aware of the locations of all the blocks of a file and their size.
  All this information is maintained persistently over the local disk in the form of two files: Fsimage (stores the information about the files and directories in the filesystem) and Edit Log (keeps track of all the write operations that the client performs). Whenever a client wants to write information to HDFS or read information from HDFS, it connects with the Namenode. The Namenode returns the location of the blocks to the client and the operation is carried out.
• Datanodes in HDFS: Datanodes are the worker nodes. They are inexpensive commodity hardware that can be easily added to the cluster. These are the nodes that store actual data in “blocks”. Datanodes are responsible for storing, retrieving, replicating, deletion, etc. of blocks when asked by the Namenode.
• Secondary Namenode in HDFS: The Secondary Namenode regularly merges the Edit Log with the Fsimage, creating check-points for the primary Namenode's file system metadata. This efficient process minimizes downtime during Namenode restarts caused by failures.

YARN

• ResourceManager: Think of it as: The traffic controller, managing resources (CPU, memory) and scheduling tasks across the cluster. Responsibilities are:
  • Receives job submissions from clients
  • Negotiates resource allocation with NodeManagers
  • Monitors the progress of running jobs
• NodeManager: Think of it as: The local supervisor on each worker node, overseeing tasks and resource allocation. Main responsibilities are as follows:
  • Starts and monitors tasks assigned by the ResourceManager
  • Manages resources on the local node (CPU, memory, network)
  • Reports resource usage and task progress to the ResourceManager

Key concepts

Hadoop Cluster?: A Hadoop cluster is a network of computers or nodes working together to handle large datasets efficiently. Think of it as a team of specialists, each with their own expertise, tackling a massive project collaboratively. In this analogy, the computers are the specialists, and the data is the project.

---

Hadoop daemons: In the world of Hadoop, daemons are like the tireless workers behind the scenes, ensuring everything runs smoothly and efficiently. Each daemon has a specific role in managing data storage, processing, and resource allocation across the cluster. The core hadoop daemons are:

1. NameNode
2. DataNode
3. Secondary NameNode
4. ResourceManager
5. NodeManagers

These daemons work together to form the backbone of a Hadoop cluster, enabling reliable and efficient data storage and processing. The specific roles and names might vary slightly depending on the Hadoop version and configuration.

---

Nodes:

Nodes are the machines that make up a cluster. They can be physical or virtual servers in a cloud computing environment. These nodes are connected with each other through LAN (Local Area Network). The nodes in a cluster share the data, work on the same task and this nodes are good enough to work as a single unit means all of them to work together. The nodes in a Hadoop cluster can be divided into two types:

• Master Node: This is the node that manages the overall operation of the cluster. It is responsible for assigning tasks to other nodes, monitoring their progress, and handling failures. The master node in a Hadoop cluster is typically a NameNode (NN) in the case of HDFS (Hadoop Distributed File System).
• Worker (slave) Nodes: These are the nodes that perform the actual processing of data. Each worker node is responsible for executing tasks assigned by the master node. In the context of HDFS, the worker nodes are called DataNodes (DN).

where:

• Client: Submits a job request to the Master Node.
• Master Node: Receives the job request and coordinates its execution.
• ResourceManager: Manages resources (CPU, memory) across the cluster and allocates them to tasks.
• NodeMangers: Run on each worker node, managing tasks and reporting progress to the ResourceManager.
• DataNodes: Store data blocks and provide access to them for task execution.
• Tasks: Individual units of work that make up a job, executed on worker nodes.
• Results: Generated by tasks and sent back to the ResourceManager, eventually reaching the client.

Types of Hadoop clusters There are mainly two types of Hadoop clusters:

• Single Node Hadoop Cluster: This type consists of a single machine running all the essential Hadoop daemons (NameNode, DataNode, ResourceManager, NodeManager) on its own.

  Pros:

  • Simple setup: Easy to install and configure, ideal for testing or learning.
  • Low cost: Requires only one machine, minimizing hardware expenses.
  • No additional infrastructure: No need for complex network configurations or cluster management tools.

  Cons:

  • Limited scalability: Cannot handle large datasets effectively due to resource constraints of a single machine.
  • No fault tolerance: A single point of failure – if the machine fails, the entire cluster goes down.
  • Not suitable for production: Primarily used for small-scale deployments or educational purposes.

• Multiple Node Hadoop Cluster: This type distributes the workload across multiple machines, each having specific roles (master nodes, worker nodes).

  Pros:

  • Scalability: Easily add more nodes to handle growing datasets and processing demands.
  • Fault tolerance: Distributed architecture ensures cluster functionality even if individual nodes fail.
  • High performance: Utilizes combined resources of multiple machines for faster processing.

  Cons:

  • Complex setup: Requires more configuration and technical expertise to manage.
  • Higher cost: Hardware and software costs increase with the number of nodes.
  • Additional infrastructure: Requires network setup, cluster management tools, and potentially dedicated resources.

Single and multi-node: Next question one can ask is how to Choose between single-node and multi-node? The choice depends on specific needs:

• For small datasets, basic testing, or learning purposes, a single-node cluster might suffice.
• For production environments with larger datasets, complex workloads, and scalability requirements, a multi-node cluster is essential.

References

1. Apache Spark™ Official Documentation.
2. Databricks Learning Academy.
3. Spark by Examples.