Monitoring Hadoop

With the exponential growth of data and many enterprises crunching more and more data, Hadoop as a data platform has gained a lot of popularity. The Hadoop platform needs to be monitored with respect to how it works and functions. There is an ever-increasing need to keep the Hadoop platform clean and healthy. This book will help you to integrate Hadoop and Nagios in a seamless and easy way. At the start, the book covers the basics of operating system logging and monitoring. Getting to grips with the characteristics of Hadoop monitoring, metrics, and log collection will help Hadoop users, especially Hadoop administrators, diagnose and troubleshoot clusters better. In essence, the book teaches you how to set up an all-inclusive and robust monitoring system for the Hadoop platform. The book also serves as a quick reference to the various metrics available in Hadoop. Concluding with the visualization of Hadoop metrics, you will get acquainted with the workings of Hadoop in a short span of time with the help of step-by-step instructions in each chapter.

Cover; Copyright; Credits; About the Author; About the Reviewers; www.PacktPub.com; Table of Contents; Preface; Chapter 1: Introduction to Monitoring; The need for monitoring; The monitoring tools available in the market; Nagios; Nagios architecture; Installing and configuring Nagios; Installing Nagios; Web interface configuration; Nagios plugins; Verification; Configuration files; Set up monitoring for clients; Ganglia; Ganglia components; Ganglia installation; System logging; Collection; Transportation; Storage; Alerting and analysis; The syslogd and rsyslogd daemons; Summary

Chapter 2: Hadoop Daemons and ServicesHadoop daemons; NameNode; DataNode and TaskTracker; Secondary NameNode; JobTracker and YARN daemons; The communication between daemons; YARN framework; Common issues faced on Hadoop cluster; Host level checks; Nagios server; Configuring Hadoop nodes for monitoring; Summary; Chapter 3: Hadoop Logging; The need for logging events; System logging; Logging levels; Logging in Hadoop; Hadoop logs; Hadoop log level; Hadoop audit; Summary; Chapter 4: HDFS Checks; HDFS overview; Nagios master configuration; The Nagios client configuration; Summary

Chapter 5: MapReduce ChecksMa…