MapReduce Tracer

This section describes Arista’s implementation of MapReduce Tracer, including configuration instructions and command descriptions. Topics covered by this section include the following:

MapReduce Tracer introduction
MapReduce Tracer Configuration
Displaying MapReduce Tracer Results
MapReduce Tracer Commands

MapReduce Tracer introduction

MapReduce Tracer provides a network tool that monitors Hadoop nodes directly connected to Arista switches. MapReduce Tracer requires the following parameters:

Hadoop clusters deployed with a L3 design.
The top of rack switch becomes the default gateway to all attached TaskTracker nodes.
JobTracker RPC ports do not require authentication.
Nodes cannot simultaneously belong to multiple Hadoop clusters.
Access all TaskTrackers within a cluster through a common HTTP access port.
The switch DNS or static host configuration facilitates TaskTracker name resolution.

Map Reduce Tracer supports these Hadoop releases:

Apache 0.20.205
Apache 1.2.1
Cloudera 3u6
Cloudera 4.1.3
Cloudera 4.3.0
HortonWorks 1.3
Cloudera 4.5.0

These sections briefly describe Hadoop, Hadoop data structures, and MapReduce Tracer.

Hadoop Description

Apache Hadoop is an open-source, Java-based software framework that supports large dataset storage and processing in a distributed computational environment. It is licensed under Apache License 2.0 and developed by a global community.

Hadoop facilitates application execution on systems comprising thousands of nodes utilizing petabytes of data. The distributed file system facilitates rapid data transfer among nodes and supports continued operations when individual nodes fail or become inaccessible.

Hadoop Distributed File System (HDFS) contains a distributed file system that stores data on the commodity machines to provide high aggregate bandwidth across the cluster.

Hadoop Cluster Structure

A cluster consists of a server group that functions as a single system to provide high availability through load balancing and parallel processing. A Hadoop cluster provides a type of computational cluster designed for storing and analyzing large amounts of unstructured data in a distributed computing environment.

Typical Hadoop clusters include one main node and multiple worker nodes. The main node consists of TaskTracker, JobTracker, NameNode, and DataNode functions. The worker nodes include a TaskTracker and DataNode.

Map Reduce

Hadoop implements a MapReduce algorithm to process large datasets by distributing parallel tasks to nodes within a cluster. The MapReduce program includes a Map procedure that filters data and a Reduce procedure that processes the data.

MapReduce manages task and data distribution to cluster nodes to execute tasks in parallel, and data transfers between cluster components support redundancy and fault tolerance.

The MapReduce engine comprises one JobTracker and multiple TaskTrackers nodes within the Hadoop cluster. The JobTracker receives MapReduce jobs from a client application and manages the completion of these jobs by submitting tasks to available TaskTracker nodes. If a TaskTracker fails to perform the assigned task, the JobTracker reschedules that part of the job to another node.

MapReduce Tracer Function

MapReduce Tracer allows you to track and interact with Hadoop nodes directly connected to Arista switches in a cluster. MapReduce Tracer communicates with a JobTracker to obtain a list of all nodes in a cluster. Then, it queries JobTracker and TaskTrackers for information regarding the running jobs and the progress of those jobs. This process creates a map of TaskTrackers with the types of jobs. Use show commands to display the data in tables using the CLI and EAPI.

MapReduce Tracer only monitors nodes directly connected to the switch on L3 networks. Directly connected nodes use the top-of-rack switch as the default gateway, and the switch learns ARP entries for these nodes. JobTracker filters nodes by tracking ARP entries to remove nodes not directly accessible and provides a list of nodes.

MapReduce Tracer creates a database of nodes from the filtered list. After creating the database, the switch queries the JobTracker and TaskTrackers to obtain the following:

Number of monitored Hadoop nodes
Llist of monitored nodes including IP addresses
Running jobs by TaskTrackers
JobTracker and TaskTracker statistics

MapReduce Tracer can simultaneously monitor multiple clusters. This means the directly connected TaskTracker nodes can belong to different clusters. eos supports a maximum of five clusters per switch.

MapReduce Tracer Configuration

The MapReduce Tracer structures configuration commands into two configuration levels::

Access the MapReduce Tracer Configuration Mode through the Global Configuration Mode and control the global MapReduce Tracer settings.
Access the Monitor-Hadoop-Cluster Configuration Mode through the Monitor-Hadoop Configuration Mode and define polling configurations to monitor individual Hadoop clusters.

These sections describe MapReduce Tracer configuration processes:

Configuring MapReduce Tracer Globally
Configuring Hadoop Cluster Access
MapReduce Tracer Example

MapReduce Tracer functions after enabling it globally. Each polling configuration can be enabled after enabling the feature.

Configuring MapReduce Tracer Globally

Configure the MapReduce Tracer global parameters in the Monitor-Hadoop Configuration Mode. Tasks performed in this mode include specifying connection parameters to Hadoop clusters and globally enabling MapReduce Tracer.

Accessing the Monitor-Hadoop Configuration Mode

Access the Monitor-Hadoop Configuration Mode using the monitor hadoop command. Monitor-Hadoop Configuration Mode does not apply group changes. The running-config immediately stores the statements when entered. The exit command returns the switch to Global Configuration mode.

Examples

Use the following commands to place the switch in the Monitor-Hadoop Configuration Mode.
```
switch(config)#monitor hadoop
switch(config-monitor-hadoop)#
```
This command exits the Monitor-Hadoop Configuration Mode.
```
switch(config-monitor-hadoop)# exit
switch(config)#
```
This command deletes all previously configured Monitor-Hadoop Configuration Mode commands.
```
switch(config)# no monitor hadoop
switch(config)#
```

Enabling MapReduce Tracer Globally

Globally enable MapReduce Tracer using the no version of the shutdown (Monitor-Hadoop) command. eos globally disables MapReduce Tracer by default.

Example

These commands globally enable MapReduce Tracer.

switch(config)# monitor hadoop
switch(config-monitor-hadoop)# no shutdown
switch(config-monitor-hadoop)# show active
 monitor hadoop
   no shutdown
switch(config-monitor-hadoop)#

Creating a Cluster Monitor

Create a cluster monitor by accessing the Monitor-Hadoop-Cluster Configuration Mode mode with the cluster (Monitor Hadoop) command. Each monitor has a cluster ID and probes one Hadoop cluster. When the command specifies a monitor with a previously defined cluster ID, subsequent commands edit the monitor parameters. Create a new monitor with a new cluster ID by entering a command that specifies a nonexistent cluster ID.

Example

Use the following commands to enter the Monitor-Hadoop-Cluster Configuration Mode to edit a cluster monitor with the cluster-id, CL2:

switch(config-monitor-hadoop)# cluster CL2
switch(config-monitor-hadoop-CL2)# show active
monitor hadoop
   cluster CL2
switch(config-monitor-hadoop-CL2)#

Configuring Hadoop Cluster Access

Configure cluster monitors in the Monitor-Hadoop-Cluster Configuration Mode. Each monitor corresponds to a Hadoop cluster with the following configurable parameters:

JobTracker access parameters including the IP address, port number, and username
TaskTracker access port
Polling interval
Cluster description
Enabled setting

The minimum explicit configuration includes the JobTracker address and username. Use the default values for all other parameters. By default, cluster monitors are disabled.

The cluster (Monitor Hadoop) command places the switch in the Monitor-Hadoop-Cluster Configuration Mode for the specified monitor with a cluster connection parameters.

Enable a cluster monitor by using the no version of the shutdown (Monitor-Hadoop) command when enabling MapReduce Tracer globally.

Configuring JobTracker

A cluster JobTracker resides on the master node and schedules work for the cluster TaskTracker nodes. The jobtracker (Monitor Hadoop Cluster) command specifies connection parameters to the monitored cluster.

JobTracker parameters include the node location as an IPv4 address or hostname, RPC port, and username. The default RPC port is 8021. Location and username parameters do not have default values and must be explicitly configured.

Example

For the CL2 monitor, these commands configure connection parameters to a JobTracker node at 10.4.4.4 with the username account1.

switch(config)# monitor hadoop
switch(config-monitor-hadoop)# cluster CL2
switch(config-monitor-hadoop-CL2)# jobtracker host 10.4.4.4 username account1
switch(config-monitor-hadoop-CL2)# show active
monitor hadoop
   cluster CL2
      jobtracker host 10.4.4.4 user account1
switch(config-monitor-hadoop-CL2)#

TaskTracker Configuration

The tasktracker (Monitor Hadoop Cluster) command specifies the HTTP port that accesses the TaskTrackers for the Hadoop cluster probed by the Configuration Mode Monitor. The switch compiles a list of the cluster TaskTracker addresses by periodically polling the cluster JobTracker.

The TaskTracker HTTP port uses 50060 by default.

Examples

For the CL2 monitor, use the following commands to configure a TaskTracker access port of 51000.

switch(config)# monitor hadoop
switch(config-monitor-hadoop)# cluster CL2
switch(config-monitor-hadoop-CL2)# tasktracker http-port 51000
switch(config-monitor-hadoop-CL2)# show active
monitor hadoop
  cluster CL2
    tasktracker http-port 51000
switch(config-monitor-hadoop-CL2)#

Use the following commands to restore the default TaskTracker HTTP access port address of 50060.

switch(config-monitor-hadoop-CL2)# no tasktracker http-port
switch(config-monitor-hadoop-CL2)# show active
monitor hadoop
  cluster CL2
switch(config-monitor-hadoop-CL2)# show active all
monitor hadoop
  cluster CL2
    jobtracker rpc-port 8021
    tasktracker http-port 50060
    interval 10
    shutdown
switch(config-monitor-hadoop-CL2)#

Configuring Polling Intervals

After completing the Hadoop Monitor configuration, the switch polls the cluster JobTracker to maintain the list of active TaskTracker nodes associated with the monitored cluster and compile Hadoop job statistics. The interval (Monitor Hadoop Cluster) command specifies the interval between polls to the JobTracker of the monitored cluster. By default, the switch uses a polling interval of 10 seconds.

Example

This command sets the JobTracker polling interval to 25 seconds for the cluster monitored by the CL2 MapReduce Tracer configuration.

switch(config)# monitor hadoop
switch(config-monitor-hadoop)# cluster CL2
switch(config-monitor-hadoop-CL2)# interval 25
switch(config-monitor-hadoop-CL2)# show active
monitor hadoop
   cluster CL2
      interval 25
switch(config-monitor-hadoop-CL2)#

MapReduce Tracer Example

Use the following commands to create monitors that probe two Hadoop clusters, enables each monitor individually, then enables MapReduce Tracer globally. Monitor parameters for the clusters include the following:

Cluster ID CL_1

Jobtracker
- IP address - 10.15.2.2
- RPC port - 8021
- username - xyz1
TaskTracker
- HTTP address - 54000
JobTracker
- Polling Interval - 10 seconds (default)

Cluster ID CL_2

Jobtracker
- IP address - 10.21.5.2
- RPC port - 9521
- username - qrst
TaskTracker
- HTTP address - 500060
JobTracker Polling Interval
- Polling Interval - 5 seconds (default)

Example

switch(config)#monitor hadoop
switch(config-monitor-hadoop)#cluster CL_1

switch(config-monitor-hadoop-CL_1)#jobtracker host 10.15.2.2 username xyz1
switch(config-monitor-hadoop-CL_1)#tasktracker http-port 54000
switch(config-monitor-hadoop-CL_1)#no shutdown
switch(config-monitor-hadoop-CL_1)#exit

switch(config-monitor-hadoop)#cluster CL_2
switch(config-monitor-hadoop-CL_2)#jobtracker host 10.21.5.2 rpc-port 9521
switch(config-monitor-hadoop-CL_2)#interval 5
switch(config-monitor-hadoop-CL_2)#no shutdown
switch(config-monitor-hadoop-CL_2)#exit

switch(config-monitor-hadoop)#no shutdown
switch(config-monitor-hadoop)#show active
 monitor hadoop
   no shutdown
   cluster CL_1
      jobtracker host 10.15.2.2 user xyz1
      tasktracker http-port 54000
      no shutdown
   !
   cluster CL_2
      jobtracker host 10.21.5.2 rpc-port 9521 user qrst4
      interval 5
      no shutdown

switch(config-monitor-hadoop)#show active all
 monitor hadoop
   no shutdown
   cluster CL_1
      jobtracker host 10.15.2.2 rpc-port 8021 user xyz1
      tasktracker http-port 54000
      interval 10
      no shutdown
   !
   cluster CL_2
      jobtracker host 10.21.5.2 rpc-port 9521 user qrst4
      tasktracker http-port 50060
      interval 5
      no shutdown
switch(config-monitor-hadoop)#exit
switch(config)#

Displaying MapReduce Tracer Results

MapReduce Tracer display commands provide information about the configuration and activity on the monitored clusters.

This section contains the following topics:

Displaying MapReduce Tracer Status
Displaying the Cluster Configuration and Connections
Displaying Job Lists Information
Displaying Job Data
Displaying TaskTracker Lists
Displaying TaskTracker Connection and Activity
Displaying Data Bursts
Low-memory Mode Overview

Displaying MapReduce Tracer Status

Display the MapReduce Tracer status through the show monitor hadoop status command. Status information includes the enabled status and the number of monitored clusters, TaskTrackers, and locally running jobs.

Example

This command displays MapReduce Tracer status for all connected clusters and TaskTrackers.

switch> show monitor hadoop status

Last updated: 2013-10-06 18:14:23
Mapreduce Tracer status:
  Admin status                            : Enabled
  Operational status                      : Enabled
  Number of clusters configured           : 3
  Number of local TaskTrackers            : 4
  Number of jobs running locally          : 4

switch>

Displaying the Cluster Configuration and Connections

Use the following cluster configuration and connection information commands to display information about the configurations and connections:

show monitor hadoop cluster all - Displays configuration and connection data for all monitored clusters.
show monitor hadoop cluster status - Displays configuration and connection data for a specified cluster.
show monitor hadoop tasktracker status - Displays connection and activity information for TaskTrackers in a specified cluster, on a specified node, or accessed through a specified interface.

Example

Use the following command to display configuration and connection data for the Cluster0 cluster.

switch>show monitor hadoop cluster Cluster0 status

Last updated: 2013-10-06 18:14:23
Cluster status for cluster:  Cluster0
  Admin status                : Enabled
  JobTracker host             : host0
  JobTracker RPC port         : 9000
  JobTracker user             : user0
  JobTracker polling interval : 100 seconds
  TaskTracker HTTP port       : 8800
  Operational status          : Enabled
  Active TaskTrackers         : 31
  Blacklisted TaskTrackers    : 1
  Decommissioned TaskTrackers : 1
  Tracker expiry interval     : 20.0
  Map slots (used/total)      : 10/100
  Reduce slots (used/total)   : 11/110
  JobTracker heap size        : 1.04GB (max: 2.08GB)

switch>

Displaying Job Lists Information

The following commands display rosters of currently running job or previously ran jobs:

show monitor hadoop - Jobs running on all monitored Hadoop clusters.
show monitor hadoop cluster counters - Jobs running on a specified cluster and byte counter data.
show monitor hadoop cluster history - Jobs previously run on a specified cluster.
- clear monitor hadoop job-history - Includes jobs that ran since enabling the monitor, reloading the switch, or clearing the job history.
show monitor hadoop cluster jobs - Jobs running on a specified cluster.
show monitor hadoop history - Jobs that ran on all configured clusters is accessed through.
- clear monitor hadoop job-history - Includes jobs that ran since enabling the monitor, reloading the switch, or clearing the job history.
show monitor hadoop tasktracker counters: Jobs running on a specified TaskTracker and byte counter data.

Examples

Use the following command to display the jobs running on all monitored clusters.

switch> show monitor hadoop

Last updated: 2013-10-06 18:14:23
Currently running jobs: 4
JobId   Job Name         Cluster   Maps(#/%)  Reduces(#/%)  Start Time          
------- ---------------- --------- ---------- ------------- ------------------- 
1       ReallyAVeryLon\  Cluster0  2/12.34%   0/13.45%      2013-10-06 17:56:03 
        gNameForAJob1                                                           
2       ShortName2       Cluster0  2/24.68%   0/26.90%      2013-10-06 17:37:43 
510001  ReallyAVeryLon\  Cluster1  2/12.34%   0/13.45%      2013-10-06 17:56:03 
        gNameForAJob11                                                          
510002  ShortName12      Cluster1  2/24.68%   0/26.90%      2013-10-06 17:37:43 

switch>

Use the following command to display data for the previously run jobs on connected Hadoop clusters.

switch> show monitor hadoop history

Job history for all clusters:
JobId  Job Name         Cluster   Start Time   End Time     Bytes In  Bytes Out 
------ ---------------- --------- ----------- ------------ ---------- --------- 
2      AReallyBigHist\  Cluster0  2013-10-06   2013-10-09   26.08GB   13.04GB   
       oricalJobName              17:41:03     06:47:43                         
442    AReallyBigHist\  Cluster1  2013-10-06   2013-10-09   26.08GB   13.04GB   
       oricalJobName              17:41:03     06:47:43                         
442    AReallyBigHist\  Cluster1  2013-10-06   2013-10-09   26.08GB   13.04GB   
       oricalJobName              17:41:03     06:47:43                         
2      AReallyBigHist\  Cluster0  2013-10-06   2013-10-09   26.08GB   13.04GB   
       oricalJobName              17:41:03     06:47:43                         
441    HistoryJob1      Cluster1  2013-10-06   2013-10-08   26.08GB   13.04GB   
                                  17:57:43     00:31:03                         
1      HistoryJob1      Cluster0  2013-10-06   2013-10-08   26.08GB   13.04GB   
                                  17:57:43     00:31:03                         

switch>

Use the following command to display jobs running on cluster Cluster0 and byte counters for each job.

switch> show monitor hadoop cluster Cluster0 counters

Last updated: 2013-10-06 18:14:23
Counters for currently running jobs on cluster: Cluster0
 JobId   Job Name          User       Bytes In   Bytes Out  Start Time          
------- ----------------- ---------- ---------- ----------- ------------------- 
 2       ShortName2        JobUser2   37.36GB    76.29MB    2013-10-06 17:37:43 
 1       ReallyAVeryLon\   JobUser1   37.36GB    76.29MB    2013-10-06 17:56:03 
         gNameForAJob1 
                                                         
switch>

Displaying Job Data

The following commands display information about active jobs or previously active jobs on monitored clusters. Available data includes Job Identifiers, JobTracker ID, Job start time, Job stop time, data consumption, and progress statistics.

show monitor hadoop cluster history jobs -Data consumption, start and stop times, and JobTracker ID for a specific job.
show monitor hadoop cluster jobs <job number> - Data consumption, start and stop times, priority, JobTracker ID, and progress statistics for a specified job.
show monitor hadoop cluster jobs counter -HDFS (Hadoop Distributed File System) data consumption and and shuffle byte counters for a specified job.
show monitor hadoop counters - Data through and start time for jobs running on all monitored clusters.
show monitor hadoop tasktracker jobs - Progress statistics and start times are available for jobs running on specified TaskTracker.
show monitor hadoop tasktracker running-tasks - Job progress and byte counts of jobs running on a specified Hadoop cluster .
show monitor hadoop tasktracker running-tasks cluster job task - Progress statistics, HDFS data consumption, start time, and progress information for the specified task of a running job.
show monitor hadoop tasktracker counters - Data consumption and start times for jobs running on a specified TaskTracker.

Examples

Use the following command to display information about job 1 on cluster Cluster0.

switch> show monitor hadoop cluster Cluster0 history job 1

Job history data for job: HistoryJob1
  Cluster                  : Cluster0
  Job Id                   : 1
  JT Id                    : 201310110013
  User                     : HistoryUser1
  Job start time           : 2013-10-06 17:57:43
  Job end time             : 2013-10-08 00:31:03

Per Interface job counters:
Interface       TaskTracker        Bytes In       Bytes Out 
--------------- ------------------ -------------- --------- 
Ethernet7       TaskTracker2       26.08GB        13.04GB   

switch>

Use the following command to display information about job 1 running on cluster Cluster0.

switch> show monitor hadoop cluster Cluster0 jobs 1

Last updated: 2013-10-06 18:14:23
Information for job: ReallyAVeryLongNameForAJob1 running on cluster: Cluster0
  Cluster                  : Cluster0
  Id                       : 1
  Name                     : ReallyAVeryLongNameForAJob1
  User                     : JobUser1
  Priority                 : veryHigh
  Running state            : running
  Number of map tasks      : 2
  Number of reduce tasks   : 0
  Start time               : 2013-10-06 17:56:03
  Bytes In                 : 37.36GB
  Bytes Out                : 76.29MB
  Map Progress             : 12.34%
  Reduce Progress          : 13.45%
  Cleanup Progress         : 14.56%
  Setup Progress           : 15.67%

switch>

Use the following command to display data for jobs running on TaskTracker1.

switch> show monitor hadoop tasktracker host TaskTracker1 jobs

Last updated: 2013-10-06 18:14:23
Running job for TaskTracker: TaskTracker1
JobId  Job Name         Cluster   Maps(#/%)   Reduces(#/%)  Start Time          
------ ---------------- --------- ---------- -------------- ------------------- 
1      ReallyAVeryLon\  Cluster0  2/12.34%    0/13.45%      2013-10-06 17:56:03 
       gNameForAJob1                                                            
2      ShortName2       Cluster0  2/24.68%    0/26.90%      2013-10-06 17:37:43 

switch>

Displaying TaskTracker Lists

These commands display lists ofactive TaskTrackers on monitored clusters:

show monitor hadoop cluster tasktracker - TaskTrackers on a specified cluster.
show monitor hadoop tasktracker all - TaskTrackers on all monitored clusters.

Example

Use the following command to display the TaskTrackers on the Cluster0 cluster.

switch> show monitor hadoop cluster Cluster0 tasktracker

Last updated: 2013-10-06 18:14:23
Total 2 TaskTrackers on cluster Cluster0:
Node          IP Address  Interface      Maps   Reduces
------------- ----------- -------------- ------ -------
TaskTracker1  10.100.0.1   Ethernet7      4      0
TaskTracker2  10.100.0.2   Port-Channel7  4      0

switch>

Displaying TaskTracker Connection and Activity

USe the following TaskTracker commands to display connection and activity data:

show monitor hadoop tasktracker status - Connection and activity information for TaskTrackers on a specified cluster or accessed through a specified interface.
show monitor hadoop tasktracker all counters - Data consumption for TaskTrackers connected to monitored clusters.

Example

Use the following command to display connection and activity data for TaskTracker on the TaskTracker1 node.

switch> show monitor hadoop tasktracker host TaskTracker1 status

Last updated: 2013-10-06 18:14:23
  TaskTracker         : TaskTracker1
  IP Address          : 10.100.0.1
  Interface           : Ethernet7
  State               : active
  Running jobs        : 2
  Running tasks       : 4
  Map Tasks           : 4  
  Reduce Tasks        : 0
  Total bytes read    : 2.08GB
  Total bytes written : 4.24MB

switch>

Displaying Data Bursts

The show monitor hadoop traffic burst command displays the largest data bursts for jobs running on a specified cluster or accessed through a specified node or interface. A data burst represents the data consumed during a polling interval.

Example

Use the following command to display traffic burst data for all running jobs that are accessible through port channel interface 7.

switch> show monitor hadoop traffic burst interface Port-Channel 7

Last updated: 2013-10-06 18:14:23
Bursts on Interface: 'Port-Channel7' in cluster: Cluster0
Top 2 input bursts:
JobId       Job Name              Burst     Time                
----------- --------------------- ------------ ------------------- 
 1           ShortName             3.07GB    2013-10-06 17:57:43 
 2           ReallyAVeryLon\       6.15GB    2013-10-06 17:41:03 
             gNameForAJob                                        

Top 2 output bursts:
JobId       Job Name              Burst     Time                
----------- --------------------- ------------ ------------------- 
 1           ShortName             4.10GB    2013-10-06 17:55:13 
 2           ReallyAVeryLon\       8.20GB    2013-10-06 17:36:03 
             gNameForAJob

Low-memory Mode Overview

Switch behavior can be improved using the memory exhaustion command when the switch gets low on available memory. Several factors can cause the low-memory or Out-of-Memory (OOM) state, including the following:

Memory leaks in eos or customer user processes.
Process leaks due to multiple processes or scripts running at one time.
Over configuration of the system that requires more memory than available.
A large number of temporary (tmpfs) files such as logs.
A large number of CLI sessions.
Any event or agents that can consume more memory than available in the system.

Once the switch reaches the low-memory state, the system uses the oom_score, which depends on the value of the oom_score_adj parameter to kill processes and free up available memory. The value for the parameter ranges from -1000 (not shut down) to 1000 (shut down first) during an OOM state. Non-essential agents are assigned oom_score_adj values of 900 or greater.

When enabled, the system implements the following processes:

Non-essential agents shut down by Linux do not restart.
CLIrestarts, and sends a message, "Restarting CLI due to memory exhaustion on the system", to all login sessions in the switch.
Disables CLI Scheduler.

The system requires manual intervention through reset system memory exhaustion command to exit the mode and restart the agents.

Configuring Low-memory Mode

The following examples apply to all platforms.

Examples

These commands configure the low-memory mode.

switch(config)# monitor system
switch(config-monitor-system)# memory exhaustion
switch(config-monitor-system-memory)# action agent non-critical hold-down

The following command disables low-memory mode in config-monitor-system-memory.
```
switch(config-monitor-system-memory)# no action agent non-critical hold-down
```
The following command exits low-memory mode after an OOM event has occurred and agents have been shut down. The command restarts the agents being held down and reenables the CLI scheduler.
```
switch(config-monitor)#reset system memory exhaustion
```

The show monitor system command shows the state of the system with the following information:

Feature enabled.
Low-memory mode (featured enabled and system experiencing OOM).
Number of times the system entered low-memory mode.
Last time the system entered low-memory mode (if applicable).
Last time the system exited low-memory mode (if applicable).

Example

switch#show monitor system

The following output displays on a system with the feature disabled and the system never entered the low-memory mode.

Memory Exhaustion Feature enabled: False
System currently in Memory Exhaustion: False
Number of times system entered Memory Exhaustion:

The following output displays on a system with the feature enabled and the system never entered the low-memory mode.

Memory Exhaustion Feature enabled: True
System currently in Memory Exhaustion: False
Number of times system entered Memory Exhaustion: 0

The following output displays on a system with the feature enabled and the system enters the low-memory mode.

Memory Exhaustion Feature enabled: True
System currently in Memory Exhaustion: True
Number of times system entered Memory Exhaustion: 1
Last time entered in Memory Exhaustion: 0:00:07 ago

The following output displays on a system with the feature enabled and the system enters the low-memory mode once, and has exited the low-memory mode.

Memory Exhaustion Feature enabled: True
System currently in Memory Exhaustion: False
Number of times system entered Memory Exhaustion: 1
Last time entered in Memory Exhaustion: 0:01:00 ago
Last time exited from Memory Exhaustion: 0:00:05 ago

Configuring oom_score_adj for a daemon agent

To configure the oom_score_adj of any configured daemon, use the following command:

switch(config-daemon-<daemon_name>)# oom score adjustment <score>

Note: Similar to change the daemon heartbeat value, the new oom_score_adj value for the daemon is in effect after restarting the daemon by using shutdown command followed by a no shutdown command.

switch(config)# daemon daemonName
switch(config-daemon-daemonName)# oom score adjustment -200
switch(config-daemon-daemonName)# shutdown
switch(config-daemon-daemonName)# no shutdown

MapReduce Tracer Commands

Global Configuration Commands

monitor hadoop

Display Commands

Hadoop Commands

cluster (Monitor Hadoop)
shutdown (Monitor-Hadoop)

Hadoop-Cluster Commands

description (Monitor Hadoop Cluster)
interval (Monitor Hadoop Cluster)
jobtracker (Monitor Hadoop Cluster)
shutdown (Monitor Hadoop Cluster)
tasktracker (Monitor Hadoop Cluster)

clear monitor hadoop burst-counters

The clear monitor hadoop burst-counters command resets MapReduce Tracer burst counters for all jobs running on specified clusters.

Command Mode

Privileged EXEC

Command Syntax

clear monitor hadoop burst-counters [clusters]

Parameters

clusters Hadoop clusters for which command displays data. Options include:
- no parameter All clusters.
- cluster c_name Cluster name.

Example

This command clears the burst counters for all jobs running on CL2 cluster.

switch# clear monitor hadoop burst-counters cluster CL2
Cleared burst counters
switch#

clear monitor hadoop job-history

The clear monitor hadoop job-history command resets the job history database for all specified clusters.

Command Mode

Privileged EXEC

Command Syntax

clear monitor hadoop job-history [clusters]

Parameters

clusters Hadoop clusters for which command displays data. Options include:
- no parameter All clusters.
- cluster c_name Cluster name.

Example

This command clears the job history on the CL2 cluster.

switch# clear monitor hadoop job-history cluster CL2
Cleared job history
switch#

cluster (Monitor Hadoop)

The cluster command is a monitor-hadoop command that places the switch in the monitor-hadoop-cluster mode for configuring and enabling a MapReduce Tracer monitor for a Hadoop cluster. The command either accesses an existing monitor configuration or creates a monitor.

The Monitor-Hadoop-Cluster Configuration Mode is not a group change mode. The running-config changes immediately after entering commands. Exiting the monitor-hadoop-cluster mode does not affect running-config. The exit command returns the switch to the monitor-hadoop configuration mode.

The Configuration Mode Monitor is enabled by the no version of the shutdown (Monitor Hadoop Cluster). Enabling a monitor also requires that MapReduce Tracer is globally enabled (shutdown (Monitor Hadoop Cluster).

The no cluster and default cluster commands remove the specified Hadoop cluster configuration from running-config.

Command Mode

Monitor-Hadoop Configuration

Command Syntax

cluster cluster_name

no cluster cluster_name

default cluster cluster_name

Parameter

cluster_name Hadoop cluster name.

Related Command

The monitor hadoop command places the switch in the monitor-hadoop configuration mode.

Commands Available in Monitor-hadoop-cluster Configuration Mode

description (Monitor Hadoop Cluster)
interval (Monitor Hadoop Cluster)
jobtracker (Monitor Hadoop Cluster)
shutdown (Monitor Hadoop Cluster)
tasktracker (Monitor Hadoop Cluster)

Examples

These commands create the CL2 monitor and enters the monitor-hadoop-cluster mode for the monitor.

switch(config)# monitor hadoop
switch(config-monitor-hadoop)# cluster CL2
switch(config-monitor-hadoop-CL2)# show active
monitor hadoop
   cluster CL2
switch(config-monitor-hadoop-CL2)#

These commands exit the monitor-hadoop-cluster mode.

switch(config-monitor-hadoop-CL2)# exit
switch(config-monitor-hadoop)# show active
 monitor hadoop
   cluster CL2
switch(config-monitor-hadoop)#

These commands remove the CL2 monitor.

switch(config-monitor-hadoop)# no cluster CL2
switch(config-monitor-hadoop)# show active
switch(config-monitor-hadoop)#

description (Monitor Hadoop Cluster)

The description command adds a text string to the configuration mode MapReduce Tracer cluster monitor. The string has no functional impact on the monitor.

The no description and default description commands remove the text string from the configuration mode monitor by removing the corresponding description command from running-config.

Command Mode

Monitor-hadoop-cluster Configuration

Command Syntax

description label_text

no description

default description

Parameter

label_text Character string assigned to the monitor configuration.

Related Command

The cluster (Monitor Hadoop) command places the switch in the monitor-hadoop-cluster configuration mode.

Example

These commands add description text to the CL2 monitor.

switch(config)# monitor hadoop
switch(config-monitor-hadoop)# cluster CL2
switch(config-monitor-hadoop-CL2)# description First Cluster
monitor hadoop
   cluster CL2
      description First Cluster
      jobtracker host 10.3.3.3 user JANE
switch(config-monitor-hadoop-CL2)#

interval (Monitor Hadoop Cluster)

The interval command specifies the polling interval between queries to the Hadoop cluster JobTracker specifed by configuration mode statements. The switch polls a cluster’s JobTracker to update its list of active TaskTracker nodes and the statistics of jobs running in the cluster. This command controls the frequency of these polls. The default interval is 10 seconds.

The no interval and default interval commands restore the default interval of 10 seconds by removing the interval command from running-config.

Command Mode

Monitor-Hadoop Configuration

Command Syntax

interval period

no interval

default interval

Parameter

period Interval (seconds) between JobTracker polls. Value ranges from 1 to 600. Default is 10.

Related Commands

The cluster (Monitor Hadoop) command places the switch in the Monitor-Hadoop Configuration Mode.

Example

This command sets the JobTracker polling interval to 25 seconds for the CL2 cluster configuration.

switch(config)# monitor hadoop
switch(config-monitor-hadoop)# cluster CL2
switch(config-monitor-hadoop-CL2)# interval 25
switch(config-monitor-hadoop-CL2)# show active
monitor hadoop
   cluster CL2
      interval 25
switch(config-monitor-hadoop-CL2)#

jobtracker (Monitor Hadoop Cluster)

The jobtracker command specifies JobTracker access parameters for the cluster monitored by configuration mode monitor statements. A cluster’s JobTracker is located on the master node and schedules work to the cluster’s TaskTracker nodes.

Parameters required to communicate with a JobTracker include its node location (IPv4 address or hostname), RPC port, and username. The default RPC port is 8021. Location and username parameters do not have default values and must be explicity configured. A JobTracker command that specifies a partial parameter list modifies the existing corresponding jobtracker statement in running-config.

The no jobtracker and default jobtracker commands perform the following:

removes the jobtracker statement from running config when it lists all command parameters.
modifies the existing jobtracker statement when it lists a subset of command parameters.

Command Mode

Monitor-Hadoop-Cluster Configuration

Command Syntax

jobtracker [location][port][user]

no jobtracker[location][port][user]

default jobtracker [location][port][user]

All parameters may be placed in any order.

Parameters

location Address or hostname of JobTracker node. Options include:
- no parameter Location remains undefined or unchanged from a previous configuration.
- host ipv4_addr IPv4 address of master (JobTracker) node.
- host hostname Hostname of master (JobTracker) node.
port JobTracker RPC port number. Default value is 8021. Options include:
- no parameter Port number remains unchanged from previous configuration.
- rdp-port port_num Port number of master (JobTracker) node. Value ranges from 1 to 65535.
user Username that accesses JobTracker node. Options include:
- no parameter Username remains undefined or unchanged from previous configuration.
- username name_string JobTracker username.

Related Command

The cluster (Monitor Hadoop) command places the switch in the Monitor-Hadoop-Cluster Configuration Mode.

Examples

For the CL2 cluster configuration, these commands establish a connection to the JobTracker node at 10.4.4.4 with the username account1. The default RPC port (8021) is implicitly specified.

switch(config)# monitor hadoop
switch(config-monitor-hadoop)# cluster CL2
switch(config-monitor-hadoop-CL2)# jobtracker host 10.4.4.4 username account1
switch(config-monitor-hadoop-CL2)# show active

monitor hadoop
   cluster CL2
      jobtracker host 10.4.4.4 user account1

switch(config-monitor-hadoop-CL2)#

These commands modify the JobTracker configuration to specify an RPC port of 9000.

switch(config-monitor-hadoop-CL2)# jobtracker rpc-port 9000
switch(config-monitor-hadoop-CL2)# show active

monitor hadoop
   cluster CL2
      jobtracker host 10.4.4.4 rpc-port 9000 user account1

switch(config-monitor-hadoop-CL2)#

monitor hadoop

The monitor hadoop command places the switch in the Monitor-Hadoop Configuration Mode for configuring MapReduce Tracer monitors. A MapReduce Tracer monitor interacts with Hadoop cluster nodes directly attached to the switch. Tasks that the switch can perform through this interaction include the following:

Compile a list of nodes in the cluster.
Compile a list of jobs the nodes are running.
Download progress of the running jobs.

The Monitor-Hadoop Configuration Mode is not a group change mode, and the running-config changes immediately upon entering commands. Exiting the Monitor-Hadoop Configuration Mode does not affect the running-config. The exit command returns the switch to the Global configuration mode. MapReduce Tracer is enabled in the monitor-hadoop mode through the no version of the shutdown (Monitor Hadoop Cluster) command.

The no monitor hadoop and default monitor hadoop commands delete previously configured monitor hadoop mode configuration commands.

Command Mode

Global Configuration

Command Syntax

monitor hadoop

no monitor hadoop

default monitor hadoop

Commands Available in Monitor-hadoop Configuration Mode

cluster (Monitor Hadoop)
shutdown (Monitor Hadoop Cluster)

Examples

These commands place the switch in the monitor-hadoop configuration mode.
```
switch(config)# monitor hadoop
switch(config-monitor-hadoop)#
```

This command exits the monitor-hadoop mode.

switch(config-monitor-hadoop)# exit
switch(config)#

This command deletes all previously configured monitor-hadoop configuration mode commands.
```
switch(config)# no monitor hadoop
switch(config)#
```

show agent oom scores

The show agent oom scores command displays information for each agent if specified.

Command Mode

EXEC

Command Syntax

show agent [ agentName] oom scores

Parameters

Display Values
- Agent name Name(s) of agent(s) - processes.
- RssAnon Size of the resident anonymous memory for the given process.
- VmSize Current total virtual memory size for the process.
- Max VmSize Peak value of total virtual memory for the process.
- oom_score Parameter used in determining which process to hold down.
- oom_score_adj Parameter used to adjust oom_score.

Example

This command displays the information for SNMP and Sysdb.

switch# show agent Snmp Sysdb oom scores
Agent Name  RssAnon  VmSize  Max VmSize  oom_score oom_score_adj                        
---------- --------- ------- ----------  --------- ------------  
Snmp         34.8MB  536.1MB   541.9MB      911          900
Sysdb        102.3MB 597.5MB   597.5MB      0           -500

show monitor hadoop

The show monitor hadoop command displays a list of jobs that are running on all monitored Hadoop clusters.

Command Mode

EXEC

Command Syntax

show monitor hadoop

Example

This command displays the jobs that are running on all monitored clusters.

switch> show monitor hadoop

Last updated: 2013-10-06 18:14:23
Currently running jobs: 4
JobId  Job Name         Cluster   Maps(#/%)  Reduces(#/%)  Start Time
-----  ---------------- --------- ---------- ------------- -------------------
1      ReallyAVeryLon\  Cluster0  2/12.34%   0/13.45%      2013-10-06 17:56:03
       gNameForAJob1
2      ShortName2       Cluster0  2/24.68%   0/26.90%      2013-10-06 17:37:43
510001 ReallyAVeryLon\  Cluster1  2/12.34%   0/13.45%      2013-10-06 17:56:03    
       gNameForAJob11
510002 ShortName12      Cluster1  2/24.68%   0/26.90%      2013-10-06 17:37:43

switch>

show monitor hadoop cluster all

The show monitor hadoop cluster all command displays configuration and connection information for all monitored Hadoop clusters.

Command Mode

EXEC

Command Syntax

show monitor hadoop cluster all

Example

This command displays configuration and connection data for all connected Hadoop clusters.

switch> show monitor hadoop cluster all

Total number of clusters configured: 3
  Cluster                     : Cluster0
  Admin status                : Enabled
  JobTracker host             : host0
  JobTracker RPC port         : 9000
  JobTracker user             : user0
  JobTracker polling interval : 100 seconds
  TaskTracker HTTP port       : 8800
  Operational status          : Enabled
  Active TaskTrackers         : 31
  Blacklisted TaskTrackers    : 1
  Decommissioned TaskTrackers : 1
  Tracker expiry interval     : 20.0
  Map slots (used/total)      : 10/100
  Reduce slots (used/total)   : 11/110
  JobTracker heap size        : 1.04GB (max: 2.08GB)

  Cluster                     : Cluster1
  Admin status                : Enabled
  JobTracker host             : host1
  JobTracker RPC port         : 9001
  JobTracker user             : user1
  JobTracker polling interval : 101 seconds
  TaskTracker HTTP port       : 8801
  Operational status          : Enabled
  Active TaskTrackers         : 32
  Blacklisted TaskTrackers    : 0
  Decommissioned TaskTrackers : 0
  Tracker expiry interval     : 40.0
  Map slots (used/total)      : 20/200
  Reduce slots (used/total)   : 22/220
  JobTracker heap size        : 2.09GB (max: 4.15GB)

  Cluster                     : Cluster2
  Admin status                : Disabled
  JobTracker host             : host2
  JobTracker RPC port         : 9002
  JobTracker user             : user2
  JobTracker polling interval : 102 seconds
  TaskTracker HTTP port       : 8802
  Operational status          : Disabled

show monitor hadoop cluster counters

The show monitor hadoop cluster counters command displays a list of jobs running on the specified Hadoop cluster and data consumption associated with these jobs.

Command Mode

EXEC

Command Syntax

show monitor hadoop cluster c_name counters

Parameter

c_name Cluster name.

Example

This command displays jobs running on cluster Cluster0.

switch> show monitor hadoop cluster Cluster0 counters

Last updated: 2013-10-06 18:14:23
Counters for currently running jobs on cluster: Cluster0
JobId   Job Name          User Bytes In Bytes   Out Start  Time
------- ----------------- ---------- ---------- ----------- -------------------
2       ShortName2        JobUser2   37.36GB    76.29MB     2013-10-06 17:37:43
1       ReallyAVeryLon\   JobUser1   37.36GB    76.29MB     2013-10-06 17:56:03
        gNameForAJob1

switch>

show monitor hadoop cluster history

The show monitor hadoop cluster history command displays all jobs that ran on the specified cluster. The list includes all jobs that ran since the switch was reloaded, the job history was cleared (clear monitor hadoop job-history), or MapReduce Tracer was enabled.

Command Mode

EXEC

Command Syntax

show monitor hadoop cluster c_name history

Parameter

c_name Cluster name.

Example

This command displays the jobs that were ran on the cluster named Cluster0.

switch> show monitor hadoop cluster Cluster0 history

Jobs history on cluster: Cluster0
JobId    Job Name           Start Time    End Time       Bytes In     Bytes Out
-------- ------------------ ------------- -------------- ------------ ---------
2        AReallyBigHist\    2013-10-06    2013-10-09     26.08GB      13.04GB        
         oricalJobName      17:41:03      06:47:43
2        AReallyBigHist\    2013-10-06    2013-10-09     26.08GB      13.04GB
         oricalJobName      17:41:03      06:47:43
1        HistoryJob1        2013-10-06    2013-10-08     26.08GB      13.04GB
                            17:57:43      00:31:03

switch>

show monitor hadoop cluster history jobs

The show monitor hadoop cluster history jobs command displays data about the specified job. Hadoop jobs are identified by job number and the cluster that ran the job.

Data that the command returns include job identifiers, JobTracker ID, start and stop times, and data consumption.

Command Mode

EXEC

Command Syntax

show monitor hadoop cluster c_name history jobs job_number

Parameters

c_name Cluster name.
job_number Job number. Value ranges from 0 to 2147483647.

Example

This command displays information about job 1 that ran on cluster Cluster0.

switch> show monitor hadoop cluster Cluster0 history job 1

Job history data for job: HistoryJob1
  Cluster               : Cluster0
  Job Id                : 1
  JT Id                 : 201310110013   
  User                  : HistoryUser1
  Job start time        : 2013-10-06 17:57:43
  Job end time          : 2013-10-08 00:31:03

Per Interface job counters:
Interface       TaskTracker        Bytes In       Bytes Out
--------------- ------------------ -------------- ---------
Ethernet7       TaskTracker2       26.08GB        13.04GB

switch>

show monitor hadoop cluster jobs

The show monitor hadoop cluster jobs command displays a list of jobs that are running on the specified cluster.

Command Mode

EXEC

Command Syntax

show monitor hadoop cluster c_name jobs

Parameter

c_name Cluster name.

Example

This command displays the list of jobs running on cluster Cluster0.

switch> show monitor hadoop cluster Cluster0 jobs

Last updated: 2013-10-06 18:14:23
Currently running jobs on cluster: Cluster0
JobId    Job Name           User        Maps    Reduces     Start Time
-------- ------------------ ----------- ------- ----------- -------------------
2        ShortName2         JobUser2    2       0           2013-10-06 17:37:43
1        ReallyAVeryLon\    JobUser1    2       0           2013-10-06 17:56:03
         gNameForAJob1

switch>

show monitor hadoop cluster jobs counter

The show monitor hadoop cluster jobs counter command displays data consumption and progress statistics for the specified job.

Command Mode

EXEC

Command Syntax

show monitor hadoop cluster c_name [jobs job_number] counter

Parameters

c_name Cluster name.
job_number Job number. Value ranges from 0 to 2147483647.

Example

This command displays byte counters for the job named 1 that is running on the cluster named Cluster0.

switch> show monitor hadoop cluster Cluster0 jobs 1 counters

Last updated: 2013-10-06 18:14:23
Cluster : Cluster0
Job Name : ReallyAVeryLongNameForAJob1
Job Id : 1
Interface         HDFS Bytes Read    HDFS Bytes Written    Reduce Shuffle Bytes
----------------- ------------------ --------------------- --------------------
Port-Channel8     4.14GB             8.48MB                12.72MB
Port-Channel9     6.21GB             12.72MB               19.07MB
Ethernet8         3.10GB             6.36MB                9.54MB
Ethernet9         5.17GB             10.60MB               15.89MB
Port-Channel7     2.07GB             4.24MB                6.36MB
Ethernet10        7.24GB             14.83MB               22.25MB
Port-Channel10    8.28GB             16.95MB               25.43MB
Ethernet7         1.03GB             2.12MB                3.18MB

switch>

show monitor hadoop cluster jobs <job number>

The show monitor hadoop cluster jobsjob number command displays information about the specified job. Hadoop jobs are identified by job ID and the cluster that is running the job.

Data that the command returns include time of update, job identifiers, start times, data consumption, and completion progress.

Command Mode

EXEC

Command Syntax

show monitor hadoop cluster c_name [jobs job_number]

Parameters

c_name Cluster name.
job_number Job number. Value ranges from 0 to 2147483647.

Example

This command displays information about job 1 that is running on cluster Cluster0.

switch> show monitor hadoop cluster Cluster0 jobs 1

Last updated: 2013-10-06 18:14:23
Information for job: ReallyAVeryLongNameForAJob1 running on cluster: Cluster0  
  Cluster : Cluster0
  Id                     : 1
  Name                   : ReallyAVeryLongNameForAJob1
  User                   : JobUser1
  Priority               : veryHigh
  Running state          : running
  Number of map tasks    : 2
  Number of reduce tasks : 0
  Start time             : 2013-10-06 17:56:03
  Bytes In               : 37.36GB
  Bytes Out              : 76.29MB
  Map Progress           : 12.34%
  Reduce Progress        : 13.45% 
  Cleanup Progress       : 14.56%
  Setup Progress         : 15.67%

switch>

show monitor hadoop cluster status

The show monitor hadoop cluster status command displays configuration and connection information for the specified cluster.

Command Mode

EXEC

Command Syntax

show monitor hadoop cluster c_name status

Parameter

c_name Cluster name.

Example

This command displays configuration and connection data for the Cluster0 cluster.

switch> show monitor hadoop cluster Cluster0 status

Last updated: 2013-10-06 18:14:23
Cluster status for cluster: Cluster0
  Admin status                  : Enabled  
  JobTracker host               : host0
  JobTracker RPC port           : 9000
  JobTracker user               : user0
  JobTracker polling interval   : 100 seconds
  TaskTracker HTTP port         : 8800  
  Operational status            : Enabled
  Active TaskTrackers           : 31 
  Blacklisted TaskTrackers      : 1 
  Decommissioned TaskTrackers   : 1
  Tracker expiry interval       : 20.0
  Map slots (used/total)        : 10/100
  Reduce slots (used/total)     : 11/110
  JobTracker heap size          : 1.04GB (max: 2.08GB)

switch>

show monitor hadoop cluster tasktracker

The show monitor hadoop cluster tasktracker command displays a list of TaskTrackers in the specified cluster. The IP address and access interface is included in the table.

Command Mode

EXEC

Command Syntax

show monitor hadoop cluster c_name tasktracker

Parameter

c_name Cluster name.

Example

This command displays the TaskTrackers on the Cluster0 cluster.

switch> show monitor hadoop cluster Cluster0 tasktracker

Last updated: 2013-10-06 18:14:23
Total 2 TaskTrackers on cluster Cluster0:
Node          IP Address   Interface      Maps   Reduces
------------- ------------ -------------- ------ -------
TaskTracker1   10.100.0.1   Ethernet7      4      0
TaskTracker2   10.100.0.2   Port-Channel7  4      0

switch>

show monitor hadoop counters

The show monitor hadoop counters command displays byte counter data for all jobs running on clusters for which MapReduce Tracer is configured.

Command Mode

EXEC

Command Syntax

show monitor hadoop counters

Example

This command displays byte counter data for all jobs running on clusters that the switch is accessing through MapReduce Tracer.

switch> show monitor hadoop counters

Last updated: 2013-10-06 18:14:23
Counters for running jobs:
JobId   Job Name          Cluster    Bytes In   Bytes Out   Start Time
------- ----------------- ---------- ---------- ----------- -------------------
510002  ShortName12       Cluster1   37.36GB    76.29MB     2013-10-06 17:37:43
510001  ReallyAVeryLon\   Cluster1   37.36GB    76.29MB     2013-10-06 17:56:03      
        gNameForAJob11
2       ShortName2        Cluster0   37.36GB    76.29MB     2013-10-06 17:37:43
1       ReallyAVeryLon\   Cluster0   37.36GB    76.29MB     2013-10-06 17:56:03    
        gNameForAJob1

switch>

show monitor hadoop history

The show monitor hadoop history command displays jobs that ran on clusters for which MapReduce Tracer is configured. The list includes all jobs that ran since the switch was reloaded, MapReduce Tracer was enabled, or the job history was cleared (clear monitor hadoop job-history).

Command Mode

EXEC

Command Syntax

show monitor hadoop history

Example

This command displays data that jobs that previously ran on connected Hadoop clusters.

switch> show monitor hadoop history

Job history for all clusters:
JobId  Job Name         Cluster   Start Time  End Time     Bytes In   Bytes Out
------ ---------------- --------- ----------- ------------ ---------- ---------
2      AReallyBigHist\  Cluster0  2013-10-06  2013-10-09   26.08GB    13.04GB
       oricalJobName              17:41:03    06:47:43
442    AReallyBigHist\  Cluster1  2013-10-06  2013-10-09   26.08GB    13.04GB       
       oricalJobName              17:41:03    06:47:43
442    AReallyBigHist\  Cluster1  2013-10-06  2013-10-09   26.08GB    13.04GB
       oricalJobName              17:41:03    06:47:43
2      AReallyBigHist\  Cluster0  2013-10-06  2013-10-09   26.08GB    13.04GB
       oricalJobName              17:41:03    06:47:43
441    HistoryJob1      Cluster1  2013-10-06  2013-10-08   26.08GB    13.04GB
                                  17:57:43    00:31:03
1      HistoryJob1      Cluster0  2013-10-06  2013-10-08   26.08GB    13.04GB
                                  17:57:43    00:31:03

switch>

show monitor hadoop status

The show monitor hadoop status command displays system status for MapReduce Tracer.

Command Mode

EXEC

Command Syntax

show monitor hadoop status

Example

This command displays MapReduce Tracer status for all connected clusters and TaskTrackers.

switch> show monitor hadoop status
Last updated: 2013-10-06 18:14:23
Mapreduce Tracer status:  
  Admin status                   : Enabled
  Operational status             : Enabled  
  Number of clusters configured  : 3
  Number of local TaskTrackers   : 4
  Number of jobs running locally : 4

switch>

show monitor hadoop tasktracker all

The show monitor hadoop tasktracker all command displays a list of TaskTrackers that are on all monitored Hadoop clusters.

Command Mode

EXEC

Command Syntax

show monitor hadoop tasktracker all

Example

This command displays the TaskTrackers of all monitored clusters that are connected to the switch.

switch> show monitor hadoop tasktracker all

Last updated: 2013-10-06 18:14:23
All local TaskTrackers:
Node             Cluster      IP Address     Interface         Maps     Reduces
---------------- ------------ -------------- ----------------- -------- -------
TaskTracker1     Cluster0     10.100.0.1      Ethernet7         4        0
TaskTracker3     Cluster1     10.100.0.3      Ethernet8         4        0
TaskTracker2     Cluster0     10.100.0.2      Port-Channel7     4        0
TaskTracker4     Cluster1     10.100.0.4      Port-Channel8     4        0

switch>

show monitor hadoop tasktracker all counters

The show monitor hadoop tasktracker all counters command displays byte counters for the TaskTrackers of all monitored Hadoop clusters.

Command Mode

EXEC

Command Syntax

show monitor hadoop tasktracker all counters

Example

This command displays byte counter data for the TaskTrackers servicing all MapReduce Tracer Hadoop clusters.

switch> show monitor hadoop tasktracker all counters

Last updated: 2013-10-06 18:14:23
Counters for all TaskTrackers:
Node             IP Address     Interface         Bytes Read      Bytes Written
---------------- -------------- ----------------- --------------- -------------
TaskTracker1     10.100.0.1     Ethernet7         2.08GB          4.24MB
TaskTracker3     10.100.0.3     Ethernet8         6.23GB          12.72MB
TaskTracker2     10.100.0.2     Port-Channel7     4.15GB          8.48MB
TaskTracker4     10.100.0.4     Port-Channel8     8.30GB          16.95MB

switch>

show monitor hadoop tasktracker counters

The show monitor hadoop tasktracker counters command displays a list of jobs running on the specified TaskTracker and output from byte counters associated with these jobs.

Command Mode

EXEC

Command Syntax

show monitor hadoop tasktracker NODES counters

Parameters

NODES TaskTracker node access point. Options include:
- host hostname Node name.
- interface ethernet e_range Ethernet interfaces through which node connects.
- interface port-channel p_range Port channel interfaces through which node connects.

Examples

This command displays the jobs running on the TaskTracker on the TaskTracker1 node.

switch> show monitor hadoop tasktracker host TaskTracker1 counters

Last updated: 2013-10-06 18:14:23
Running job for TaskTracker: TaskTracker1
JobId   Job Name          Cluster    Bytes In   Bytes Out   Start Time
------- ----------------- ---------- ---------- ----------- -------------------
2       ShortName2        Cluster0   37.36GB    76.29MB     2013-10-06 17:37:43
1       ReallyAVeryLon\   Cluster0   37.36GB    76.29MB     2013-10-06 17:56:03
        gNameForAJob1

Note: these counters are derived from Hadoop counters and represent approximate
network bandwidth utilization

switch>

This command displays jobs running on TaskTrackers accessed through Ethernet interfaces 7 and 8.

switch> show monitor hadoop tasktracker interface Ethernet 7,8 counters

Last updated: 2013-10-06 18:14:23
Running job for TaskTracker: TaskTracker3
JobId   Job Name          Cluster    Bytes In   Bytes Out   Start Time
------- ----------------- ---------- ---------- ----------- -------------------
510002  ShortName12       Cluster1   37.36GB    76.29MB     2013-10-06 17:37:43
510001  ReallyAVeryLon\   Cluster1   37.36GB    76.29MB     2013-10-06 17:56:03
        gNameForAJob11

Note: These counters are derived from Hadoop counters and represent approximate
network bandwidth utilization

Running job for TaskTracker: TaskTracker1
JobId   Job Name          Cluster    Bytes In   Bytes Out   Start Time
------- ----------------- ---------- ---------- ----------- -------------------
2       ShortName2        Cluster0   37.36GB    76.29MB     2013-10-06 17:37:43
1       ReallyAVeryLon\   Cluster0   37.36GB    76.29MB     2013-10-06 17:56:03
        gNameForAJob1

Note: these counters are derived from Hadoop counters and represent approximate
network bandwidth utilization

switch>

This command displays jobs running on TaskTrackers accessed through port channel interface 7.

switch> show monitor hadoop tasktracker interface Port-Channel 7 counters

Last updated: 2013-10-06 18:14:23
Running job for TaskTracker: TaskTracker2
JobId   Job Name          Cluster    Bytes In   Bytes Out   Start Time
------- ----------------- ---------- ---------- ----------- -------------------
2       ShortName2        Cluster0   37.36GB    76.29MB     2013-10-06 17:37:43
1       ReallyAVeryLon\   Cluster0   37.36GB    76.29MB     2013-10-06 17:56:03
        gNameForAJob1

Note: these counters are derived from Hadoop counters and represent approximate
network bandwidth utilization

switch>

show monitor hadoop tasktracker jobs

The show monitor hadoop tasktracker jobs command displays data about the jobs that are running on TaskTrackers located on the specified node or accessed through the listed interfaces.

Including a cluster parameter filters results to include data only from the cluster polled by the specified monitor.

Command Mode

EXEC

Command Syntax

show monitor hadoop tasktracker NODES jobs [CLUSTERS]

Parameters

NODES TaskTracker node access point. Options include:
- host hostname Node name.
- interface ethernet e_range Ethernet interfaces through which node connects.
- interface port-channel p_range Port channel interfaces through which node connects.
CLUSTERS Hadoop cluster for which command displays data. Options include:
- no parameter TaskTracker on specified NODE can be in any cluster.
- cluster c_name TaskCluster on specified NODE must be in named cluster.

Examples

This command displays data for jobs running on TaskTracker1.

switch> show monitor hadoop tasktracker host TaskTracker1 jobs

Last updated: 2013-10-06 18:14:23
Running job for TaskTracker: TaskTracker1
JobId  Job Name         Cluster   Maps(#/%)  Reduces(#/%)   Start Time
------ ---------------- --------- ---------- -------------- -------------------
1      ReallyAVeryLon\  Cluster0  2/12.34%   0/13.45%       2013-10-06 17:56:03
       gNameForAJob1
2      ShortName2       Cluster0  2/24.68%   0/26.90%       2013-10-06 17:37:43

switch>

This command displays data for jobs on TaskTrackers accessed through Ethernet interfaces 7 and 8.

switch> show monitor hadoop tasktracker interface Ethernet 7,8 jobs

Last updated: 2013-10-06 18:14:23
Running job for TaskTracker: TaskTracker3
JobId   Job Name         Cluster   Maps(#/%)  Reduces(#/%)  Start Time
------- ---------------- --------- ---------- ------------- -------------------
510001  ReallyAVeryLon\  Cluster1  2/12.34%   0/13.45%      2013-10-06 17:56:03
        gNameForAJob11
510002  ShortName12      Cluster1  2/24.68%   0/26.90%      2013-10-06 17:37:43

Running job for TaskTracker: TaskTracker1
JobId  Job Name         Cluster   Maps(#/%)  Reduces(#/%)   Start Time
------ ---------------- --------- ---------- -------------- -------------------
1      ReallyAVeryLon\  Cluster0  2/12.34%   0/13.45%       2013-10-06 17:56:03
       gNameForAJob1
2      ShortName2       Cluster0  2/24.68%   0/26.90%       2013-10-06 17:37:43

switch>

This command displays data for jobs on TaskTrackers accessed through port channel interface 7.

switch> show monitor hadoop tasktracker interface Port-Channel 7 jobs

Last updated: 2013-10-06 18:14:23
Running job for TaskTracker: TaskTracker2
JobId  Job Name         Cluster   Maps(#/%)  Reduces(#/%)   Start Time
------ ---------------- --------- ---------- -------------- -------------------
1      ReallyAVeryLon\  Cluster0  2/12.34%   0/13.45%       2013-10-06 17:56:03
       gNameForAJob1
2      ShortName2       Cluster0  2/24.68%   0/26.90%       2013-10-06 17:37:43

switch>

This command displays data for jobs on TaskTrackers on the Cluster0 cluster that are accessed through Ethernet interfaces 7 and 8.

switch> show monitor hadoop tasktracker interface Ethernet 7,8 jobs cluster Cluster0

Last updated: 2013-10-06 18:14:23
Running job for TaskTracker: TaskTracker1
JobId  Job Name         Cluster   Maps(#/%)  Reduces(#/%)   Start Time
------ ---------------- --------- ---------- -------------- -------------------
1      ReallyAVeryLon\  Cluster0  2/12.34%   0/13.45%       2013-10-06 17:56:03
       gNameForAJob1
2      ShortName2       Cluster0  2/24.68%   0/26.90%       2013-10-06 17:37:43

switch>

This command displays data for jobs on TaskTracker named TaskTracker1 on the Cluster0 cluster.

switch> show monitor hadoop tasktracker host TaskTracker1 jobs cluster Cluster0

Last updated: 2013-10-06 18:14:23
Running job for TaskTracker: TaskTracker1
JobId  Job Name         Cluster   Maps(#/%)  Reduces(#/%)   Start Time
------ ---------------- --------- ---------- -------------- -------------------
1      ReallyAVeryLon\  Cluster0  2/12.34%   0/13.45%       2013-10-06 17:56:03
       gNameForAJob1
2      ShortName2       Cluster0  2/24.68%   0/26.90%       2013-10-06 17:37:43

switch>

This command displays data for jobs on TaskTrackers on the Cluster0 cluster that are accessed through port channel interface 7.

switch> show monitor hadoop tasktracker interface Port-Channel 7 jobs cluster Cluster0

Last updated: 2013-10-06 18:14:23
Running job for TaskTracker: TaskTracker2
JobId  Job Name         Cluster   Maps(#/%)  Reduces(#/%)   Start Time
------ ---------------- --------- ---------- -------------- -------------------
1      ReallyAVeryLon\  Cluster0  2/12.34%   0/13.45%       2013-10-06 17:56:03
       gNameForAJob1
2      ShortName2       Cluster0  2/24.68%   0/26.90%       2013-10-06 17:37:43

switch>

show monitor hadoop tasktracker running-tasks

The show monitor hadoop tasktracker running-tasks command displays progress and byte counts of tasks executed by TaskTrackers located on the specified node or accessed through the listed interfaces.

Including a cluster-ID parameter filters results to include data only from the specified cluster.

Command Mode

EXEC

Command Syntax

show monitor hadoop tasktracker NODES running-tasks [CLUSTERS][JOBS]

Parameters

NODES TaskTracker node access point. Options include:
- host hostname Node name.
- interface ethernet e_range Ethernet interfaces through which node connects.
- interface port-channel p_range Port channel interfaces through which node connects.
CLUSTERS Hadoop cluster for which command displays data. Options include:
- no parameter TaskTracker on specified NODE can be in any cluster.
- cluster c_name TaskCluster on specified NODE must be in named cluster.
JOBS Job list. Options include:
- no parameter all jobs.
- job 0 to 2147483647 Specifies number of single job.

Examples

This command displays data for tasks running on TaskTracker named TaskTracker1.

switch> show monitor hadoop tasktracker host TaskTracker1 running-tasks

Last updated: 2013-10-06 18:14:23
Running tasks for TaskTracker: TaskTracker1 on interface Ethernet7
JobId TaskId  Cluster   Type  Progress  Status   HDFS Read  HDFS Write  Shuffle
----- ------- --------- ----- --------- -------- ---------- ----------- -------
1     2       Cluster0  Map   33.33%    running  2.10MB     2.14MB      2.96MB
2     2       Cluster0  Map   33.33%    running  2.10MB     2.14MB      2.96MB
1     1       Cluster0  Map   50.00%    running  1.05MB     1.07MB      1.48MB
2     1       Cluster0  Map   50.00%    running  1.05MB     1.07MB      1.48MB

switch>

This command displays data for tasks running on the TaskTracker named TaskTracker1 of the Cluster0 cluster.

switch> show monitor hadoop tasktracker host TaskTracker1 running-tasks cluster
Cluster0

Last updated: 2013-10-06 18:14:23
Running tasks for TaskTracker: TaskTracker1 on interface Ethernet7
JobId TaskId  Cluster   Type  Progress  Status   HDFS Read  HDFS Write  Shuffle
----- ------- --------- ----- --------- -------- ---------- ----------- -------
1     2       Cluster0  Map   33.33%    running  2.10MB     2.14MB      2.96MB
2     2       Cluster0  Map   33.33%    running  2.10MB     2.14MB      2.96MB
1     1       Cluster0  Map   50.00%    running  1.05MB     1.07MB      1.48MB
2     1       Cluster0  Map   50.00%    running  1.05MB     1.07MB      1.48MB

switch>

This command displays data for tasks running for job 1 on the TaskTracker named TaskTracker1 of the Cluster0 cluster.

switch> show monitor hadoop tasktracker host TaskTracker1 running-tasks cluster
Cluster0 job 1

Last updated: 2013-10-06 18:14:23
Running tasks for TaskTracker: TaskTracker1 on interface Ethernet7
JobId TaskId  Cluster   Type  Progress  Status   HDFS Read  HDFS Write  Shuffle
----- ------- --------- ----- --------- -------- ---------- ----------- -------
1     2       Cluster0  Map   33.33%    running  2.10MB     2.14MB      2.96MB
1     1       Cluster0  Map   50.00%    running  1.05MB     1.07MB      1.48MB

switch>

This command displays data for tasks running on TaskTrackers accessed through Ethernet interfaces 7 and 8.

switch> show monitor hadoop tasktracker interface Ethernet 7,8 running-tasks

Last updated: 2013-10-06 18:14:23
Running tasks for TaskTracker: TaskTracker1 on interface Ethernet7
JobId TaskId  Cluster   Type  Progress  Status   HDFS Read  HDFS Write  Shuffle
----- ------- --------- ----- --------- -------- ---------- ----------- -------
2     2       Cluster0  Map   33.33%    running  2.10MB     2.14MB      2.96MB
1     1       Cluster0  Map   50.00%    running  1.05MB     1.07MB      1.48MB
2     1       Cluster0  Map   50.00%    running  1.05MB     1.07MB      1.48MB

Running tasks for TaskTracker: TaskTracker3 on interface Ethernet8
JobId  TaskId  Cluster  Type  Progress  Status   HDFS Read  HDFS Write  Shuffle
------ ------ --------- ----- --------- -------- ---------- ----------- -------
510002 222    Cluster1  Map   33.33%    running  2.10MB     2.14MB      2.96MB
510001 112    Cluster1  Map   33.33%    running  2.10MB     2.14MB      2.96MB
510002 221    Cluster1  Map   50.00%    running  1.05MB     1.07MB      1.48MB
510001 111    Cluster1  Map   50.00%    running  1.05MB     1.07MB      1.48MB

switch>

This command displays data for tasks running on TaskTrackers of Cluster0 cluster that are accessed through Ethernet interfaces 7 and 8.

switch> show monitor hadoop tasktracker interface Ethernet 7,8 running-tasks
cluster Cluster0

Last updated: 2013-10-06 18:14:23
Running tasks for TaskTracker: TaskTracker1 on interface Ethernet7
JobId TaskId  Cluster   Type  Progress  Status   HDFS Read  HDFS Write  Shuffle
----- ------- --------- ----- --------- -------- ---------- ----------- -------
1     2      Cluster0   Map   33.33%    running  2.10MB     2.14MB      2.96MB
2     2      Cluster0   Map   33.33%    running  2.10MB     2.14MB      2.96MB
1     1      Cluster0   Map   50.00%    running  1.05MB     1.07MB      1.48MB
2     1      Cluster0   Map   50.00%    running  1.05MB     1.07MB      1.48MB

switch>

This command displays data for tasks running for job 1 on the TaskTrackers of Cluster0 cluster that are accessed through Ethernet interfaces 7 and 8

switch> show monitor hadoop tasktracker interface Ethernet 7,8 running-tasks
cluster Cluster0 job 1

Last updated: 2013-10-06 18:14:23
Running tasks for TaskTracker: TaskTracker1 on interface Ethernet7
JobId TaskId  Cluster   Type  Progress  Status   HDFS Read  HDFS Write  Shuffle
----- ------- --------- ----- --------- -------- ---------- ----------- -------
1     2       Cluster0  Map   33.33%    running  2.10MB     2.14MB      2.96MB
1     1       Cluster0  Map   50.00%    running  1.05MB     1.07MB      1.48MB

switch>

This command displays data for tasks running on TaskTrackers accessed through port channe l interfaces 7 and 8.

switch> show monitor hadoop tasktracker interface Port-Channel 7-8 running-tasks

Last updated: 2013-10-06 18:14:23
Running tasks for TaskTracker: TaskTracker2 on interface Port-Channel7
JobId TaskId  Cluster   Type  Progress  Status   HDFS Read  HDFS Write  Shuffle
----- ------- --------- ----- --------- -------- ---------- ----------- -------
1     2       Cluster0  Map   33.33%    running  2.10MB     2.14MB      2.96MB
1     1       Cluster0  Map   50.00%    running  1.05MB     1.07MB      1.48MB
2     1       Cluster0  Map   50.00%    running  1.05MB     1.07MB      1.48MB

Running tasks for TaskTracker: TaskTracker4 on interface Port-Channel8
JobId  TaskId Cluster   Type  Progress  Status   HDFS Read  HDFS Write  Shuffle
------ ------ --------- ----- --------- -------- ---------- ----------- -------
510002 222    Cluster1  Map   33.33%    running  2.10MB     2.14MB      2.96MB
510001 112    Cluster1  Map   33.33%    running  2.10MB     2.14MB      2.96MB
510001 111    Cluster1  Map   50.00%    running  1.05MB     1.07MB      1.48MB

switch>

This command displays data for tasks running on TaskTrackers of Cluster0 cluster accessed through port channel interface 7.

switch> show monitor hadoop tasktracker interface Port-Channel 7 running-tasks
cluster Cluster0

Last updated: 2013-10-06 18:14:23
Running tasks for TaskTracker: TaskTracker2 on interface Port-Channel7
JobId TaskId  Cluster   Type  Progress  Status   HDFS Read  HDFS Write  Shuffle
----- ------- --------- ----- --------- -------- ---------- ----------- -------
1     2       Cluster0  Map   33.33%    running  2.10MB     2.14MB      2.96MB
1     1       Cluster0  Map   50.00%    running  1.05MB     1.07MB      1.48MB
2     1       Cluster0  Map   50.00%    running  1.05MB     1.07MB      1.48MB

switch>

This command displays data for job 510001 running on TaskTrackers of Cluster1 cluster that are accessed through port channel interface 8.

switch> show monitor hadoop tasktracker interface Port-Channel 8 running-tasks
cluster Cluster1 job 510001

Last updated: 2013-10-06 18:14:23
Running tasks for TaskTracker: TaskTracker4 on interface Port-Channel8
JobId  TaskId Cluster   Type  Progress  Status   HDFS Read  HDFS Write  Shuffle
------ ------ --------- ----- --------- -------- ---------- ----------- -------
510001 112    Cluster1  Map   33.33%    running  2.10MB     2.14MB      2.96MB
510001 111    Cluster1  Map   50.00%    running  1.05MB     1.07MB      1.48MB

switch>

show monitor hadoop tasktracker running-tasks cluster job task

The show monitor hadoop tasktracker running-tasks cluster job task command displays detailed data for the specified task.

Command Mode

EXEC

Command Syntax

show monitor hadoop tasktracker NODE [running-tasks cluster name][ job jnum ][task tnum]

Parameters

NODE TaskTracker node access point. Options include:
- host hostname Node name.
- interface ethernet e_range Ethernet interfaces through which node connects.
- interface port-channel p_range Port channel interfaces through which node connects.
name Cluster name.
jnum Job number. Value ranges from 0 to 2147483647.
tnum Task number. Value ranges from 0 to 2147483647.

Examples

This command displays data for task 1 of job 1 on TaskTracker1 of Cluster0.

switch> show monitor hadoop tasktracker host TaskTracker1 running-tasks cluster Cluster0 job 1 task 1

Last updated: 2013-10-06 18:14:23
Task details for one task as given below: 
  TaskTracker name     : TaskTracker1
  Interface            : 'Ethernet7'
  Cluster              : Cluster0
  Job Id               : 1
  Task Id              : 1
  Attempt Id           : 0
  Task type            : Map
  Status               : running
  State                : running  
  Start time           : 2013-10-06 17:57:43
  Progress             : 50.00%  
  HDFS bytes read      : 1.05MB
  HDFS bytes written   : 1.07MB
  Reduce shuffle bytes : 1.48MB

switch>

This command displays data for task 1 of job 1 on the Cluster0 TaskTracker that is accessible through Ethernet interface 7.

switch> show monitor hadoop tasktracker interface Ethernet 7 running-tasks cluster Cluster0 job 1 task 1

Last updated: 2013-10-06 18:14:23
Task details for one task as given below:  
  TaskTracker name     : TaskTracker1
  Interface            : 'Ethernet7'
  Cluster              : Cluster0
  Job Id               : 1
  Task Id              : 1
  Attempt Id           : 0   
  Task type            : Map
  Status               : running
  State                : running
  Start time           : 2013-10-06 17:57:43
  Progress             : 50.00%
  HDFS bytes read      : 1.05MB
  HDFS bytes written   : 1.07MB
  Reduce shuffle bytes : 1.48MB

switch>

This command displays data for task 111 of job 510001 on the Cluster0 TaskTracker that is accessible through port channel interface 8.

switch> show monitor hadoop tasktracker interface Port-Channel 8 running-tasks cluster Cluster1 job 510001 task 111

Last updated: 2013-10-06 18:14:23
Task details for one task as given below:
  TaskTracker name     : TaskTracker4
  Interface            : 'Port-Channel8'
  Cluster              : Cluster1
  Job Id               : 510001
  Task Id              : 111  
  Attempt Id           : 0
  Task type            : Map  
  Status               : running
  State                : running
  Start time           : 2013-10-06 17:57:43
  Progress             : 50.00%
  HDFS bytes read      : 1.05MB
  HDFS bytes written   : 1.07MB
  Reduce shuffle bytes : 1.48MB

switch>

show monitor hadoop tasktracker status

The show monitor hadoop tasktracker status command displays connection and activity information for the TaskTracker on the specified clusters or accessed through the specified interface. The following command formats display the listed TaskTracker information:

show monitor hadoop cluster c_name tasktracker status: TaskTrackers on specified cluster.
show monitor hadoop tasktracker node status: TaskTrackers on specified nodes or interfaces.
show monitor hadoop tasktracker all status: all connected TaskTrackers.

Command Mode

EXEC

Command Syntax

show monitor hadoop cluster c_name tasktracker status

show monitor hadoop tasktracker NODES status

show monitor hadoop tasktracker all status

Parameters

c_name Cluster name.
NODES TaskTracker node access point. Options include:
- host hostname Node name.
- interface ethernet e_range Ethernet interfaces through which node connects.
- interface port-channel p_range Port channel interfaces through which node connects.

Examples

This command displays connection and activity information for all TaskTrackers connected through Ethernet interfaces 7 and 8.

switch> show monitor hadoop tasktracker interface Ethernet7,8 status

Last updated: 2013-10-06 18:14:23
  TaskTracker            : TaskTracker1
  IP Address             : 10.100.0.1
  Interface              : Ethernet7 
  State                  : active
  Running jobs           : 2
  Running tasks          : 4
  Map Tasks              : 4
  Reduce Tasks           : 0
  Total bytes read       : 2.08GB
  Total bytes written    : 4.24MB
  TaskTracker            : TaskTracker3
  IP Address             : 10.100.0.3
  Interface              : Ethernet8
  State                  : active
  Running jobs           : 2
  Running tasks          : 4
  Map Tasks              : 4
  Reduce Tasks           : 0
  Total bytes read       : 6.23GB
  Total bytes written    : 12.72MB

switch>

This command displays connection and activity information for all connected TaskTrackers.

switch> show monitor hadoop tasktracker all status

Last updated: 2013-10-06 18:14:23
All local TaskTrackers:
  TaskTracker            : TaskTracker4
  IP Address             : 10.100.0.4
  Interface              : Port-Channel8 
  State                  : active
  Running jobs           : 2
  Running tasks          : 4
  Map Tasks              : 4
  Reduce Tasks           : 0
  Total bytes read       : 8.30GB
  Total bytes written    : 16.95MB 

  TaskTracker            : TaskTracker3
  IP Address             : 10.100.0.3
  Interface              : Ethernet8  
  State                  : active
  Running jobs           : 2
  Running tasks          : 4
  Map Tasks              : 4
  Reduce Tasks           : 0
  Total bytes read       : 6.23GB
  Total bytes written    : 12.72MB

  TaskTracker            : TaskTracker2
  IP Address             : 10.100.0.2
  Interface              : Port-Channel7
  State                  : active
  Running jobs           : 2
  Running tasks          : 4
  Map Tasks              : 4
  Reduce Tasks           : 0
  Total bytes read       : 4.15GB
  Total bytes written    : 8.48MB

  TaskTracker            : TaskTracker1
  IP Address             : 10.100.0.1
  Interface              : Ethernet7
  State                  : active
  Running jobs           : 2
  Running tasks          : 4  
  Map Tasks              : 4
  Reduce Tasks           : 0
  Total bytes read       : 2.08GB
  Total bytes written    : 4.24MB

switch>

This command displays connection and activity data for TaskTracker on the TaskTracker1 node.

switch> show monitor hadoop tasktracker host TaskTracker1 status

Last updated: 2013-10-06 18:14:23
  TaskTracker            : TaskTracker1
  IP Address             : 10.100.0.1
  Interface              : Ethernet7  
  State                  : active
  Running jobs           : 2
  Running tasks          : 4
  Map Tasks              : 4
  Reduce Tasks           : 0
  Total bytes read       : 2.08GB
  Total bytes written    : 4.24MB

switch>

This command displays connection and activity data for all TaskTracker connected through Port Channel 7.

switch> show monitor hadoop tasktracker interface Port-Channel 7 status

Last updated: 2013-10-06 18:14:23
  TaskTracker            : TaskTracker2  
  IP Address             : 10.100.0.2
  Interface              : Port-Channel7
  State                  : active  
  Running jobs           : 2
  Running tasks          : 4    
  Map Tasks              : 4
  Reduce Tasks           : 0
  Total bytes read       : 4.15GB
  Total bytes written    : 8.48MB

switch>

show monitor hadoop traffic burst

The show monitor hadoop traffic burst command displays the largest data bursts for specified Hadoop cluster jobs. A data burst is the data consumed during a polling interval. The command displays input and output burst:

Input bursts include bytes written to the host.
Output bursts include bytes written by the host.

Command Mode

EXEC

Command Syntax

show monitor hadoop [CLUSTERS] traffic burst [NODE]

Parameters

CLUSTERS Hadoop clusters for which command displays data. Options include:
- no parameter All clusters.
- cluster c_name Cluster name.
NODES TaskTracker node access point. Options include:
- host hostname Node name.
- interface ethernet e_range Ethernet interfaces through which node connects.
- interface port-channel p_range Port channel interfaces through which node connects.

Examples

This command displays traffic burst data for all running jobs.

switch> show monitor hadoop traffic burst
Last updated: 2013-10-06 18:14:23
Bursts on Interface: 'Ethernet7' in cluster: Cluster0
Top 2 input bursts:
JobId  Job Name         Burst    Time
------ ---------------- -------- -------------------
1      ShortName         3.07GB  2013-10-06 17:57:43
2      ReallyAVeryLon\   6.15GB  2013-10-06 17:41:03
       gNameForAJob

Top 2 output bursts:
JobId  Job Name          Burst    Time
------ ---------------- -------- -------------------
1       ShortName         4.10GB  2013-10-06 17:55:13
2       ReallyAVeryLon\   8.20GB  2013-10-06 17:36:03   
        gNameForAJob

Bursts on Interface: 'Port-Channel7' in cluster: Cluster0
Top 2 input bursts:
JobId  Job Name         Burst    Time
------ ---------------- -------- -------------------
1       ShortName        3.07GB  2013-10-06 17:57:43
2       ReallyAVeryLon\  6.15GB  2013-10-06 17:41:03 
        gNameForAJob

Top 2 output bursts:
JobId  Job Name         Burst    Time
------ ---------------- -------- -------------------
1       ShortName        4.10GB  2013-10-06 17:55:13
2       ReallyAVeryLon\  8.20GB  2013-10-06 17:36:03
        gNameForAJob

Bursts on Interface: 'Ethernet8' in cluster: Cluster1
Top 4 input bursts:
JobId   Job Name         Burst    Time
------- ---------------- -------- -------------------
510001   ShortName        3.07GB  2013-10-06 17:57:43
510002   ReallyAVeryLon\  6.15GB  2013-10-06 17:41:03
         gNameForAJob
510003   ShortName        9.22GB   2013-10-06 17:24:23
510004   ReallyAVeryLon\  12.29GB  2013-10-06 17:07:43
         gNameForAJob

Top 4 output bursts:
JobId  Job Name          Burst    Time
------ ----------------- -------- -------------------
510001  ShortName         4.10GB  2013-10-06 17:55:13
510002  ReallyAVeryLon\   8.20GB  2013-10-06 17:36:03
        gNameForAJob
510003  ShortName         12.29GB  2013-10-06 17:16:53
510004  ReallyAVeryLon\   16.39GB  2013-10-06 16:57:43
        gNameForAJob

Bursts on Interface: 'Port-Channel8' in cluster: Cluster1
Top 4 input bursts:
JobId  Job Name         Burst     Time
------ ---------------- --------- -------------------
3101
510001  ShortName        3.07GB    2013-10-06 17:57:43
510002  ReallyAVeryLon\  6.15GB    2013-10-06 17:41:03
        gNameForAJob
510003  ShortName         9.22GB    2013-10-06 17:24:23
510004  ReallyAVeryLon\   12.29GB   2013-10-06 17:07:43
        gNameForAJob

Top 4 output bursts:
JobId  Job Name         Burst    Time
------ ---------------- -------- -------------------
510001  ShortName        4.10GB  2013-10-06 17:55:13
510002  ReallyAVeryLon\  8.20GB  2013-10-06 17:36:03
        gNameForAJob
510003  ShortName        12.29GB  2013-10-06 17:16:53
510004  ReallyAVeryLon\  16.39GB  2013-10-06 16:57:43
        gNameForAJob

switch>

This command displays traffic burst for all jobs running on TaskTrackers that are accessible through Ethernet interfaces 7 and 8.

switch> show monitor hadoop traffic burst interface Ethernet 7,8

Last updated: 2013-10-06 18:14:23
Bursts on Interface: 'Ethernet7' in cluster: Cluster0
Top 2 input bursts:
JobId  Job Name        Burst  Time
------ --------------- ------ -------------------
1      ShortName       3.07GB 2013-10-06 17:57:43
2      ReallyAVeryLon\ 6.15GB 2013-10-06 17:41:03
       gNameForAJob

Top 2 output bursts:
JobId  Job Name        Burst   Time
------ --------------- ------- -------------------
1      ShortName        4.10GB 2013-10-06 17:55:13
2      ReallyAVeryLon\  8.20GB 2013-10-06 17:36:03
       gNameForAJob

Bursts on Interface: 'Ethernet8' in cluster: Cluster1
Top 4 input bursts:
JobId  Job Name         Burst    Time
------ ---------------- -------- -------------------
510001  ShortName       3.07GB   2013-10-06 17:57:43
510002  ReallyAVeryLon\ 6.15GB   2013-10-06 17:41:03
        gNameForAJob
510003  ShortName        9.22GB   2013-10-06 17:24:23
510004  ReallyAVeryLon\  12.29GB  2013-10-06 17:07:43
        gNameForAJob

Top 4 output bursts:
JobId  Job Name        Burst    Time
------ --------------- -------- -------------------
510001 ShortName        4.10GB  2013-10-06 17:55:13
510002 ReallyAVeryLon\  8.20GB  2013-10-06 17:36:03 
       gNameForAJob
510003 ShortName        12.29GB  2013-10-06 17:16:53
510004 ReallyAVeryLon\  16.39GB  2013-10-06 16:57:43
       gNameForAJob

switch>

This command displays traffic burst data for all running jobs that are accessible through port channel interface 7.

switch> show monitor hadoop traffic burst interface Port-Channel 7

Last updated: 2013-10-06 18:14:23
Bursts on Interface: 'Port-Channel7' in cluster: Cluster0
Top 2 input bursts:
JobId  Job Name        Burst   Time
------ --------------- ------- -------------------
1      ShortName       3.07GB  2013-10-06 17:57:43
2      ReallyAVeryLon\ 6.15GB  2013-10-06 17:41:03
       gNameForAJob

Top 2 output bursts:
JobId  Job Name        Burst   Time
------ --------------- ------- -------------------
1      ShortName       4.10GB  2013-10-06 17:55:13
2      ReallyAVeryLon\ 8.20GB  2013-10-06 17:36:03    
       gNameForAJob

show monitor system

The show monitor system command displays information about the memory usage status of the system including when low-memory situations have led to Out-Of-Memory (OOM) events.

Command Mode

EXEC

Command Syntax

show monitor system

Examples

The following output is for a system where the feature is disabled and the system has never entered the low-memory mode.

Memory Exhaustion Feature enabled: False
System currently in Memory Exhaustion: False
Number of times system entered Memory Exhaustion: 0

The following output is for a system where the feature is enabled and the system has never entered the low-memory mode.

Memory Exhaustion Feature enabled: True
System currently in Memory Exhaustion: False
Number of times system entered Memory Exhaustion: 0

The following output is for a system where the feature is enabled and the system enters the low-memory mode.

Memory Exhaustion Feature enabled: True
System currently in Memory Exhaustion: True
Number of times system entered Memory Exhaustion: 1
Last time entered in Memory Exhaustion: 0:00:07 ago

The following output is for a system where the feature is enabled and the system enters the low-memory mode once, and has exited the low-memory mode.

Memory Exhaustion Feature enabled: True
System currently in Memory Exhaustion: False
Last time entered in Memory Exhaustion: 0:01:00 ago
Last time exited from Memory Exhaustion: 0:00:05 ago

shutdown (Monitor-Hadoop)

The shutdown command globally disables MapReduce Tracer on the switch. Enabling MapReduce Tracer for an individual cluster requires the feature to be globally enabled through this command and enabled for the individual cluster thorough the shutdown (Monitor Hadoop Cluster) command. By default, MapReduce Tracer is globally disabled.

The no shutdown command globally enables MapReduce Tracer. The shutdown and default shutdown commands globally disable MapReduce Tracer by removing the corresponding no shutdown command from running-config.

Command Mode

Monitor-Hadoop Configuration

Command Syntax

shutdown

no shutdown

default shutdown

Related Command

The monitor hadoop command places the switch in the Monitor-Hadoop Configuration Mode.

Examples

These commands globally enable MapReduce Tracer.

switch(config)# monitor hadoop
switch(config-monitor-hadoop)# no shutdown
switch(config-monitor-hadoop)# show active
 monitor hadoop
   no shutdown
switch(config-monitor-hadoop)#

This command globally disables MapReduce Tracer.

switch(config-monitor-hadoop)# shutdown
switch(config-monitor-hadoop)# show active
switch(config-monitor-hadoop)#

shutdown (Monitor Hadoop Cluster)

The shutdown command disables MapReduce Tracer for the configuration mode cluster. Globally disabling MapReduce Tracer (show monitor hadoop) also disables the function on the individual cluster. Enabling MapReduce Tracer for the cluster requires the function to be enabled globally and for the individual cluster.

The no shutdown command configures the MapReduce Tracer setting as enabled for the configuration mode cluster. The shutdown and default shutdown commands disable MapReduce Tracer for the cluster by removing the corresponding no shutdown command from running-config.

Command Mode

Monitor-Hadoop-Cluster Configuration

Command Syntax

shutdown

no shutdown

default shutdown

Related Command

The cluster (Monitor Hadoop) command places the switch in the monitor-hadoop-cluster configuration mode.

Examples

These commands globally enable MapReduce Tracer, then enables it for the CL2 cluster.

switch(config)# monitor hadoop
switch(config-monitor-hadoop)# no shutdown
switch(config-monitor-hadoop)# cluster CL2
switch(config-monitor-hadoop-CL2)# no shutdown
switch(config-monitor-hadoop-CL2)# show active
monitor hadoop
   cluster CL2
      no shutdown
switch(config-monitor-hadoop-CL2)# exit
switch(config-monitor-hadoop)# show active
 monitor hadoop
   no shutdown
   cluster CL2
      no shutdown
switch(config-monitor-hadoop)#

These commands disable MapReduce Tracer for the CL2 cluster. MapReduce Tracer remains globally enabled.

switch(config-monitor-hadoop)# cluster CL2
switch(config-monitor-hadoop-CL2)# shutdown
switch(config-monitor-hadoop-CL2)# show active
monitor hadoop
   cluster CL2
switch(config-monitor-hadoop-CL2)# exit
switch(config-monitor-hadoop)# show active
 monitor hadoop
   no shutdown
   cluster CL2
switch(config-monitor-hadoop)#

tasktracker (Monitor Hadoop Cluster)

The tasktracker command specifies the HTTP port for accessing TaskTrackers of the Hadoop cluster monitored through configuration mode statements. The switch compiles a list of the cluster’s TaskTracker addresses by periodically polling the cluster’s JobTracker (jobtracker (Monitor Hadoop Cluster)). The default TaskTracker HTTP port is 50060.

The no tasktracker and default tasktracker commands restore the configuration mode TaskTracker HTTP port to 50060 by removing the corresponding tasktracker command from running-config.

Command Mode

Monitor-Hadoop Configuration

Command Syntax

tasktracker http-port port_number

no tasktracker http-port

default tasktracker http-port

Parameter

port_num TaskTracker HTTP port number. Value ranges from 1 to 65535. Default value is 50060.

Related Command

The cluster (Monitor Hadoop) command places the switch in the Monitor-Hadoop Configuration Mode.

Examples

These commands specify a TaskTracker HTTP port address of 51000.

switch(config)# monitor hadoop
switch(config-monitor-hadoop)# cluster CL2
switch(config-monitor-hadoop-CL2)# tasktracker http-port 51000
switch(config-monitor-hadoop-CL2)# show active
monitor hadoop
   cluster CL2
      tasktracker http-port 51000
switch(config-monitor-hadoop-CL2)#

These commands restore the default TaskTracker HTTP port address of 50060.

switch(config-monitor-hadoop-CL2)# no tasktracker http-port
switch(config-monitor-hadoop-CL2)# show active
monitor hadoop
   cluster CL2
switch(config-monitor-hadoop-CL2)# show active all
monitor hadoop
   cluster CL2
      jobtracker rpc-port 8021
      tasktracker http-port 50060
      interval 10
      shutdown
switch(config-monitor-hadoop-CL2)#

EOS 4.34.0F User Manual

MapReduce Tracer

MapReduce Tracer introduction

Hadoop Description

Hadoop Cluster Structure

Map Reduce

MapReduce Tracer Function

MapReduce Tracer Configuration

Configuring MapReduce Tracer Globally

Accessing the Monitor-Hadoop Configuration Mode

Enabling MapReduce Tracer Globally

Creating a Cluster Monitor

Configuring Hadoop Cluster Access

Configuring JobTracker

TaskTracker Configuration

Configuring Polling Intervals

MapReduce Tracer Example

Displaying MapReduce Tracer Results

Displaying MapReduce Tracer Status

Displaying the Cluster Configuration and Connections

Displaying Job Lists Information

Displaying Job Data

Displaying TaskTracker Lists

Displaying TaskTracker Connection and Activity

Displaying Data Bursts

Low-memory Mode Overview

Configuring Low-memory Mode

Configuring oom_score_adj for a daemon agent

MapReduce Tracer Commands

Global Configuration Commands

Clear Hadoop Monitor Commands

Display Commands

Hadoop Commands

Hadoop-Cluster Commands

clear monitor hadoop burst-counters

clear monitor hadoop job-history

cluster (Monitor Hadoop)

description (Monitor Hadoop Cluster)

interval (Monitor Hadoop Cluster)

jobtracker (Monitor Hadoop Cluster)

monitor hadoop

show agent oom scores

show monitor hadoop

show monitor hadoop cluster all

show monitor hadoop cluster counters

show monitor hadoop cluster history

show monitor hadoop cluster history jobs

show monitor hadoop cluster jobs

show monitor hadoop cluster jobs counter

show monitor hadoop cluster jobs <job number>

show monitor hadoop cluster status

show monitor hadoop cluster tasktracker

show monitor hadoop counters

show monitor hadoop history

show monitor hadoop status

show monitor hadoop tasktracker all

show monitor hadoop tasktracker all counters

show monitor hadoop tasktracker counters

show monitor hadoop tasktracker jobs

show monitor hadoop tasktracker running-tasks

show monitor hadoop tasktracker running-tasks cluster job task

show monitor hadoop tasktracker status

show monitor hadoop traffic burst

show monitor system

shutdown (Monitor-Hadoop)

shutdown (Monitor Hadoop Cluster)

tasktracker (Monitor Hadoop Cluster)