MapReduce Tracer

This section describes Arista’s implementation of MapReduce Tracer, including configuration instructions and command descriptions. Topics covered by this section include:

MapReduce Tracer Introduction

MapReduce Tracer is a network tool that monitors Hadoop nodes that are directly connected to Arista switches. MapReduce Tracer requires the following:
  • Hadoop clusters are deployed with a L3 design.
  • The top of rack switch is the default gateway to all attached TaskTracker nodes.
  • JobTracker RPC ports do not require authentication.
  • Nodes cannot simultaneously belong to multiple Hadoop clusters.
  • All TaskTrackers within a cluster are accessed through a common HTTP access port.
  • The switch’s DNS or static host configuration facilitates TaskTracker name resolution.
Map Reduce Tracer supports these Hadoop releases:
  • Apache 0.20.205
  • Apache 1.2.1
  • Cloudera 3u6
  • Cloudera 4.1.3
  • Cloudera 4.3.0
  • HortonWorks 1.3
  • Cloudera 4.5.0

These sections briefly describe Hadoop, Hadoop data structures, and MapReduce Tracer.

Hadoop Description

Apache Hadoop is an open-source, Java-based software framework that supports large dataset storage and processing in a distributed computational environment. Hadoop is licensed under Apache License 2.0 and developed through a global community.

Hadoop facilitates application execution on systems composed of thousands of nodes utilizing petabytes of data. Its distributed file system facilitates rapid data transfer among nodes and supports continued operations when individual nodes fail or become inaccessible.

Hadoop Distributed File System (HDFS) is a distributed file-system that stores data on the commodity machines to provide high aggregate bandwidth across the cluster.

Hadoop Cluster Structure

A cluster is a group of servers that function as a single system to provide high availability through load balancing and parallel processing. A Hadoop cluster is a type of computational cluster designed for storing and analyzing large amounts of unstructured data in a distributed computing environment.

Typical Hadoop clusters include one master and multiple worker nodes. The master node consists of a TaskTracker, JobTracker, NameNode and DataNode. Worker nodes include a TaskTracker and DataNode.

Map Reduce

MapReduce is an algorithm that Hadoop implements to process large datasets by distributing parallel tasks to nodes within a cluster. The MapReduce program includes a Map procedure that filters data and a Reduce procedure that processes the data.

MapReduce manages task and data distribution to cluster nodes such that tasks are executed in parallel, and data transfers between cluster components support redundancy and fault tolerance.

The MapReduce engine consists of one JobTracker and multiple TaskTrackers – all nodes within the Hadoop cluster. The JobTracker receives MapReduce jobs from a client application and manages the completion of these jobs by submitting tasks to available TaskTracker nodes. If a TaskTracker fails to perform the assigned task, the JobTracker reschedules that part of the job to another node.

MapReduce Tracer Function

MapReduce Tracer is a feature that tracks and interacts with Hadoop nodes directly connected to Arista switches in a cluster. It communicates with a JobTracker to obtain a list of all nodes in a cluster and then queries JobTracker and TaskTrackers on these nodes for information regarding the jobs they are running and progress of those jobs. This creates a map of TaskTrackers with kinds of jobs they are running. Commands are available to display this data in tables through the CLI and EAPI.

MapReduce Tracer monitors only nodes that connect directly to the switch in L3 networks. Directly connected nodes use the top-of-rack switch as their default gateway and the switch learns ARP entries for these nodes. The list of nodes provided by JobTracker is filtered by tracking ARP entries to remove nodes that are not directly accessible.

MapReduce Tracer creates a database of nodes from the filtered list. After the database is created, the switch queries the JobTracker and TaskTrackers to obtain the following:
  • The number of monitored Hadoop nodes.
  • The list of monitored nodes, including their IP addresses.
  • Jobs that the TaskTrackers are running.
  • JobTracker and TaskTracker statistics.

MapReduce Tracer can simultaneously monitor multiple clusters. This means the directly connected TaskTracker nodes can belong to different clusters. A maximum of five clusters are supported per switch.

MapReduce Tracer Configuration

The MapReduce Tracer configuration commands are structured into two configuration levels:
  • The monitor-hadoop configuration mode is a child of global configuration mode and controls the global MapReduce Tracer settings.
  • The monitor-hadoop-cluster configuration mode is a child of the monitor-hadoop configuration mode and defines polling configurations that monitor individual Hadoop clusters.
These sections describe MapReduce Tracer configuration processes:

MapReduce Tracer functions after it is enabled globally. Each polling configuration can be individually enabled after the feature is enabled globally.

MapReduce Tracer Global Configuration

MapReduce Tracer global parameters are configured in the monitor-hadoop configuration mode. Tasks performed from this mode include specifying connection parameters to Hadoop clusters and globally enabling MapReduce Tracer.

Entering the Monitor-Hadoop Configuration Mode

Monitor-hadoop configuration mode is entered using the monitor hadoop command. Monitor-hadoop configuration mode is not a group change mode; statements are immediately stored in running-config when they are entered through the CLI. The exit command returns the switch to global configuration mode.

Examples
  • These commands place the switch in the monitor-hadoop configuration mode.
    switch(config)# monitor hadoop
    switch(config-monitor-hadoop)#
  • This command exits the monitor-hadoop mode.
    switch(config-monitor-hadoop)# exit
    switch(config)#
  • This command deletes all previously configured monitor-hadoop configuration mode commands.
    switch(config)# no monitor hadoop
    switch(config)#

Globally Enabling MapReduce Tracer

MapReduce Tracer is globally enabled by the no version of the shutdown (Monitor-Hadoop) command. MapReduce Tracer is globally disabled by default.

Example
These commands globally enable MapReduce Tracer.
switch(config)# monitor hadoop
switch(config-monitor-hadoop)# no shutdown
switch(config-monitor-hadoop)# show active
 monitor hadoop
   no shutdown
switch(config-monitor-hadoop)#

Creating a Cluster Monitor

A cluster monitor is created by entering the monitor-hadoop-cluster mode with the cluster (Monitor Hadoop) command. Each monitor is labeled with a cluster ID and probes one Hadoop cluster. When the command specifies a monitor with a previously defined cluster ID, subsequent commands edit that monitor’s parameters. A monitor with a new cluster ID is created by a command that specifies a nonexistent cluster ID.

Example
These commands enter the monitor-hadoop-cluster mode to edit a cluster monitor. The monitor’s cluster-id is CL2.
switch(config-monitor-hadoop)# cluster CL2
switch(config-monitor-hadoop-CL2)# show active
monitor hadoop
   cluster CL2
switch(config-monitor-hadoop-CL2)#

Hadoop Cluster Access Configuration

Cluster monitors are configured in the monitor-hadoop-cluster configuration mode. Each monitor corresponds to a hadoop cluster through these configurable parameters:
  • JobTracker access parameters (address, port number, and username)
  • TaskTracker access port
  • Polling interval
  • Cluster description
  • Enabled setting

The minimum explicit configuration includes JobTracker address and username; default values are defined for all other parameters. By default, cluster monitors are disabled.

The cluster (Monitor Hadoop) command places the switch in the monitor-hadoop-cluster mode for the specified monitor, where a cluster’s connection parameters are specified. The monitor-hadoop-cluster mode is not a group change mode.

A cluster monitor is enabled by using the no version of the shutdown (Monitor-Hadoop) command when MapReduce Tracer is globally enabled.

JobTracker Configuration

A cluster’s JobTracker is located on the master node and schedules work to the cluster’s TaskTracker nodes. The jobtracker (Monitor Hadoop Cluster) command specifies connection parameters to the monitored cluster.

JobTracker parameters include its node location (IPv4 address or hostname), RPC port, and username. The default RPC port is 8021. Location and username parameters do not have default values and must be explicitly configured.

Example
For the CL2 monitor, these commands configure connection parameters to a JobTracker node at 10.4.4.4 with the username account1. The default RPC port (8021) is implicitly specified.
switch(config)# monitor hadoop
switch(config-monitor-hadoop)# cluster CL2
switch(config-monitor-hadoop-CL2)# jobtracker host 10.4.4.4 username account1
switch(config-monitor-hadoop-CL2)# show active
monitor hadoop
   cluster CL2
      jobtracker host 10.4.4.4 user account1
switch(config-monitor-hadoop-CL2)#

TaskTracker Configuration

The tasktracker (Monitor Hadoop Cluster) command specifies the HTTP port that access TaskTrackers of the Hadoop cluster probed by the configuration mode monitor. The switch compiles a list of the cluster’s TaskTracker addresses by periodically polling the cluster’s JobTracker.

The default TaskTracker HTTP port is 50060.

Examples
  • For the CL2 monitor, these commands configure a TaskTracker access port of 51000.
    switch(config)# monitor hadoop
    switch(config-monitor-hadoop)# cluster CL2
    switch(config-monitor-hadoop-CL2)# tasktracker http-port 51000
    switch(config-monitor-hadoop-CL2)# show active
    monitor hadoop
      cluster CL2
        tasktracker http-port 51000
    switch(config-monitor-hadoop-CL2)#
  • These commands restore the default TaskTracker HTTP access port address of 50060.
    switch(config-monitor-hadoop-CL2)# no tasktracker http-port
    switch(config-monitor-hadoop-CL2)# show active
    monitor hadoop
      cluster CL2
    switch(config-monitor-hadoop-CL2)# show active all
    monitor hadoop
      cluster CL2
        jobtracker rpc-port 8021
        tasktracker http-port 50060
        interval 10
        shutdown
    switch(config-monitor-hadoop-CL2)#

Polling Interval Configuration

When the monitor configuration is complete, the switch polls the cluster’s JobTracker to maintain the list of active TaskTracker nodes associated with the monitored cluster and compile Hadoop job statistics. The interval (Monitor Hadoop Cluster) command specifies the interval between polls to the JobTracker of the monitored cluster. The default interval is 10 seconds.

Example
This command sets the JobTracker polling interval to 25 seconds for the cluster monitored by the CL2 MapReduce Tracer configuration.
switch(config)# monitor hadoop
switch(config-monitor-hadoop)# cluster CL2
switch(config-monitor-hadoop-CL2)# interval 25
switch(config-monitor-hadoop-CL2)# show active
monitor hadoop
   cluster CL2
      interval 25
switch(config-monitor-hadoop-CL2)#

MapReduce Tracer Example

The commands in this section create monitors that probe two Hadoop clusters, enables each monitor individually, then enables MapReduce Tracer globally. Monitor parameters for the clusters include:
  • Cluster ID: CL_1
    • Jobtracker: IP address: 10.15.2.2; RPC port: 8021; username: xyz1
    • TaskTracker: HTTP address 54000
    • JobTracker polling interval: 10 seconds (default)
  • Cluster ID: CL_2
    • Jobtracker: IP address: 10.21.5.2; RPC port: 9521; username: qrst4
    • TaskTracker: HTTP address 50060 (default)
    • JobTracker polling interval: 5 seconds
Example
switch(config)# monitor hadoop
switch(config-monitor-hadoop)# cluster CL_1

switch(config-monitor-hadoop-CL_1)# jobtracker host 10.15.2.2 username xyz1
switch(config-monitor-hadoop-CL_1)# tasktracker http-port 54000
switch(config-monitor-hadoop-CL_1)# no shutdown
switch(config-monitor-hadoop-CL_1)# exit

switch(config-monitor-hadoop)# cluster CL_2
switch(config-monitor-hadoop-CL_2)# jobtracker host 10.21.5.2 rpc-port 9521
switch(config-monitor-hadoop-CL_2)# interval 5
switch(config-monitor-hadoop-CL_2)# no shutdown
switch(config-monitor-hadoop-CL_2)# exit

switch(config-monitor-hadoop)# no shutdown
switch(config-monitor-hadoop)# show active
 monitor hadoop
   no shutdown
   cluster CL_1
      jobtracker host 10.15.2.2 user xyz1
      tasktracker http-port 54000
      no shutdown
   !
   cluster CL_2
      jobtracker host 10.21.5.2 rpc-port 9521 user qrst4
      interval 5
      no shutdown

switch(config-monitor-hadoop)# show active all
 monitor hadoop
   no shutdown
   cluster CL_1
      jobtracker host 10.15.2.2 rpc-port 8021 user xyz1
      tasktracker http-port 54000
      interval 10
      no shutdown
   !
   cluster CL_2
      jobtracker host 10.21.5.2 rpc-port 9521 user qrst4
      tasktracker http-port 50060
      interval 5
      no shutdown
switch(config-monitor-hadoop)# exit
switch(config)#

Displaying MapReduce Tracer Results

MapReduce Tracer display commands provide information about the configuration and activity on the monitored clusters.

MapReduce Tracer Status

MapReduce Tracer status is accessed through show monitor hadoop status. Status information includes the enabled status and the number of monitored clusters, TaskTrackers, and locally running jobs.

Example
This command displays MapReduce Tracer status for all connected clusters and TaskTrackers.
switch> show monitor hadoop status

Last updated: 2013-10-06 18:14:23
Mapreduce Tracer status:
  Admin status                            : Enabled
  Operational status                      : Enabled
  Number of clusters configured           : 3
  Number of local TaskTrackers            : 4
  Number of jobs running locally          : 4

switch>

Cluster Configuration and Connections

The following cluster configuration and connection information is available through these commands:
Example
This command displays configuration and connection data for the Cluster0 cluster.
switch> show monitor hadoop cluster Cluster0 status

Last updated: 2013-10-06 18:14:23
Cluster status for cluster:  Cluster0
  Admin status                : Enabled
  JobTracker host             : host0
  JobTracker RPC port         : 9000
  JobTracker user             : user0
  JobTracker polling interval : 100 seconds
  TaskTracker HTTP port       : 8800
  Operational status          : Enabled
  Active TaskTrackers         : 31
  Blacklisted TaskTrackers    : 1
  Decommissioned TaskTrackers : 1
  Tracker expiry interval     : 20.0
  Map slots (used/total)      : 10/100
  Reduce slots (used/total)   : 11/110
  JobTracker heap size        : 1.04GB (max: 2.08GB)

switch>

Job Lists

The following commands display rosters of currently running job or jobs that previously ran:
Examples
  • This command displays the jobs that are running on all monitored clusters.
    switch> show monitor hadoop
    
    Last updated: 2013-10-06 18:14:23
    Currently running jobs: 4
    JobId   Job Name         Cluster   Maps(#/%)  Reduces(#/%)  Start Time          
    ------- ---------------- --------- ---------- ------------- ------------------- 
    1       ReallyAVeryLon\  Cluster0  2/12.34%   0/13.45%      2013-10-06 17:56:03 
            gNameForAJob1                                                           
    2       ShortName2       Cluster0  2/24.68%   0/26.90%      2013-10-06 17:37:43 
    510001  ReallyAVeryLon\  Cluster1  2/12.34%   0/13.45%      2013-10-06 17:56:03 
            gNameForAJob11                                                          
    510002  ShortName12      Cluster1  2/24.68%   0/26.90%      2013-10-06 17:37:43 
    
    switch>
  • This command displays data the jobs that previously ran on connected Hadoop clusters.
    switch> show monitor hadoop history
    
    Job history for all clusters:
    JobId  Job Name         Cluster   Start Time   End Time     Bytes In  Bytes Out 
    ------ ---------------- --------- ----------- ------------ ---------- --------- 
    2      AReallyBigHist\  Cluster0  2013-10-06   2013-10-09   26.08GB   13.04GB   
           oricalJobName              17:41:03     06:47:43                         
    442    AReallyBigHist\  Cluster1  2013-10-06   2013-10-09   26.08GB   13.04GB   
           oricalJobName              17:41:03     06:47:43                         
    442    AReallyBigHist\  Cluster1  2013-10-06   2013-10-09   26.08GB   13.04GB   
           oricalJobName              17:41:03     06:47:43                         
    2      AReallyBigHist\  Cluster0  2013-10-06   2013-10-09   26.08GB   13.04GB   
           oricalJobName              17:41:03     06:47:43                         
    441    HistoryJob1      Cluster1  2013-10-06   2013-10-08   26.08GB   13.04GB   
                                      17:57:43     00:31:03                         
    1      HistoryJob1      Cluster0  2013-10-06   2013-10-08   26.08GB   13.04GB   
                                      17:57:43     00:31:03                         
    
    switch>
  • This command displays jobs running on cluster Cluster0 and byte counters for each job.
    switch> show monitor hadoop cluster Cluster0 counters
    
    Last updated: 2013-10-06 18:14:23
    Counters for currently running jobs on cluster: Cluster0
     JobId   Job Name          User       Bytes In   Bytes Out  Start Time          
    ------- ----------------- ---------- ---------- ----------- ------------------- 
     2       ShortName2        JobUser2   37.36GB    76.29MB    2013-10-06 17:37:43 
     1       ReallyAVeryLon\   JobUser1   37.36GB    76.29MB    2013-10-06 17:56:03 
             gNameForAJob1 
                                                             
    switch>

Job Data

The following commands display information about jobs that are running or previously ran on monitored clusters. Available data include job identifiers, JobTracker ID, start time, stop time, data consumption, and progress statistics.

Examples
  • This command displays information about job 1 that ran on cluster Cluster0.
    switch> show monitor hadoop cluster Cluster0 history job 1
    
    Job history data for job: HistoryJob1
      Cluster                  : Cluster0
      Job Id                   : 1
      JT Id                    : 201310110013
      User                     : HistoryUser1
      Job start time           : 2013-10-06 17:57:43
      Job end time             : 2013-10-08 00:31:03
    
    Per Interface job counters:
    Interface       TaskTracker        Bytes In       Bytes Out 
    --------------- ------------------ -------------- --------- 
    Ethernet7       TaskTracker2       26.08GB        13.04GB   
    
    switch>
  • This command displays information about job 1 that is running on cluster Cluster0.
    switch> show monitor hadoop cluster Cluster0 jobs 1
    
    Last updated: 2013-10-06 18:14:23
    Information for job: ReallyAVeryLongNameForAJob1 running on cluster: Cluster0
      Cluster                  : Cluster0
      Id                       : 1
      Name                     : ReallyAVeryLongNameForAJob1
      User                     : JobUser1
      Priority                 : veryHigh
      Running state            : running
      Number of map tasks      : 2
      Number of reduce tasks   : 0
      Start time               : 2013-10-06 17:56:03
      Bytes In                 : 37.36GB
      Bytes Out                : 76.29MB
      Map Progress             : 12.34%
      Reduce Progress          : 13.45%
      Cleanup Progress         : 14.56%
      Setup Progress           : 15.67%
    
    switch>
  • This command displays data for jobs running on TaskTracker1.
    switch> show monitor hadoop tasktracker host TaskTracker1 jobs
    
    Last updated: 2013-10-06 18:14:23
    Running job for TaskTracker: TaskTracker1
    JobId  Job Name         Cluster   Maps(#/%)   Reduces(#/%)  Start Time          
    ------ ---------------- --------- ---------- -------------- ------------------- 
    1      ReallyAVeryLon\  Cluster0  2/12.34%    0/13.45%      2013-10-06 17:56:03 
           gNameForAJob1                                                            
    2      ShortName2       Cluster0  2/24.68%    0/26.90%      2013-10-06 17:37:43 
    
    switch>

TaskTracker Lists

These commands display lists of TaskTrackers that are active on monitored clusters:
Example
This command displays the TaskTrackers on the Cluster0 cluster.
switch> show monitor hadoop cluster Cluster0 tasktracker

Last updated: 2013-10-06 18:14:23
Total 2 TaskTrackers on cluster Cluster0:
Node          IP Address  Interface      Maps   Reduces
------------- ----------- -------------- ------ -------
TaskTracker1  10.100.0.1   Ethernet7      4      0
TaskTracker2  10.100.0.2   Port-Channel7  4      0

switch>

TaskTracker Connection and Activity

The following TaskTracker connection and activity data is available through these commands:
Example
This command displays connection and activity data for TaskTracker on the TaskTracker1 node.
switch> show monitor hadoop tasktracker host TaskTracker1 status

Last updated: 2013-10-06 18:14:23
  TaskTracker         : TaskTracker1
  IP Address          : 10.100.0.1
  Interface           : Ethernet7
  State               : active
  Running jobs        : 2
  Running tasks       : 4
  Map Tasks           : 4  
  Reduce Tasks        : 0
  Total bytes read    : 2.08GB
  Total bytes written : 4.24MB

switch>

Data Bursts

The show monitor hadoop traffic burst command displays the largest data bursts for jobs running on a specified cluster or accessed through a specified node or interface. A data burst is the data consumed during a polling interval.

Example

This command displays traffic burst data for all running jobs that are accessible through port channel interface 7.
switch> show monitor hadoop traffic burst interface Port-Channel 7

Last updated: 2013-10-06 18:14:23
Bursts on Interface: 'Port-Channel7' in cluster: Cluster0
Top 2 input bursts:
JobId       Job Name              Burst     Time                
----------- --------------------- ------------ ------------------- 
 1           ShortName             3.07GB    2013-10-06 17:57:43 
 2           ReallyAVeryLon\       6.15GB    2013-10-06 17:41:03 
             gNameForAJob                                        

Top 2 output bursts:
JobId       Job Name              Burst     Time                
----------- --------------------- ------------ ------------------- 
 1           ShortName             4.10GB    2013-10-06 17:55:13 
 2           ReallyAVeryLon\       8.20GB    2013-10-06 17:36:03 
             gNameForAJob

Low-memory Mode

Switch behavior can be improved using the memory exhaustion command when the switch gets low on available memory. This low-memory or Out-of-Memory (OOM) state is caused by a number of individual or a combination of factors including the one listed below:
  • Memory leaks in EOS or customer user processes.
  • Process leaks to too many processes or scripts running at one time.
  • Over configuration of the system requiring more memory than available.
  • Large number of temporary (tmpfs) files such as logs.
  • Large number of CLI sessions.
  • Any event or agents that can consume more memory than available in the system.

Once the low-memory state is reached, the system uses the oom_score, which depends on the value of the oom_score_adj parameter to kill processes to free up available memory. The value for the parameter ranges from -1000 (will not be shut down) to 1000 (will be shut down first) during an OOM state. Non-essential agents are assigned oom_score_adj values of 900 or greater.

When enabled, the system implements the following:
  • Non-essential agents shut down by Linux are not restarted.
  • CLI is restarted, and a message: "Restarting CLI due to memory exhaustion on the system" is sent to all login sessions in the switch.
  • CLI Scheduler is disabled.

The system requires manual intervention through reset system memory exhaustion command to exit the mode and restart the agents that were shut down.

Configuring Low-memory Mode

The following examples are applicable to all platforms.

Example

These commands configure the low-memory mode.

switch(config)# monitor system
switch(config-monitor-system)# memory exhaustion
switch(config-monitor-system-memory)# action agent non-critical hold-down
Example

The following command disables low-memory mode in config-monitor-system-memory.

switch(config-monitor-system-memory)# no action agent non-critical hold-down

The following command exits low-memory mode after an OOM event has occurred and agents have been shut down. The command restarts the agents being held down and reenables the CLI scheduler.

switch(config-monitor)# reset system memory exhaustion
The show monitor system command shows the state of the system with the following information:
  • If the feature is enabled.
  • If the system is in low-memory mode (featured enabled and system experiencing OOM).
  • How many times the system entered low-memory mode.
  • Last time the system entered low-memory mode (if applicable).
  • Last time the system exited low-memory mode (if applicable).
Example
switch# show monitor system

The following output is for a system where the feature is disabled and the system has never entered the low-memory mode.

Memory Exhaustion Feature enabled: False
System currently in Memory Exhaustion: False
Number of times system entered Memory Exhaustion:

The following output is for a system where the feature is enabled and the system has never entered the low-memory mode.

Memory Exhaustion Feature enabled: True
System currently in Memory Exhaustion: False
Number of times system entered Memory Exhaustion: 0

The following output is for a system where the feature is enabled and the system enters the low-memory mode.

Memory Exhaustion Feature enabled: True
System currently in Memory Exhaustion: True
Number of times system entered Memory Exhaustion: 1
Last time entered in Memory Exhaustion: 0:00:07 ago

The following output is for a system where the feature is enabled and the system enters the low-memory mode once, and has exited the low-memory mode.

Memory Exhaustion Feature enabled: True
System currently in Memory Exhaustion: False
Number of times system entered Memory Exhaustion: 1
Last time entered in Memory Exhaustion: 0:01:00 ago
Last time exited from Memory Exhaustion: 0:00:05 ago

Configuring oom_score_adj for a daemon agent

To configure the oom_score_adj of any configured daemon, use the following command:

switch(config-daemon-<daemon_name>)# oom score adjustment <score>
Note: Similar to change the daemon heartbeat value, the new oom_score_adj value for the daemon is in effect after restarting the daemon by using shutdown command followed by a no shutdown command.
switch(config)# daemon daemonName
switch(config-daemon-daemonName)# oom score adjustment -200
switch(config-daemon-daemonName)# shutdown
switch(config-daemon-daemonName)# no shutdown

MapReduce Tracer Commands

clear monitor hadoop burst-counters

The clear monitor hadoop burst-counters command resets MapReduce Tracer burst counters for all jobs running on specified clusters.

Command Mode

Privileged EXEC

Command Syntax

clear monitor hadoop burst-counters [CLUSTERS]

Parameters
  • CLUSTERS Hadoop clusters for which command displays data. Options include:
    • no parameter      All clusters.
    • cluster c_name      Cluster name.
Example
This command clears the burst counters for all jobs running on CL2 cluster.
switch# clear monitor hadoop burst-counters cluster CL2
Cleared burst counters
switch#

clear monitor hadoop job-history

The clear monitor hadoop job-history command resets the job history database for all specified clusters.

Command Mode

Privileged EXEC

Command Syntax

clear monitor hadoop job-history [CLUSTERS]

Parameters
  • CLUSTERS Hadoop clusters for which command displays data. Options include:
    • no parameter      All clusters.
    • cluster c_name      Cluster name.
Example

This command clears the job history on the CL2 cluster.

switch# clear monitor hadoop job-history cluster CL2
Cleared job history
switch#

cluster (Monitor Hadoop)

The cluster command is a monitor-hadoop command that places the switch in the monitor-hadoop-cluster mode for configuring and enabling a MapReduce Tracer monitor for a Hadoop cluster. The command either accesses an existing monitor configuration or creates a monitor.

The monitor-hadoop-cluster configuration mode is not a group change mode; running-config is changed immediately upon entering commands. Exiting the monitor-hadoop-cluster mode does not affect running-config. The exit command returns the switch to the monitor-hadoop configuration mode.

The configuration mode monitor is enabled by the no version of the shutdown (Monitor Hadoop Cluster). Enabling a monitor also requires that MapReduce Tracer is globally enabled (shutdown (Monitor Hadoop Cluster).

The no cluster and default cluster commands remove the specified Hadoop cluster configuration from running-config.

Command Mode

Monitor-hadoop Configuration

Command Syntax

cluster cluster_name

no cluster cluster_name

default cluster cluster_name

Parameters

cluster_name      Hadoop cluster name.

Related Command

The monitor hadoop command places the switch in the monitor-hadoop configuration mode.

Examples
  • These commands create the CL2 monitor and enters the monitor-hadoop-cluster mode for the monitor.
    switch(config)# monitor hadoop
    switch(config-monitor-hadoop)# cluster CL2
    switch(config-monitor-hadoop-CL2)# show active
    monitor hadoop
       cluster CL2
    switch(config-monitor-hadoop-CL2)#
  • These commands exit the monitor-hadoop-cluster mode.
    switch(config-monitor-hadoop-CL2)# exit
    switch(config-monitor-hadoop)# show active
     monitor hadoop
       cluster CL2
    switch(config-monitor-hadoop)#
  • These commands remove the CL2 monitor.
    switch(config-monitor-hadoop)# no cluster CL2
    switch(config-monitor-hadoop)# show active
    switch(config-monitor-hadoop)#

description (Monitor Hadoop Cluster)

The description command adds a text string to the configuration mode MapReduce Tracer cluster monitor. The string has no functional impact on the monitor.

The no description and default description commands remove the text string from the configuration mode monitor by removing the corresponding description command from running-config.

Command Mode

Monitor-hadoop-cluster Configuration

Command Syntax

description label_text

no description

default description

Parameters

label_text      Character string assigned to the monitor configuration.

Related Commands

The cluster (Monitor Hadoop) command places the switch in the monitor-hadoop-cluster configuration mode.

Example
These commands add description text to the CL2 monitor.
switch(config)# monitor hadoop
switch(config-monitor-hadoop)# cluster CL2
switch(config-monitor-hadoop-CL2)# description First Cluster
monitor hadoop
   cluster CL2
      description First Cluster
      jobtracker host 10.3.3.3 user JANE
switch(config-monitor-hadoop-CL2)#

interval (Monitor Hadoop Cluster)

The interval command specifies the polling interval between queries to the Hadoop cluster JobTracker specifed by configuration mode statements. The switch polls a cluster’s JobTracker to update its list of active TaskTracker nodes and the statistics of jobs running in the cluster. This command controls the frequency of these polls. The default interval is 10 seconds.

The no interval and default interval commands restore the default interval of 10 seconds by removing the interval command from running-config.

Command Mode

Monitor-hadoop-cluster Configuration

Command Syntax

interval period

no interval

default interval

Parameters

period Interval (seconds) between JobTracker polls. Value ranges from 1 to 600. Default is 10.

Related Commands

The cluster (Monitor Hadoop) command places the switch in the monitor-hadoop-cluster configuration mode.

Example
This command sets the JobTracker polling interval to 25 seconds for the CL2 cluster configuration.
switch(config)# monitor hadoop
switch(config-monitor-hadoop)# cluster CL2
switch(config-monitor-hadoop-CL2)# interval 25
switch(config-monitor-hadoop-CL2)# show active
monitor hadoop
   cluster CL2
      interval 25
switch(config-monitor-hadoop-CL2)#

jobtracker (Monitor Hadoop Cluster)

The jobtracker command specifies JobTracker access parameters for the cluster monitored by configuration mode monitor statements. A cluster’s JobTracker is located on the master node and schedules work to the cluster’s TaskTracker nodes.

Parameters required to communicate with a JobTracker include its node location (IPv4 address or hostname), RPC port, and username. The default RPC port is 8021. Location and username parameters do not have default values and must be explicity configured. A JobTracker command that specifies a partial parameter list modifies the existing corresponding jobtracker statement in running-config.

The no jobtracker and default jobtracker commands perform the following:
  • removes the jobtracker statement from running config when it lists all command parameters.
  • modifies the existing jobtracker statement when it lists a subset of command parameters.

Command Mode

Monitor-hadoop-cluster Configuration

Command Syntax

jobtracker [LOCATION][PORT][USER]

no jobtracker[LOCATION][PORT][USER]

default jobtracker [LOCATION][][USER]

All parameters may be placed in any order.

Parameters
  • LOCATION Address or hostname of JobTracker node. Options include:
    • no parameter      Location remains undefined or unchanged from a previous configuration.
    • host ipv4_addr      IPv4 address of master (JobTracker) node.
    • host hostname      Hostname of master (JobTracker) node.
  • PORT JobTracker RPC port number. Default value is 8021. Options include:
    • no parameter      Port number remains unchanged from previous configuration.
    • rdp-port port_num      Port number of master (JobTracker) node. Value ranges from 1 to 65535.
  • USER Username that accesses JobTracker node. Options include:
    • no parameter      Username remains undefined or unchanged from previous configuration.
    • username name_string      JobTracker username.

Related Command

The cluster (Monitor Hadoop) command places the switch in the monitor-hadoop-cluster configuration mode.

Examples
  • For the CL2 cluster configuration, these commands establish a connection to the JobTracker node at 10.4.4.4 with the username account1. The default RPC port (8021) is implicitly specified.
    switch(config)# monitor hadoop
    switch(config-monitor-hadoop)# cluster CL2
    switch(config-monitor-hadoop-CL2)# jobtracker host 10.4.4.4 username account1
    switch(config-monitor-hadoop-CL2)# show active
    
    monitor hadoop
       cluster CL2
          jobtracker host 10.4.4.4 user account1
    
    switch(config-monitor-hadoop-CL2)#
  • These commands modify the JobTracker configuration to specify an RPC port of 9000.
    switch(config-monitor-hadoop-CL2)# jobtracker rpc-port 9000
    switch(config-monitor-hadoop-CL2)# show active
    
    monitor hadoop
       cluster CL2
          jobtracker host 10.4.4.4 rpc-port 9000 user account1
    
    switch(config-monitor-hadoop-CL2)#

monitor hadoop

The monitor hadoop command places the switch in the monitor-hadoop configuration mode for configuring MapReduce Tracer monitors. A MapReduce Tracer monitor interacts with Hadoop cluster nodes that are directly attached to the switch. Tasks that the switch can perform through this interaction include:
  • compile a list of nodes in the cluster.
  • compile a list of jobs the nodes are running.
  • download progress of the running jobs.

The monitor-hadoop configuration mode is not a group change mode; the running-config is changed immediately upon entering commands. Exiting the monitor-hadoop configuration mode does not affect the running-config. The exit command returns the switch to the global configuration mode. MapReduce Tracer is enabled in the monitor-hadoop mode through the no version of the shutdown (Monitor Hadoop Cluster) command.

The no monitor hadoop and default monitor hadoop commands delete previously configured monitor hadoop mode configuration commands.

Command Mode

Global Configuration

Command Syntax

monitor hadoop

no monitor hadoop

default monitor hadoop

Commands Available in Monitor-hadoop Configuration Mode
Examples
  • These commands place the switch in the monitor-hadoop configuration mode.
    switch(config)# monitor hadoop
    switch(config-monitor-hadoop)#
  • This command exits ythe monitor-hadoop mode.
    switch(config-monitor-hadoop)# exit
    switch(config)#
  • This command deletes all previously configured monitor-hadoop configuration mode commands.
    switch(config)# no monitor hadoop
    switch(config)#

show agent oom scores

The show agent oom scores command displays information for each agent if specified.

Command Mode

EXEC

Command Syntax

show agent [ agentName] oom scores

Parameters
  • Display Values
    • Agent name Name(s) of agent(s) - processes.
    • RssAnon Size of the resident anonymous memory for the given process.
    • VmSize Current total virtual memory size for the process.
    • Max VmSize Peak value of total virtual memory for the process.
    • oom_score Parameter used in determining which process to hold down.
    • oom_score_adj Parameter used to adjust oom_score.

Example

This command displays the information for SNMP and Sysdb.
switch# show agent Snmp Sysdb oom scores
Agent Name  RssAnon  VmSize  Max VmSize  oom_score oom_score_adj                        
---------- --------- ------- ----------  --------- ------------  
Snmp         34.8MB  536.1MB   541.9MB      911          900
Sysdb        102.3MB 597.5MB   597.5MB      0           -500

show monitor hadoop

The show monitor hadoop command displays a list of jobs that are running on all monitored Hadoop clusters.

Command Mode

EXEC

Command Syntax

show monitor hadoop

Example
This command displays the jobs that are running on all monitored clusters.
switch> show monitor hadoop

Last updated: 2013-10-06 18:14:23
Currently running jobs: 4
JobId  Job Name         Cluster   Maps(#/%)  Reduces(#/%)  Start Time
-----  ---------------- --------- ---------- ------------- -------------------
1      ReallyAVeryLon\  Cluster0  2/12.34%   0/13.45%      2013-10-06 17:56:03
       gNameForAJob1
2      ShortName2       Cluster0  2/24.68%   0/26.90%      2013-10-06 17:37:43
510001 ReallyAVeryLon\  Cluster1  2/12.34%   0/13.45%      2013-10-06 17:56:03    
       gNameForAJob11
510002 ShortName12      Cluster1  2/24.68%   0/26.90%      2013-10-06 17:37:43

switch>

show monitor hadoop cluster all

The show monitor hadoop cluster all command displays configuration and connection information for all monitored Hadoop clusters.

Command Mode

EXEC

Command Syntax

show monitor hadoop cluster all

Example
This command displays configuration and connection data for all connected Hadoop clusters.
switch> show monitor hadoop cluster all

Total number of clusters configured: 3
  Cluster                     : Cluster0
  Admin status                : Enabled
  JobTracker host             : host0
  JobTracker RPC port         : 9000
  JobTracker user             : user0
  JobTracker polling interval : 100 seconds
  TaskTracker HTTP port       : 8800
  Operational status          : Enabled
  Active TaskTrackers         : 31
  Blacklisted TaskTrackers    : 1
  Decommissioned TaskTrackers : 1
  Tracker expiry interval     : 20.0
  Map slots (used/total)      : 10/100
  Reduce slots (used/total)   : 11/110
  JobTracker heap size        : 1.04GB (max: 2.08GB)

  Cluster                     : Cluster1
  Admin status                : Enabled
  JobTracker host             : host1
  JobTracker RPC port         : 9001
  JobTracker user             : user1
  JobTracker polling interval : 101 seconds
  TaskTracker HTTP port       : 8801
  Operational status          : Enabled
  Active TaskTrackers         : 32
  Blacklisted TaskTrackers    : 0
  Decommissioned TaskTrackers : 0
  Tracker expiry interval     : 40.0
  Map slots (used/total)      : 20/200
  Reduce slots (used/total)   : 22/220
  JobTracker heap size        : 2.09GB (max: 4.15GB)

  Cluster                     : Cluster2
  Admin status                : Disabled
  JobTracker host             : host2
  JobTracker RPC port         : 9002
  JobTracker user             : user2
  JobTracker polling interval : 102 seconds
  TaskTracker HTTP port       : 8802
  Operational status          : Disabled

show monitor hadoop cluster counters

The show monitor hadoop cluster counters command displays a list of jobs running on the specified Hadoop cluster and data consumption associated with these jobs.

Command Mode

EXEC

Command Syntax

show monitor hadoop cluster c_name counters

Parameters

c_name     Cluster name.

Example
This command displays jobs running on cluster Cluster0.
switch> show monitor hadoop cluster Cluster0 counters

Last updated: 2013-10-06 18:14:23
Counters for currently running jobs on cluster: Cluster0
JobId   Job Name          User Bytes In Bytes   Out Start  Time
------- ----------------- ---------- ---------- ----------- -------------------
2       ShortName2        JobUser2   37.36GB    76.29MB     2013-10-06 17:37:43
1       ReallyAVeryLon\   JobUser1   37.36GB    76.29MB     2013-10-06 17:56:03
        gNameForAJob1

switch>

show monitor hadoop cluster history

The show monitor hadoop cluster history command displays all jobs that ran on the specified cluster. The list includes all jobs that ran since the switch was reloaded, the job history was cleared (clear monitor hadoop job-history), or MapReduce Tracer was enabled.

Command Mode

EXEC

Command Syntax

show monitor hadoop cluster c_name history

Parameters

c_name      Cluster name.

Example
This command displays the jobs that were ran on the cluster named Cluster0.
switch> show monitor hadoop cluster Cluster0 history

Jobs history on cluster: Cluster0
JobId    Job Name           Start Time    End Time       Bytes In     Bytes Out
-------- ------------------ ------------- -------------- ------------ ---------
2        AReallyBigHist\    2013-10-06    2013-10-09     26.08GB      13.04GB        
         oricalJobName      17:41:03      06:47:43
2        AReallyBigHist\    2013-10-06    2013-10-09     26.08GB      13.04GB
         oricalJobName      17:41:03      06:47:43
1        HistoryJob1        2013-10-06    2013-10-08     26.08GB      13.04GB
                            17:57:43      00:31:03

switch>

show monitor hadoop cluster history jobs

The show monitor hadoop cluster history jobs command displays data about the specified job. Hadoop jobs are identified by job number and the cluster that ran the job.

Data that the command returns include job identifiers, JobTracker ID, start and stop times, and data consumption.

Command Mode

EXEC

Command Syntax

show monitor hadoop cluster c_name history jobs job_number

Parameters
  • c_name     Cluster name.
  • job_number     Job number. Value ranges from 0 to 2147483647.
Example
This command displays information about job 1 that ran on cluster Cluster0.
switch> show monitor hadoop cluster Cluster0 history job 1

Job history data for job: HistoryJob1
  Cluster               : Cluster0
  Job Id                : 1
  JT Id                 : 201310110013   
  User                  : HistoryUser1
  Job start time        : 2013-10-06 17:57:43
  Job end time          : 2013-10-08 00:31:03

Per Interface job counters:
Interface       TaskTracker        Bytes In       Bytes Out
--------------- ------------------ -------------- ---------
Ethernet7       TaskTracker2       26.08GB        13.04GB

switch>

show monitor hadoop cluster jobs

The show monitor hadoop cluster jobs command displays a list of jobs that are running on the specified cluster.

Command Mode

EXEC

Command Syntax

show monitor hadoop cluster c_name jobs

Parameters

c_name      Cluster name.

Example
This command displays the list of jobs running on cluster Cluster0.
switch> show monitor hadoop cluster Cluster0 jobs

Last updated: 2013-10-06 18:14:23
Currently running jobs on cluster: Cluster0
JobId    Job Name           User        Maps    Reduces     Start Time
-------- ------------------ ----------- ------- ----------- -------------------
2        ShortName2         JobUser2    2       0           2013-10-06 17:37:43
1        ReallyAVeryLon\    JobUser1    2       0           2013-10-06 17:56:03
         gNameForAJob1

switch>

show monitor hadoop cluster jobs counter

The show monitor hadoop cluster jobs counter command displays data consumption and progress statistics for the specified job.

Command Mode

EXEC

Command Syntax

show monitor hadoop cluster c_name [jobs job_number] counter

Parameters
  • c_name     Cluster name.
  • job_number      Job number. Value ranges from 0 to 2147483647.
Example
This command displays byte counters for the job named 1 that is running on the cluster named Cluster0.
switch> show monitor hadoop cluster Cluster0 jobs 1 counters

Last updated: 2013-10-06 18:14:23
Cluster : Cluster0
Job Name : ReallyAVeryLongNameForAJob1
Job Id : 1
Interface         HDFS Bytes Read    HDFS Bytes Written    Reduce Shuffle Bytes
----------------- ------------------ --------------------- --------------------
Port-Channel8     4.14GB             8.48MB                12.72MB
Port-Channel9     6.21GB             12.72MB               19.07MB
Ethernet8         3.10GB             6.36MB                9.54MB
Ethernet9         5.17GB             10.60MB               15.89MB
Port-Channel7     2.07GB             4.24MB                6.36MB
Ethernet10        7.24GB             14.83MB               22.25MB
Port-Channel10    8.28GB             16.95MB               25.43MB
Ethernet7         1.03GB             2.12MB                3.18MB

switch>

show monitor hadoop cluster jobs <job number>

The show monitor hadoop cluster jobsjob number command displays information about the specified job. Hadoop jobs are identified by job ID and the cluster that is running the job.

Data that the command returns include time of update, job identifiers, start times, data consumption, and completion progress.

Command Mode

EXEC

Command Syntax

show monitor hadoop cluster c_name [jobs job_number]

Parameters
  • c_name      Cluster name.
  • job_number      Job number. Value ranges from 0 to 2147483647.
Example
This command displays information about job 1 that is running on cluster Cluster0.
switch> show monitor hadoop cluster Cluster0 jobs 1

Last updated: 2013-10-06 18:14:23
Information for job: ReallyAVeryLongNameForAJob1 running on cluster: Cluster0  
  Cluster : Cluster0
  Id                     : 1
  Name                   : ReallyAVeryLongNameForAJob1
  User                   : JobUser1
  Priority               : veryHigh
  Running state          : running
  Number of map tasks    : 2
  Number of reduce tasks : 0
  Start time             : 2013-10-06 17:56:03
  Bytes In               : 37.36GB
  Bytes Out              : 76.29MB
  Map Progress           : 12.34%
  Reduce Progress        : 13.45% 
  Cleanup Progress       : 14.56%
  Setup Progress         : 15.67%

switch>

show monitor hadoop cluster status

The show monitor hadoop cluster status command displays configuration and connection information for the specified cluster.

Command Mode

EXEC

Command Syntax

show monitor hadoop cluster c_name status

Parameters

c_name      Cluster name.

Example
This command displays configuration and connection data for the Cluster0 cluster.
switch> show monitor hadoop cluster Cluster0 status

Last updated: 2013-10-06 18:14:23
Cluster status for cluster: Cluster0
  Admin status                  : Enabled  
  JobTracker host               : host0
  JobTracker RPC port           : 9000
  JobTracker user               : user0
  JobTracker polling interval   : 100 seconds
  TaskTracker HTTP port         : 8800  
  Operational status            : Enabled
  Active TaskTrackers           : 31 
  Blacklisted TaskTrackers      : 1 
  Decommissioned TaskTrackers   : 1
  Tracker expiry interval       : 20.0
  Map slots (used/total)        : 10/100
  Reduce slots (used/total)     : 11/110
  JobTracker heap size          : 1.04GB (max: 2.08GB)

switch>

show monitor hadoop cluster tasktracker

The show monitor hadoop cluster tasktracker command displays a list of TaskTrackers in the specified cluster. The IP address and access interface is included in the table.

Command Mode

EXEC

Command Syntax

show monitor hadoop cluster c_name tasktracker

Parameters

c_name     Cluster name.

Example
This command displays the TaskTrackers on the Cluster0 cluster.
switch> show monitor hadoop cluster Cluster0 tasktracker

Last updated: 2013-10-06 18:14:23
Total 2 TaskTrackers on cluster Cluster0:
Node          IP Address   Interface      Maps   Reduces
------------- ------------ -------------- ------ -------
TaskTracker1   10.100.0.1   Ethernet7      4      0
TaskTracker2   10.100.0.2   Port-Channel7  4      0

switch>

show monitor hadoop counters

The show monitor hadoop counters command displays byte counter data for all jobs running on clusters for which MapReduce Tracer is configured.

Command Mode

EXEC

Command Syntax

show monitor hadoop counters

Example
This command displays byte counter data for all jobs running on clusters that the switch is accessing through MapReduce Tracer.
switch> show monitor hadoop counters

Last updated: 2013-10-06 18:14:23
Counters for running jobs:
JobId   Job Name          Cluster    Bytes In   Bytes Out   Start Time
------- ----------------- ---------- ---------- ----------- -------------------
510002  ShortName12       Cluster1   37.36GB    76.29MB     2013-10-06 17:37:43
510001  ReallyAVeryLon\   Cluster1   37.36GB    76.29MB     2013-10-06 17:56:03      
        gNameForAJob11
2       ShortName2        Cluster0   37.36GB    76.29MB     2013-10-06 17:37:43
1       ReallyAVeryLon\   Cluster0   37.36GB    76.29MB     2013-10-06 17:56:03    
        gNameForAJob1

switch>

show monitor hadoop history

The show monitor hadoop history command displays jobs that ran on clusters for which MapReduce Tracer is configured. The list includes all jobs that ran since the switch was reloaded, MapReduce Tracer was enabled, or the job history was cleared (clear monitor hadoop job-history).

Command Mode

EXEC

Command Syntax

show monitor hadoop history

Example
This command displays data that jobs that previously ran on connected Hadoop clusters.
switch> show monitor hadoop history

Job history for all clusters:
JobId  Job Name         Cluster   Start Time  End Time     Bytes In   Bytes Out
------ ---------------- --------- ----------- ------------ ---------- ---------
2      AReallyBigHist\  Cluster0  2013-10-06  2013-10-09   26.08GB    13.04GB
       oricalJobName              17:41:03    06:47:43
442    AReallyBigHist\  Cluster1  2013-10-06  2013-10-09   26.08GB    13.04GB       
       oricalJobName              17:41:03    06:47:43
442    AReallyBigHist\  Cluster1  2013-10-06  2013-10-09   26.08GB    13.04GB
       oricalJobName              17:41:03    06:47:43
2      AReallyBigHist\  Cluster0  2013-10-06  2013-10-09   26.08GB    13.04GB
       oricalJobName              17:41:03    06:47:43
441    HistoryJob1      Cluster1  2013-10-06  2013-10-08   26.08GB    13.04GB
                                  17:57:43    00:31:03
1      HistoryJob1      Cluster0  2013-10-06  2013-10-08   26.08GB    13.04GB
                                  17:57:43    00:31:03

switch>

show monitor hadoop status

The show monitor hadoop status command displays system status for MapReduce Tracer.

Command Mode

EXEC

Command Syntax

show monitor hadoop status

Example
This command displays MapReduce Tracer status for all connected clusters and TaskTrackers.
switch> show monitor hadoop status
Last updated: 2013-10-06 18:14:23
Mapreduce Tracer status:  
  Admin status                   : Enabled
  Operational status             : Enabled  
  Number of clusters configured  : 3
  Number of local TaskTrackers   : 4
  Number of jobs running locally : 4

switch>

show monitor hadoop tasktracker all

The show monitor hadoop tasktracker all command displays a list of TaskTrackers that are on all monitored Hadoop clusters.

Command Mode

EXEC

Command Syntax

show monitor hadoop tasktracker all

Example
This command displays the TaskTrackers of all monitored clusters that are connected to the switch.
switch> show monitor hadoop tasktracker all

Last updated: 2013-10-06 18:14:23
All local TaskTrackers:
Node             Cluster      IP Address     Interface         Maps     Reduces
---------------- ------------ -------------- ----------------- -------- -------
TaskTracker1     Cluster0     10.100.0.1      Ethernet7         4        0
TaskTracker3     Cluster1     10.100.0.3      Ethernet8         4        0
TaskTracker2     Cluster0     10.100.0.2      Port-Channel7     4        0
TaskTracker4     Cluster1     10.100.0.4      Port-Channel8     4        0

switch>

show monitor hadoop tasktracker all counters

The show monitor hadoop tasktracker all counters command displays byte counters for the TaskTrackers of all monitored Hadoop clusters.

Command Mode

EXEC

Command Syntax

show monitor hadoop tasktracker all counters

Example
This command displays byte counter data for the TaskTrackers servicing all MapReduce Tracer Hadoop clusters.
switch> show monitor hadoop tasktracker all counters

Last updated: 2013-10-06 18:14:23
Counters for all TaskTrackers:
Node             IP Address     Interface         Bytes Read      Bytes Written
---------------- -------------- ----------------- --------------- -------------
TaskTracker1     10.100.0.1     Ethernet7         2.08GB          4.24MB
TaskTracker3     10.100.0.3     Ethernet8         6.23GB          12.72MB
TaskTracker2     10.100.0.2     Port-Channel7     4.15GB          8.48MB
TaskTracker4     10.100.0.4     Port-Channel8     8.30GB          16.95MB

switch>

show monitor hadoop tasktracker counters

The show monitor hadoop tasktracker counters command displays a list of jobs running on the specified TaskTracker and output from byte counters associated with these jobs.

Command Mode

EXEC

Command Syntax

show monitor hadoop tasktracker NODES counters

Parameters
  • NODES     TaskTracker node access point. Options include:
    • host hostname      Node name.
    • interface ethernet e_range      Ethernet interfaces through which node connects.
    • interface port-channel p_range      Port channel interfaces through which node connects.
Examples
    • This command displays the jobs running on the TaskTracker on the TaskTracker1 node.
      switch> show monitor hadoop tasktracker host TaskTracker1 counters
      
      Last updated: 2013-10-06 18:14:23
      Running job for TaskTracker: TaskTracker1
      JobId   Job Name          Cluster    Bytes In   Bytes Out   Start Time
      ------- ----------------- ---------- ---------- ----------- -------------------
      2       ShortName2        Cluster0   37.36GB    76.29MB     2013-10-06 17:37:43
      1       ReallyAVeryLon\   Cluster0   37.36GB    76.29MB     2013-10-06 17:56:03
              gNameForAJob1
      
      Note: these counters are derived from Hadoop counters and represent approximate
      network bandwidth utilization
      
      switch>
    • This command displays jobs running on TaskTrackers accessed through Ethernet interfaces 7 and 8.
      switch> show monitor hadoop tasktracker interface Ethernet 7,8 counters
      
      Last updated: 2013-10-06 18:14:23
      Running job for TaskTracker: TaskTracker3
      JobId   Job Name          Cluster    Bytes In   Bytes Out   Start Time
      ------- ----------------- ---------- ---------- ----------- -------------------
      510002  ShortName12       Cluster1   37.36GB    76.29MB     2013-10-06 17:37:43
      510001  ReallyAVeryLon\   Cluster1   37.36GB    76.29MB     2013-10-06 17:56:03
              gNameForAJob11
      
      Note: These counters are derived from Hadoop counters and represent approximate
      network bandwidth utilization
      
      Running job for TaskTracker: TaskTracker1
      JobId   Job Name          Cluster    Bytes In   Bytes Out   Start Time
      ------- ----------------- ---------- ---------- ----------- -------------------
      2       ShortName2        Cluster0   37.36GB    76.29MB     2013-10-06 17:37:43
      1       ReallyAVeryLon\   Cluster0   37.36GB    76.29MB     2013-10-06 17:56:03
              gNameForAJob1
      
      Note: these counters are derived from Hadoop counters and represent approximate
      network bandwidth utilization
      
      switch>
    • This command displays jobs running on TaskTrackers accessed through port channel interface 7.
      switch> show monitor hadoop tasktracker interface Port-Channel 7 counters
      
      Last updated: 2013-10-06 18:14:23
      Running job for TaskTracker: TaskTracker2
      JobId   Job Name          Cluster    Bytes In   Bytes Out   Start Time
      ------- ----------------- ---------- ---------- ----------- -------------------
      2       ShortName2        Cluster0   37.36GB    76.29MB     2013-10-06 17:37:43
      1       ReallyAVeryLon\   Cluster0   37.36GB    76.29MB     2013-10-06 17:56:03
              gNameForAJob1
      
      Note: these counters are derived from Hadoop counters and represent approximate
      network bandwidth utilization
      
      switch>

show monitor hadoop tasktracker jobs

The show monitor hadoop tasktracker jobs command displays data about the jobs that are running on TaskTrackers located on the specified node or accessed through the listed interfaces.

Including a cluster parameter filters results to include data only from the cluster polled by the specified monitor.

Command Mode

EXEC

Command Syntax

show monitor hadoop tasktracker NODES jobs [CLUSTERS]

Parameters
  • NODES     TaskTracker node access point. Options include:
    • host hostname     Node name.
    • interface ethernet e_range      Ethernet interfaces through which node connects.
    • interface port-channel p_range     Port channel interfaces through which node connects.
  • CLUSTERS     Hadoop cluster for which command displays data. Options include:
    • no parameter     TaskTracker on specified NODE can be in any cluster.
    • cluster c_name     TaskCluster on specified NODE must be in named cluster.
Examples
  • This command displays data for jobs running on TaskTracker1.
    switch> show monitor hadoop tasktracker host TaskTracker1 jobs
    
    Last updated: 2013-10-06 18:14:23
    Running job for TaskTracker: TaskTracker1
    JobId  Job Name         Cluster   Maps(#/%)  Reduces(#/%)   Start Time
    ------ ---------------- --------- ---------- -------------- -------------------
    1      ReallyAVeryLon\  Cluster0  2/12.34%   0/13.45%       2013-10-06 17:56:03
           gNameForAJob1
    2      ShortName2       Cluster0  2/24.68%   0/26.90%       2013-10-06 17:37:43
    
    switch>
  • This command displays data for jobs on TaskTrackers accessed through Ethernet interfaces 7 and 8.
    switch> show monitor hadoop tasktracker interface Ethernet 7,8 jobs
    
    Last updated: 2013-10-06 18:14:23
    Running job for TaskTracker: TaskTracker3
    JobId   Job Name         Cluster   Maps(#/%)  Reduces(#/%)  Start Time
    ------- ---------------- --------- ---------- ------------- -------------------
    510001  ReallyAVeryLon\  Cluster1  2/12.34%   0/13.45%      2013-10-06 17:56:03
            gNameForAJob11
    510002  ShortName12      Cluster1  2/24.68%   0/26.90%      2013-10-06 17:37:43
    
    Running job for TaskTracker: TaskTracker1
    JobId  Job Name         Cluster   Maps(#/%)  Reduces(#/%)   Start Time
    ------ ---------------- --------- ---------- -------------- -------------------
    1      ReallyAVeryLon\  Cluster0  2/12.34%   0/13.45%       2013-10-06 17:56:03
           gNameForAJob1
    2      ShortName2       Cluster0  2/24.68%   0/26.90%       2013-10-06 17:37:43
    
    switch>
  • This command displays data for jobs on TaskTrackers accessed through port channel interface 7.
    switch> show monitor hadoop tasktracker interface Port-Channel 7 jobs
    
    Last updated: 2013-10-06 18:14:23
    Running job for TaskTracker: TaskTracker2
    JobId  Job Name         Cluster   Maps(#/%)  Reduces(#/%)   Start Time
    ------ ---------------- --------- ---------- -------------- -------------------
    1      ReallyAVeryLon\  Cluster0  2/12.34%   0/13.45%       2013-10-06 17:56:03
           gNameForAJob1
    2      ShortName2       Cluster0  2/24.68%   0/26.90%       2013-10-06 17:37:43
    
    switch>
  • This command displays data for jobs on TaskTrackers on the Cluster0 cluster that are accessed through Ethernet interfaces 7 and 8.
    switch> show monitor hadoop tasktracker interface Ethernet 7,8 jobs cluster Cluster0
    
    Last updated: 2013-10-06 18:14:23
    Running job for TaskTracker: TaskTracker1
    JobId  Job Name         Cluster   Maps(#/%)  Reduces(#/%)   Start Time
    ------ ---------------- --------- ---------- -------------- -------------------
    1      ReallyAVeryLon\  Cluster0  2/12.34%   0/13.45%       2013-10-06 17:56:03
           gNameForAJob1
    2      ShortName2       Cluster0  2/24.68%   0/26.90%       2013-10-06 17:37:43
    
    switch>
  • This command displays data for jobs on TaskTracker named TaskTracker1 on the Cluster0 cluster.
    switch> show monitor hadoop tasktracker host TaskTracker1 jobs cluster Cluster0
    
    Last updated: 2013-10-06 18:14:23
    Running job for TaskTracker: TaskTracker1
    JobId  Job Name         Cluster   Maps(#/%)  Reduces(#/%)   Start Time
    ------ ---------------- --------- ---------- -------------- -------------------
    1      ReallyAVeryLon\  Cluster0  2/12.34%   0/13.45%       2013-10-06 17:56:03
           gNameForAJob1
    2      ShortName2       Cluster0  2/24.68%   0/26.90%       2013-10-06 17:37:43
    
    switch>
  • This command displays data for jobs on TaskTrackers on the Cluster0 cluster that are accessed through port channel interface 7.
    switch> show monitor hadoop tasktracker interface Port-Channel 7 jobs cluster Cluster0
    
    Last updated: 2013-10-06 18:14:23
    Running job for TaskTracker: TaskTracker2
    JobId  Job Name         Cluster   Maps(#/%)  Reduces(#/%)   Start Time
    ------ ---------------- --------- ---------- -------------- -------------------
    1      ReallyAVeryLon\  Cluster0  2/12.34%   0/13.45%       2013-10-06 17:56:03
           gNameForAJob1
    2      ShortName2       Cluster0  2/24.68%   0/26.90%       2013-10-06 17:37:43
    
    switch>

show monitor hadoop tasktracker running-tasks

The show monitor hadoop tasktracker running-tasks command displays progress and byte counts of tasks executed by TaskTrackers located on the specified node or accessed through the listed interfaces.

Including a cluster-ID parameter filters results to include data only from the specified cluster.

Command Mode

EXEC

Command Syntax

show monitor hadoop tasktracker NODES running-tasks [CLUSTERS][JOBS]

Parameters
  • NODES     TaskTracker node access point. Options include:
    • host hostname     Node name.
    • interface ethernet e_range      Ethernet interfaces through which node connects.
    • interface port-channel p_range     Port channel interfaces through which node connects.
  • CLUSTERS     Hadoop cluster for which command displays data. Options include:
    • no parameter     TaskTracker on specified NODE can be in any cluster.
    • cluster c_name     TaskCluster on specified NODE must be in named cluster.
  • JOBS     Job list. Options include:
    • no parameter     all jobs.
    • job 0 to 2147483647      Specifies number of single job.
Examples
  • This command displays data for tasks running on TaskTracker named TaskTracker1.
    switch> show monitor hadoop tasktracker host TaskTracker1 running-tasks
    
    Last updated: 2013-10-06 18:14:23
    Running tasks for TaskTracker: TaskTracker1 on interface Ethernet7
    JobId TaskId  Cluster   Type  Progress  Status   HDFS Read  HDFS Write  Shuffle
    ----- ------- --------- ----- --------- -------- ---------- ----------- -------
    1     2       Cluster0  Map   33.33%    running  2.10MB     2.14MB      2.96MB
    2     2       Cluster0  Map   33.33%    running  2.10MB     2.14MB      2.96MB
    1     1       Cluster0  Map   50.00%    running  1.05MB     1.07MB      1.48MB
    2     1       Cluster0  Map   50.00%    running  1.05MB     1.07MB      1.48MB
    
    switch>
  • This command displays data for tasks running on the TaskTracker named TaskTracker1 of the Cluster0 cluster.
    switch> show monitor hadoop tasktracker host TaskTracker1 running-tasks cluster
    Cluster0
    
    Last updated: 2013-10-06 18:14:23
    Running tasks for TaskTracker: TaskTracker1 on interface Ethernet7
    JobId TaskId  Cluster   Type  Progress  Status   HDFS Read  HDFS Write  Shuffle
    ----- ------- --------- ----- --------- -------- ---------- ----------- -------
    1     2       Cluster0  Map   33.33%    running  2.10MB     2.14MB      2.96MB
    2     2       Cluster0  Map   33.33%    running  2.10MB     2.14MB      2.96MB
    1     1       Cluster0  Map   50.00%    running  1.05MB     1.07MB      1.48MB
    2     1       Cluster0  Map   50.00%    running  1.05MB     1.07MB      1.48MB
    
    switch>
  • This command displays data for tasks running for job 1 on the TaskTracker named TaskTracker1 of the Cluster0 cluster.
    switch> show monitor hadoop tasktracker host TaskTracker1 running-tasks cluster
    Cluster0 job 1
    
    Last updated: 2013-10-06 18:14:23
    Running tasks for TaskTracker: TaskTracker1 on interface Ethernet7
    JobId TaskId  Cluster   Type  Progress  Status   HDFS Read  HDFS Write  Shuffle
    ----- ------- --------- ----- --------- -------- ---------- ----------- -------
    1     2       Cluster0  Map   33.33%    running  2.10MB     2.14MB      2.96MB
    1     1       Cluster0  Map   50.00%    running  1.05MB     1.07MB      1.48MB
    
    switch>
  • This command displays data for tasks running on TaskTrackers accessed through Ethernet interfaces 7 and 8.
    switch> show monitor hadoop tasktracker interface Ethernet 7,8 running-tasks
    
    Last updated: 2013-10-06 18:14:23
    Running tasks for TaskTracker: TaskTracker1 on interface Ethernet7
    JobId TaskId  Cluster   Type  Progress  Status   HDFS Read  HDFS Write  Shuffle
    ----- ------- --------- ----- --------- -------- ---------- ----------- -------
    2     2       Cluster0  Map   33.33%    running  2.10MB     2.14MB      2.96MB
    1     1       Cluster0  Map   50.00%    running  1.05MB     1.07MB      1.48MB
    2     1       Cluster0  Map   50.00%    running  1.05MB     1.07MB      1.48MB
    
    Running tasks for TaskTracker: TaskTracker3 on interface Ethernet8
    JobId  TaskId  Cluster  Type  Progress  Status   HDFS Read  HDFS Write  Shuffle
    ------ ------ --------- ----- --------- -------- ---------- ----------- -------
    510002 222    Cluster1  Map   33.33%    running  2.10MB     2.14MB      2.96MB
    510001 112    Cluster1  Map   33.33%    running  2.10MB     2.14MB      2.96MB
    510002 221    Cluster1  Map   50.00%    running  1.05MB     1.07MB      1.48MB
    510001 111    Cluster1  Map   50.00%    running  1.05MB     1.07MB      1.48MB
    
    switch>
  • This command displays data for tasks running on TaskTrackers of Cluster0 cluster that are accessed through Ethernet interfaces 7 and 8.
    switch> show monitor hadoop tasktracker interface Ethernet 7,8 running-tasks
    cluster Cluster0
    
    Last updated: 2013-10-06 18:14:23
    Running tasks for TaskTracker: TaskTracker1 on interface Ethernet7
    JobId TaskId  Cluster   Type  Progress  Status   HDFS Read  HDFS Write  Shuffle
    ----- ------- --------- ----- --------- -------- ---------- ----------- -------
    1     2      Cluster0   Map   33.33%    running  2.10MB     2.14MB      2.96MB
    2     2      Cluster0   Map   33.33%    running  2.10MB     2.14MB      2.96MB
    1     1      Cluster0   Map   50.00%    running  1.05MB     1.07MB      1.48MB
    2     1      Cluster0   Map   50.00%    running  1.05MB     1.07MB      1.48MB
    
    switch>
  • This command displays data for tasks running for job 1 on the TaskTrackers of Cluster0 cluster that are accessed through Ethernet interfaces 7 and 8
    switch> show monitor hadoop tasktracker interface Ethernet 7,8 running-tasks
    cluster Cluster0 job 1
    
    Last updated: 2013-10-06 18:14:23
    Running tasks for TaskTracker: TaskTracker1 on interface Ethernet7
    JobId TaskId  Cluster   Type  Progress  Status   HDFS Read  HDFS Write  Shuffle
    ----- ------- --------- ----- --------- -------- ---------- ----------- -------
    1     2       Cluster0  Map   33.33%    running  2.10MB     2.14MB      2.96MB
    1     1       Cluster0  Map   50.00%    running  1.05MB     1.07MB      1.48MB
    
    switch>
  • This command displays data for tasks running on TaskTrackers accessed through port channe l interfaces 7 and 8.
    switch> show monitor hadoop tasktracker interface Port-Channel 7-8 running-tasks
    
    Last updated: 2013-10-06 18:14:23
    Running tasks for TaskTracker: TaskTracker2 on interface Port-Channel7
    JobId TaskId  Cluster   Type  Progress  Status   HDFS Read  HDFS Write  Shuffle
    ----- ------- --------- ----- --------- -------- ---------- ----------- -------
    1     2       Cluster0  Map   33.33%    running  2.10MB     2.14MB      2.96MB
    1     1       Cluster0  Map   50.00%    running  1.05MB     1.07MB      1.48MB
    2     1       Cluster0  Map   50.00%    running  1.05MB     1.07MB      1.48MB
    
    Running tasks for TaskTracker: TaskTracker4 on interface Port-Channel8
    JobId  TaskId Cluster   Type  Progress  Status   HDFS Read  HDFS Write  Shuffle
    ------ ------ --------- ----- --------- -------- ---------- ----------- -------
    510002 222    Cluster1  Map   33.33%    running  2.10MB     2.14MB      2.96MB
    510001 112    Cluster1  Map   33.33%    running  2.10MB     2.14MB      2.96MB
    510001 111    Cluster1  Map   50.00%    running  1.05MB     1.07MB      1.48MB
    
    switch>
  • This command displays data for tasks running on TaskTrackers of Cluster0 cluster accessed through port channel interface 7.
    switch> show monitor hadoop tasktracker interface Port-Channel 7 running-tasks
    cluster Cluster0
    
    Last updated: 2013-10-06 18:14:23
    Running tasks for TaskTracker: TaskTracker2 on interface Port-Channel7
    JobId TaskId  Cluster   Type  Progress  Status   HDFS Read  HDFS Write  Shuffle
    ----- ------- --------- ----- --------- -------- ---------- ----------- -------
    1     2       Cluster0  Map   33.33%    running  2.10MB     2.14MB      2.96MB
    1     1       Cluster0  Map   50.00%    running  1.05MB     1.07MB      1.48MB
    2     1       Cluster0  Map   50.00%    running  1.05MB     1.07MB      1.48MB
    
    switch>
  • This command displays data for job 510001 running on TaskTrackers of Cluster1 cluster that are accessed through port channel interface 8.
    switch> show monitor hadoop tasktracker interface Port-Channel 8 running-tasks
    cluster Cluster1 job 510001
    
    Last updated: 2013-10-06 18:14:23
    Running tasks for TaskTracker: TaskTracker4 on interface Port-Channel8
    JobId  TaskId Cluster   Type  Progress  Status   HDFS Read  HDFS Write  Shuffle
    ------ ------ --------- ----- --------- -------- ---------- ----------- -------
    510001 112    Cluster1  Map   33.33%    running  2.10MB     2.14MB      2.96MB
    510001 111    Cluster1  Map   50.00%    running  1.05MB     1.07MB      1.48MB
    
    switch>

show monitor hadoop tasktracker running-tasks cluster job task

The show monitor hadoop tasktracker running-tasks cluster job task command displays detailed data for the specified task.

Command Mode

EXEC

Command Syntax

show monitor hadoop tasktracker NODE [running-tasks cluster name][ job jnum ][task tnum]

Parameters
  • NODE     TaskTracker node access point. Options include:
    • host hostname     Node name.
    • interface ethernet e_range      Ethernet interfaces through which node connects.
    • interface port-channel p_range     Port channel interfaces through which node connects.
  • name     Cluster name.
  • jnum     Job number. Value ranges from 0 to 2147483647.
  • tnum     Task number. Value ranges from 0 to 2147483647.
Examples
  • This command displays data for task 1 of job 1 on TaskTracker1 of Cluster0.
    switch> show monitor hadoop tasktracker host TaskTracker1 running-tasks cluster Cluster0 job 1 task 1
    
    Last updated: 2013-10-06 18:14:23
    Task details for one task as given below: 
      TaskTracker name     : TaskTracker1
      Interface            : 'Ethernet7'
      Cluster              : Cluster0
      Job Id               : 1
      Task Id              : 1
      Attempt Id           : 0
      Task type            : Map
      Status               : running
      State                : running  
      Start time           : 2013-10-06 17:57:43
      Progress             : 50.00%  
      HDFS bytes read      : 1.05MB
      HDFS bytes written   : 1.07MB
      Reduce shuffle bytes : 1.48MB
    
    switch>
  • This command displays data for task 1 of job 1 on the Cluster0 TaskTracker that is accessible through Ethernet interface 7.
    switch> show monitor hadoop tasktracker interface Ethernet 7 running-tasks cluster Cluster0 job 1 task 1
    
    Last updated: 2013-10-06 18:14:23
    Task details for one task as given below:  
      TaskTracker name     : TaskTracker1
      Interface            : 'Ethernet7'
      Cluster              : Cluster0
      Job Id               : 1
      Task Id              : 1
      Attempt Id           : 0   
      Task type            : Map
      Status               : running
      State                : running
      Start time           : 2013-10-06 17:57:43
      Progress             : 50.00%
      HDFS bytes read      : 1.05MB
      HDFS bytes written   : 1.07MB
      Reduce shuffle bytes : 1.48MB
    
    switch>
  • This command displays data for task 111 of job 510001 on the Cluster0 TaskTracker that is accessible through port channel interface 8.
    switch> show monitor hadoop tasktracker interface Port-Channel 8 running-tasks cluster Cluster1 job 510001 task 111
    
    Last updated: 2013-10-06 18:14:23
    Task details for one task as given below:
      TaskTracker name     : TaskTracker4
      Interface            : 'Port-Channel8'
      Cluster              : Cluster1
      Job Id               : 510001
      Task Id              : 111  
      Attempt Id           : 0
      Task type            : Map  
      Status               : running
      State                : running
      Start time           : 2013-10-06 17:57:43
      Progress             : 50.00%
      HDFS bytes read      : 1.05MB
      HDFS bytes written   : 1.07MB
      Reduce shuffle bytes : 1.48MB
    
    switch>

show monitor hadoop tasktracker status

The show monitor hadoop tasktracker status command displays connection and activity information for the TaskTracker on the specified clusters or accessed through the specified interface. The following command formats display the listed TaskTracker information:
  • show monitor hadoop cluster c_name tasktracker status: TaskTrackers on specified cluster.
  • show monitor hadoop tasktracker node status: TaskTrackers on specified nodes or interfaces.
  • show monitor hadoop tasktracker all status: all connected TaskTrackers.

Command Mode

EXEC

Command Syntax

show monitor hadoop cluster c_name tasktracker status

show monitor hadoop tasktracker NODES status

show monitor hadoop tasktracker all status

Parameters
  • c_name     Cluster name.
  • NODES TaskTracker node access point. Options include:
    • host hostname      Node name.
    • interface ethernet e_range       Ethernet interfaces through which node connects.
    • interface port-channel p_range      Port channel interfaces through which node connects.
Examples
  • This command displays connection and activity information for all TaskTrackers connected through Ethernet interfaces 7 and 8.
    switch> show monitor hadoop tasktracker interface Ethernet7,8 status
    
    Last updated: 2013-10-06 18:14:23
      TaskTracker            : TaskTracker1
      IP Address             : 10.100.0.1
      Interface              : Ethernet7 
      State                  : active
      Running jobs           : 2
      Running tasks          : 4
      Map Tasks              : 4
      Reduce Tasks           : 0
      Total bytes read       : 2.08GB
      Total bytes written    : 4.24MB
      TaskTracker            : TaskTracker3
      IP Address             : 10.100.0.3
      Interface              : Ethernet8
      State                  : active
      Running jobs           : 2
      Running tasks          : 4
      Map Tasks              : 4
      Reduce Tasks           : 0
      Total bytes read       : 6.23GB
      Total bytes written    : 12.72MB
    
    switch>
  • This command displays connection and activity information for all connected TaskTrackers.
    switch> show monitor hadoop tasktracker all status
    
    Last updated: 2013-10-06 18:14:23
    All local TaskTrackers:
      TaskTracker            : TaskTracker4
      IP Address             : 10.100.0.4
      Interface              : Port-Channel8 
      State                  : active
      Running jobs           : 2
      Running tasks          : 4
      Map Tasks              : 4
      Reduce Tasks           : 0
      Total bytes read       : 8.30GB
      Total bytes written    : 16.95MB 
    
      TaskTracker            : TaskTracker3
      IP Address             : 10.100.0.3
      Interface              : Ethernet8  
      State                  : active
      Running jobs           : 2
      Running tasks          : 4
      Map Tasks              : 4
      Reduce Tasks           : 0
      Total bytes read       : 6.23GB
      Total bytes written    : 12.72MB
    
      TaskTracker            : TaskTracker2
      IP Address             : 10.100.0.2
      Interface              : Port-Channel7
      State                  : active
      Running jobs           : 2
      Running tasks          : 4
      Map Tasks              : 4
      Reduce Tasks           : 0
      Total bytes read       : 4.15GB
      Total bytes written    : 8.48MB
    
      TaskTracker            : TaskTracker1
      IP Address             : 10.100.0.1
      Interface              : Ethernet7
      State                  : active
      Running jobs           : 2
      Running tasks          : 4  
      Map Tasks              : 4
      Reduce Tasks           : 0
      Total bytes read       : 2.08GB
      Total bytes written    : 4.24MB
    
    switch>
  • This command displays connection and activity data for TaskTracker on the TaskTracker1 node.
    switch> show monitor hadoop tasktracker host TaskTracker1 status
    
    Last updated: 2013-10-06 18:14:23
      TaskTracker            : TaskTracker1
      IP Address             : 10.100.0.1
      Interface              : Ethernet7  
      State                  : active
      Running jobs           : 2
      Running tasks          : 4
      Map Tasks              : 4
      Reduce Tasks           : 0
      Total bytes read       : 2.08GB
      Total bytes written    : 4.24MB
    
    switch>
  • This command displays connection and activity data for all TaskTracker connected through Port Channel 7.
    switch> show monitor hadoop tasktracker interface Port-Channel 7 status
    
    Last updated: 2013-10-06 18:14:23
      TaskTracker            : TaskTracker2  
      IP Address             : 10.100.0.2
      Interface              : Port-Channel7
      State                  : active  
      Running jobs           : 2
      Running tasks          : 4    
      Map Tasks              : 4
      Reduce Tasks           : 0
      Total bytes read       : 4.15GB
      Total bytes written    : 8.48MB
    
    switch>

show monitor hadoop traffic burst

The show monitor hadoop traffic burst command displays the largest data bursts for specified Hadoop cluster jobs. A data burst is the data consumed during a polling interval. The command displays input and output burst:
  • Input bursts include bytes written to the host.
  • Output bursts include bytes written by the host.

Command Mode

EXEC

Command Syntax

show monitor hadoop [CLUSTERS] traffic burst [NODE]

Parameters
  • CLUSTERS Hadoop clusters for which command displays data. Options include:
    • no parameter      All clusters.
    • cluster c_name     Cluster name.
  • NODES TaskTracker node access point. Options include:
    • host hostname      Node name.
    • interface ethernet e_range       Ethernet interfaces through which node connects.
    • interface port-channel p_range      Port channel interfaces through which node connects.
Examples
  • This command displays traffic burst data for all running jobs.
    switch> show monitor hadoop traffic burst
    Last updated: 2013-10-06 18:14:23
    Bursts on Interface: 'Ethernet7' in cluster: Cluster0
    Top 2 input bursts:
    JobId  Job Name         Burst    Time
    ------ ---------------- -------- -------------------
    1      ShortName         3.07GB  2013-10-06 17:57:43
    2      ReallyAVeryLon\   6.15GB  2013-10-06 17:41:03
           gNameForAJob
    
    Top 2 output bursts:
    JobId  Job Name          Burst    Time
    ------ ---------------- -------- -------------------
    1       ShortName         4.10GB  2013-10-06 17:55:13
    2       ReallyAVeryLon\   8.20GB  2013-10-06 17:36:03   
            gNameForAJob
    
    Bursts on Interface: 'Port-Channel7' in cluster: Cluster0
    Top 2 input bursts:
    JobId  Job Name         Burst    Time
    ------ ---------------- -------- -------------------
    1       ShortName        3.07GB  2013-10-06 17:57:43
    2       ReallyAVeryLon\  6.15GB  2013-10-06 17:41:03 
            gNameForAJob
    
    Top 2 output bursts:
    JobId  Job Name         Burst    Time
    ------ ---------------- -------- -------------------
    1       ShortName        4.10GB  2013-10-06 17:55:13
    2       ReallyAVeryLon\  8.20GB  2013-10-06 17:36:03
            gNameForAJob
    
    Bursts on Interface: 'Ethernet8' in cluster: Cluster1
    Top 4 input bursts:
    JobId   Job Name         Burst    Time
    ------- ---------------- -------- -------------------
    510001   ShortName        3.07GB  2013-10-06 17:57:43
    510002   ReallyAVeryLon\  6.15GB  2013-10-06 17:41:03
             gNameForAJob
    510003   ShortName        9.22GB   2013-10-06 17:24:23
    510004   ReallyAVeryLon\  12.29GB  2013-10-06 17:07:43
             gNameForAJob
    
    Top 4 output bursts:
    JobId  Job Name          Burst    Time
    ------ ----------------- -------- -------------------
    510001  ShortName         4.10GB  2013-10-06 17:55:13
    510002  ReallyAVeryLon\   8.20GB  2013-10-06 17:36:03
            gNameForAJob
    510003  ShortName         12.29GB  2013-10-06 17:16:53
    510004  ReallyAVeryLon\   16.39GB  2013-10-06 16:57:43
            gNameForAJob
    
    Bursts on Interface: 'Port-Channel8' in cluster: Cluster1
    Top 4 input bursts:
    JobId  Job Name         Burst     Time
    ------ ---------------- --------- -------------------
    3101
    510001  ShortName        3.07GB    2013-10-06 17:57:43
    510002  ReallyAVeryLon\  6.15GB    2013-10-06 17:41:03
            gNameForAJob
    510003  ShortName         9.22GB    2013-10-06 17:24:23
    510004  ReallyAVeryLon\   12.29GB   2013-10-06 17:07:43
            gNameForAJob
    
    Top 4 output bursts:
    JobId  Job Name         Burst    Time
    ------ ---------------- -------- -------------------
    510001  ShortName        4.10GB  2013-10-06 17:55:13
    510002  ReallyAVeryLon\  8.20GB  2013-10-06 17:36:03
            gNameForAJob
    510003  ShortName        12.29GB  2013-10-06 17:16:53
    510004  ReallyAVeryLon\  16.39GB  2013-10-06 16:57:43
            gNameForAJob
    
    switch>
  • This command displays traffic burst for all jobs running on TaskTrackers that are accessible through Ethernet interfaces 7 and 8.
    switch> show monitor hadoop traffic burst interface Ethernet 7,8
    
    Last updated: 2013-10-06 18:14:23
    Bursts on Interface: 'Ethernet7' in cluster: Cluster0
    Top 2 input bursts:
    JobId  Job Name        Burst  Time
    ------ --------------- ------ -------------------
    1      ShortName       3.07GB 2013-10-06 17:57:43
    2      ReallyAVeryLon\ 6.15GB 2013-10-06 17:41:03
           gNameForAJob
    
    Top 2 output bursts:
    JobId  Job Name        Burst   Time
    ------ --------------- ------- -------------------
    1      ShortName        4.10GB 2013-10-06 17:55:13
    2      ReallyAVeryLon\  8.20GB 2013-10-06 17:36:03
           gNameForAJob
    
    Bursts on Interface: 'Ethernet8' in cluster: Cluster1
    Top 4 input bursts:
    JobId  Job Name         Burst    Time
    ------ ---------------- -------- -------------------
    510001  ShortName       3.07GB   2013-10-06 17:57:43
    510002  ReallyAVeryLon\ 6.15GB   2013-10-06 17:41:03
            gNameForAJob
    510003  ShortName        9.22GB   2013-10-06 17:24:23
    510004  ReallyAVeryLon\  12.29GB  2013-10-06 17:07:43
            gNameForAJob
    
    Top 4 output bursts:
    JobId  Job Name        Burst    Time
    ------ --------------- -------- -------------------
    510001 ShortName        4.10GB  2013-10-06 17:55:13
    510002 ReallyAVeryLon\  8.20GB  2013-10-06 17:36:03 
           gNameForAJob
    510003 ShortName        12.29GB  2013-10-06 17:16:53
    510004 ReallyAVeryLon\  16.39GB  2013-10-06 16:57:43
           gNameForAJob
    
    switch>
  • This command displays traffic burst data for all running jobs that are accessible through port channel interface 7.
    switch> show monitor hadoop traffic burst interface Port-Channel 7
    
    Last updated: 2013-10-06 18:14:23
    Bursts on Interface: 'Port-Channel7' in cluster: Cluster0
    Top 2 input bursts:
    JobId  Job Name        Burst   Time
    ------ --------------- ------- -------------------
    1      ShortName       3.07GB  2013-10-06 17:57:43
    2      ReallyAVeryLon\ 6.15GB  2013-10-06 17:41:03
           gNameForAJob
    
    Top 2 output bursts:
    JobId  Job Name        Burst   Time
    ------ --------------- ------- -------------------
    1      ShortName       4.10GB  2013-10-06 17:55:13
    2      ReallyAVeryLon\ 8.20GB  2013-10-06 17:36:03    
           gNameForAJob

show monitor system

The show monitor system command displays information about the memory usage status of the system including when low-memory situations have led to Out-Of-Memory (OOM) events.

Command Mode

EXEC

Command Syntax

show monitor system

Examples
  • The following output is for a system where the feature is disabled and the system has never entered the low-memory mode.
    Memory Exhaustion Feature enabled: False
    System currently in Memory Exhaustion: False
    Number of times system entered Memory Exhaustion: 0
  • The following output is for a system where the feature is enabled and the system has never entered the low-memory mode.

    Memory Exhaustion Feature enabled: True
    System currently in Memory Exhaustion: False
    Number of times system entered Memory Exhaustion: 0
  • The following output is for a system where the feature is enabled and the system enters the low-memory mode.

    Memory Exhaustion Feature enabled: True
    System currently in Memory Exhaustion: True
    Number of times system entered Memory Exhaustion: 1
    Last time entered in Memory Exhaustion: 0:00:07 ago
  • The following output is for a system where the feature is enabled and the system enters the low-memory mode once, and has exited the low-memory mode.

    Memory Exhaustion Feature enabled: True
    System currently in Memory Exhaustion: False
    Last time entered in Memory Exhaustion: 0:01:00 ago
    Last time exited from Memory Exhaustion: 0:00:05 ago

shutdown (Monitor-Hadoop)

The shutdown command globally disables MapReduce Tracer on the switch. Enabling MapReduce Tracer for an individual cluster requires the feature to be globally enabled through this command and enabled for the individual cluster thorough the shutdown (Monitor Hadoop Cluster) command. By default, MapReduce Tracer is globally disabled.

The no shutdown command globally enables MapReduce Tracer. The shutdown and default shutdown commands globally disable MapReduce Tracer by removing the corresponding no shutdown command from running-config.

Command Mode

Monitor-hadoop Configuration

Command Syntax

shutdown

no shutdown

default shutdown

Related Commands

The monitor hadoop command places the switch in the monitor-hadoop configuration mode.

Examples
  • These commands globally enable MapReduce Tracer.
    switch(config)# monitor hadoop
    switch(config-monitor-hadoop)# no shutdown
    switch(config-monitor-hadoop)# show active
     monitor hadoop
       no shutdown
    switch(config-monitor-hadoop)#
  • This command globally disables MapReduce Tracer.
    switch(config-monitor-hadoop)# shutdown
    switch(config-monitor-hadoop)# show active
    switch(config-monitor-hadoop)#

shutdown (Monitor Hadoop Cluster)

The shutdown command disables MapReduce Tracer for the configuration mode cluster. Globally disabling MapReduce Tracer (show monitor hadoop) also disables the function on the individual cluster. Enabling MapReduce Tracer for the cluster requires the function to be enabled globally and for the individual cluster.

The no shutdown command configures the MapReduce Tracer setting as enabled for the configuration mode cluster. The shutdown and default shutdown commands disable MapReduce Tracer for the cluster by removing the corresponding no shutdown command from running-config.

Command Mode

Monitor-hadoop-cluster Configuration

Command Syntax

shutdown

no shutdown

default shutdown

Related Commands

The cluster (Monitor Hadoop) command places the switch in the monitor-hadoop-cluster configuration mode.

Examples
  • These commands globally enable MapReduce Tracer, then enables it for the CL2 cluster.
    switch(config)# monitor hadoop
    switch(config-monitor-hadoop)# no shutdown
    switch(config-monitor-hadoop)# cluster CL2
    switch(config-monitor-hadoop-CL2)# no shutdown
    switch(config-monitor-hadoop-CL2)# show active
    monitor hadoop
       cluster CL2
          no shutdown
    switch(config-monitor-hadoop-CL2)# exit
    switch(config-monitor-hadoop)# show active
     monitor hadoop
       no shutdown
       cluster CL2
          no shutdown
    switch(config-monitor-hadoop)#
  • These commands disable MapReduce Tracer for the CL2 cluster. MapReduce Tracer remains globally enabled.
    switch(config-monitor-hadoop)# cluster CL2
    switch(config-monitor-hadoop-CL2)# shutdown
    switch(config-monitor-hadoop-CL2)# show active
    monitor hadoop
       cluster CL2
    switch(config-monitor-hadoop-CL2)# exit
    switch(config-monitor-hadoop)# show active
     monitor hadoop
       no shutdown
       cluster CL2
    switch(config-monitor-hadoop)#

tasktracker (Monitor Hadoop Cluster)

The tasktracker command specifies the HTTP port for accessing TaskTrackers of the Hadoop cluster monitored through configuration mode statements. The switch compiles a list of the cluster’s TaskTracker addresses by periodically polling the cluster’s JobTracker (jobtracker (Monitor Hadoop Cluster)). The default TaskTracker HTTP port is 50060.

The no tasktracker and default tasktracker commands restore the configuration mode TaskTracker HTTP port to 50060 by removing the corresponding tasktracker command from running-config.

Command Mode

Monitor-hadoop-cluster Configuration

Command Syntax

tasktracker http-port port_number

no tasktracker http-port

default tasktracker http-port

Parameters

port_num      TaskTracker HTTP port number. Value ranges from 1 to 65535. Default value is 50060.

Related Command

The cluster (Monitor Hadoop) command places the switch in the monitor-hadoop-cluster configuration mode.

Examples
  • These commands specify a TaskTracker HTTP port address of 51000.
    switch(config)# monitor hadoop
    switch(config-monitor-hadoop)# cluster CL2
    switch(config-monitor-hadoop-CL2)# tasktracker http-port 51000
    switch(config-monitor-hadoop-CL2)# show active
    monitor hadoop
       cluster CL2
          tasktracker http-port 51000
    switch(config-monitor-hadoop-CL2)#
  • These commands restore the default TaskTracker HTTP port address of 50060.
    switch(config-monitor-hadoop-CL2)# no tasktracker http-port
    switch(config-monitor-hadoop-CL2)# show active
    monitor hadoop
       cluster CL2
    switch(config-monitor-hadoop-CL2)# show active all
    monitor hadoop
       cluster CL2
          jobtracker rpc-port 8021
          tasktracker http-port 50060
          interval 10
          shutdown
    switch(config-monitor-hadoop-CL2)#