=======================

### Monitoring Python Script

========================

Refer: https://docs.aerospike.com/pages/viewpage.action?pageId=3309766 

The Aerospike command line tool called asmonitor (installed in /opt/aerospike/bin/asmonitor) gets the server/cluster statistics and presents it in an easy to read format,it also allows you to dynamically change the configuration parameters for the cluster or particular nodes. It also allows you to customize the console to display different values.

By default, without a pre-existing ~/.asmonitor/asmonitor.conf file, the asmonitor command will try to connect to 127.0.0.1:3000. For all practical purposes you will not need to edit the configuration file, as you can directly give the seed host or Comma separated for multi hosts using -h parameter, If the host does not connect you would see the following error, Make sure the Aerospike server is running and responding on port 3000

```
$ asmonitor 

Enter help for help

request to  127.0.0.1 : 3000  returned error
***error* Failed to connect to any hosts.
          Please edit the ~/.asmonitor/asmonitor.conf file with the correct hosts parameter in the [main] section or provide with the -h parameter
```

-------------------------
User may use the -h param to provide the ip:port info during the initial asmonitor invocation.

### Features:

* Display summary of cluster in tabular format ( typing info will display  Defaults: Node,Namespace and XDR section if enabled)
* Create Custom tables in Node, Namespace, Sets or XDR sections, to monitor sections of interest.
* Compare server config for the entire cluster, Prints the values which are different, Does not compare Namespace config
* Edit custom or default pre-created tables, to delete or replace columns.Delete tables if needed.
* Monitor latency statistics for the entire cluster or some nodes.
* Sort Default or custom tables on any column in the table in ascending or descending.
* Look for specific statistics or configuration value from the entire cluster or some nodes.
* To get usage information for the asmonitor script enter help will drop into the console:
* Run **watch** on commands used by asmonitor

```
Monitor> help

Aerospike Interactive Shell
Version 2.1.3

Commands:

	add
	 -h <comma separated hostip:port list>
	 [-p default port. 3000 assumed if missing]
	Add a single or list of hostip:ports to the current list of monitored nodes

	clinfo
	 -v <Value to get or set>
	 [-h <comma separated hostip:port list>]
	 [-p default port. 3000 assumed if missing]
	Sets configuration for the values for the entire cluster, unless the -h parameter is given.

	compareconfig
	 [-h <comma separated hostip:port list>]
	 [-p default port. 3000 assumed if missing]
	Compare server config for the entire cluster, Prints the values which are different, Does not compare Namespace config

	createtable
	Create custom tables in Node,Namespace,Sets or XDR sections.

	edittable
	Edit default or custom tables (can only delete or replace columns)

	exit
	Exit the monitor console

	info [<TABLENAME>]
	 [-h comma separated hostip:port list]
	 [-p default port. 3000 assumed if missing]
	 [  to see all table names]
	 [ info TABLENAME (Autocomplete by tab) ]
	Gets cluster information in tabular format, Show custom tables or defaults 
	 Default tables are Node,Namespace and XDR

	latency
	 [-v <histogram name filter>]
	 [-h <comma separated hostip:port list>]
	 [-p default port. 3000 assumed if missing]
	 [-k histogram key (reads, writes_master, writes, writes_reply, proxy)]
	 [-d duraction_sec]
	 [-v value spec for filtering with ? and * wildcard expressions]
	 [-b number of seconds (before now) to look back to]
	 [-d duration, the number of seconds from start to search]
	 [-s slice_sec, i.e. interval in sec to analyze]
	 [-t flag (want throughput). If present, display the throughtputs data instead]
	 [-m flag (by machine). If present, display the output group by machine names also]
	 [-c flag (config). If present, run the histogram configuration step]
	Get the latency histogram data

	looptest
	 [-h comma separated hostip:port list]
	 [-p default port. 3000 assumed if missing]
	 [-v number of loops for the test. Default is 1000]
	run a loop test agains the node with the service command (EXPERIMENTAL)

	printconfig
	 [-h <comma separated hostip:port list>]
	 [-p default port. 3000 assumed if missing]
	 [-v value spec for filtering with ? and * wildcard expressions]
	Print out server config info (EXPERIMENTAL)

	remove
	 -h <comma separated hostip:port list>
	 [-p default port. 3000 assumed if missing]
	Remove a single or list of hostip:ports from the current list of monitored nodes

	shell command_to_os
	Execute shell commands from asmonitor and display output (EXPERIMENTAL)

	sorttable
	Sort a table by a column, Ascending or descending (color will be disabled)

	stat [-v <Value to filter>]
	 [-h <comma separated hostip:port list>]
	 [-p default port. 3000 assumed if missing]
	 [-v value spec for filtering with ? and * wildcard expressions]
	Get the statistics data of the entire cluster, unless -h parameter is given
```

### Usage:

The -u param or any param that the asmonitor does not understand will kick off the usage message:

```
$ asmonitor -u

Enter help for help

Usage:
 -c monitor.conf	   Optional. Location of configuration file. Default is ~/.asmonitor/asmonitor.conf
 -e 'Command'		  Optional. eg1: '-e info' execute the info command in the shell. eg2: '-e help' show the shell help text
 -h hostip:port		Optional. Connection info of the host(s). It can be in the 127.0.0.1 or 127.0.0.1:3000 format. Comma separated for multi hosts
 -p default_port	   Optional. Default host port number. If missing, 3000 is assumed
 -u					Print out this usage text

 The '[main]' section in configuration file (default is ~/.asmonitor/asmonitor.conf, auto generated after the first asmonitor invocation):
	  hosts = Seed-HostIP:port or List of HostIP:ports (comma separated, each in format of 127.0.0.1:3000)
	  namespaces = List of namespaces (auto generated. user should not edit)
	  crawl = True Find all nodes in cluster- Works only with internal IPs
	  xdr = False. Disabled by default. To enable'xdr=True'
	  xdrport=xdr monitor port
```


### How To

Note the configuration file is auto updated after each run
1. Running asmonitor will drop you into a monitor console which will allow you run various commands, if you do not want to drop into a shell and directly want to run a command you can use the -e option i.e. asmonitor -e [command] e.g. asmonitor -e "info Node" will directly display Node statistics without dropping in the console.
2. To see a list of commands allowed by the asmonitor console enter , As asmonitor automatically inherit bash-like history-list editing (e.g. Control-P scrolls back to the last command, Control-N forward to the next one, Control-F moves the cursor to the right non-destructively, Control-B moves the cursor to the left non-destructively,Autocomplete commands whereever available, Control-R to search history etc.) Special characters like ? will show help and ! will send shell commands to the os.
3. There are 4 pre-created tables which you could call with info command, i.e. Node,Namespace,SETS and XDR
4. If you just enter info then default tables are displayed or you could run info
for a particular table, info should give you all tables asmonitor can display.
5. You can edit/delete a table if you wish to do so. i.e. if a default table or custom table does not column you want you can replace or delete that column
6. If you want to restore all tables to default, enter restoredefault command
7. If you want to create new tables createtable and follow on screen commands
	i. If you are creating tables to statistics about sets or see sets summary (like number of sets in a namespace and set names, You need to create a new table) , Default info SETS does not show set names and number of sets
	ii To create a custom table to see sets summary, for the 1st column key Only values allowed are: mcnamesets or mcnamens , use mcnamens if you want to see Sets Summary (i.e. To see setnames and Number of sets) use mcnamesets if you want to see Set details like objects in sets etc., Default info SETS uses mcnamesets
9. To see latency statistics for the entire cluster enter latency, if latency tracking is not enabled on servers then enter latency -c which will allow you to configure latency configuration on the servers.
	i. Default latency will show all the histograms, i.e reads,writes, proxy and writes_reply.
	ii. If you only want to see some histograms for the entire cluster, enter latency -k reads to see reads for the entire cluster,
	iii.  There are many knobs you can use to customize it, enter latency -u to see usage.
10. To change any dynamic settings for the entire cluster enter clinfo -v [commands from dynamic configuration page] , You can also get/set values on some nodes by entering -h parameter or the -p port values if setting for XDR.
11. If you need to grep some statistics quickly for the entire cluster you can use stat -v *err* ,
this command allows wildcard expressions.
12. If you need to get some config values from the entire cluster , printconfig -v *nsup* , which would show all defrag config values from the cluster. You can also specify the -h host or -p port for the cluster.
13. Enter help to see list of commands and options available.

Here is an example of running asmonitor on a one node cluster:

```
$ asmonitor -h 127.0.0.1:3000

Enter help for help

1 host names are discovered in the cluster
Citrusleaf Interactive Shell, version 2.1.3

Monitor>
Monitor> info
===NODES===
2012-12-21 13:20:10.454209
ip:port                 Build   Cluster      Cluster   Free   Free   Migrates              Node   Replicated    Sys
                            .      Size   Visibility   Disk    Mem          .                id      Objects   Free
                            .         .            .    pct    pct          .                 .            .    Mem
192.168.105.168:3000    2.1.3         1         true      0     99      (0,0)   BB93E0303CA0568      0.100 M     98
Number of nodes displayed: 1

 ===NAMESPACE===
Total(unique) objects in cluster for usermap : 0.100 M

ip/namespace               Avail   Evicted   Objects     Repl     Stop   Used   Used     Used   Used    hwm   hwm
                             Pct   Objects         .   Factor   Writes   Disk   Disk      Mem    Mem   Disk   Mem
                               .         .         .        .        .      .      %        .      %      .     .
192.168.105.168/usermap       99         0   0.100 M        1    false    n/a    n/a   6.10 M      1     95    95
Number of rows displayed: 1

 ===XDR===
Nodes                   Build      Bytes   Free     Lag           Req    Req.        Req.   Uptime
                            .    Shipped   dlog    secs   Outstanding   Relog     Shipped        .
192.168.105.168:3004    2.1.3   256.73 M    18%       0          2300   1,416   4,016,552   77,977
Node of XDR responding: 1
```
