asbackup/asrestore

(1) Taking a backup
Usage: ./backup [options]
-h <host> [default 127.0.0.1]
-p <port> [default 3000]
-n <namespace> (required)
-s <set> [default *all*]
-d <directory> (required)
-f <scan priority> [0=auto;1=low;2=med;3=high default 0]
-c fail on cluster change
-v verbose
-x nobin [backup without bin data]
-w <max parallel degree> [number of nodes backed up in parallel, default 10]
-l <list of nodes> [input string containing host:port names separated by semicolon in double quotes ]
-% <scan percentage> [percentage of backup from each partition, default 100] 
--usage show usage

The 'asbackup' program requires at least two parameters: 
the namespace to backup, and the local backup directory to populate.
One can also optionally specify the host, port , set , max parallel degree, list of nodes.


The directory will be created/cleaned before use, and the backup process will 
create many '.clb' ('Aerospike Backup') files:

	(I) backup of a cluster 
		asbackup -h 127.0.0.1 -p 3000 -d backup_10_23_2011 -n test
	(II) backup of list of nodes from a cluster
		asbackup -l "host1:port1;host2:port2" -d backup_01_23_2013 -n test 

Backup will concurrently query all nodes in the cluster at the same time.
On each node, data is backed up on a partition-by-partition basis. If an 
object is created/updated, and the partition it belongs to has already 
been processed, the create/update will not be picked up by backup.

When a cluster changes (for example, a node goes down), data will be 
rebalancing between the nodes. If a backup is running while data rebalances,
it is possible to get the same object backed-up twice, or have some data 
that may be missed. To have enforce strict data uniqueness, one can specify 
the "-c" to fail on cluster change. 

"Scan priority" controls the level of concurrency on a node. Increasing this
will increase the speed of the backup, but should be done while monitoring
normal read/write performance to make sure performance is not affected. 

"nobin" controls the backup information about object. If it is set then backup will
store only digest information of an object otherwise it stores both digest and bin data.

If -w option is set, backup will concurrently query for that many nodes.
Once backup for one node is done immediately another node will added for backup.

"list_nodes" takes user input for list of nodes for which backup will run.
Format of this input is- string containing host:port names separated by semicolon in double quotes.
With -l option no need to provide -h and -p option. 1st host and port from list_nodes is used
to connect to the cluster.

Before starting backup it is good to check disk space availability based on used-bytes-disk(or memory).
Recommended- disk-space-available >  used-bytes-disk(or memory)/repl-factor 

(2) Doing a restore
Usage: ./restore [options]
-h <host> [default 127.0.0.1]
-p <port> [default 3000]
-t <threads> [default 40]
-n <namespace> [default namespace is in backup files]
-d <directory> (required)
-u insert only missing records (existing records not updated)
-v verbose
--usage show usage

The 'restore' program is very similar, and pushes the contents of the backup 
directory to the cluster.

Here is how you can restore all data in the system from a previously taken 
backup (asrestore is part of the citrusleaf-tools install package):

	asrestore -h 127.0.0.1 -p 3000 -d backup_10_23_2011

The restore program is intelligent enough to not restore data which has been 
modified in the cluster since the backup was taken. This avoids the scenario where 
a more recent update to a data item is lost due to a restore from an earlier copy.
It will also skip over data which is already expired.

An example ticker output from the restore:

Sep 12 2012 22:57:15 GMT: expired 4333 : attempted 3563832 : [updated 1354433 not-updated (existed 0 gen-old 2100432)]

This indicate a total of 3563832+4333 records has been processed from the *.clb files.
-- 4333 of the records are ignored because they have expired
-- 3563832 was attempted to write back to the cluster. 
-- Of those, 1354433 are updated successfully, 2100432 did not update 
due to generation mismatch. (ie, server already has same or later version)
-- If the "-u" option was used, the "existed" field may be non-zero. This accounts
for number of objects not updated becuase record already exists on server.

(3) File format
The file format was created for ease of parsing within the C language. Liberal use 
is made of the Base64 encoding.

Record lines
Record lines start with a '+' and identify the record uniquely.
'+ n test' identifies the namespace
'+ d BASE64STRING' identifies the RIPE160 hash, encoded in Base64
'+ g integer-value' identifies the current generation count
'+ b integer-value' identifies the number of bins to follow

Bin lines
Bin lines start with '-' and contain all the bin values in a given record, 
one per record. The format of each line is:

 '- T binname [type specific fields] value'

Where T is a type byte, and binname is a UTF-8 string containing the bin's name. 
Some Types use a type-specific field, which is often the length of the value itself.
 
type		byte	bin specific value	meaning
Null		N	No Values               Null bins should never be backed up, as they do not exist
String		S	Length field            UTF-8 encoded string, possibly including line breaks and nulls
Integer	I	No-values                   ASCII integer representation, currently always unsigned
Blob		B	Length field            Base64 encoded binary value. The length is the length of the Base64 encoding.
Java Blob	J	Length field            Base64 encoded binary value. Intended to be serialized to a java object.
Python Blob	P	Length field            Base64 encoded binary value. Intended to be serialized to a python object.
Ruby Blob	R	Length field            Base64 encoded binary value. Intended to be serialized to a Ruby object.

