Multidimensional Profiler (mprofiler)
=====================================
Author: Ignacio Laguna
Contact: ilagunap@gmail.com

This python program collects multiple performance measurements from a running 
program (i.e. a Linux process) and save them in a file. It collects a variety of
metrics such as CPU and memory related metrics, I/O, scheduling and network 
metrics. Measurements are saved in different files for each process ID, i.e., a 
file for the parent process and files for all the child processes (that the 
parent process creates).

The collected measurements can be used to locate performance bugs using 
statistical methods.

How to run it
-------------
For help on running the script, execute:

$ ./mprofiler.py --help
Usage: mprofiler.py [options] [program]

Options:
  -h, --help            show this help message and exit
  -s RATE, --sampling-rate=RATE
                        Sampling rate (seconds). Default is 5 seconds.
  -d DIR, --dir=DIR     Output directory.
  -n NUM, --samples-number=NUM
                        Number of samples. Default is 10.
  -p PID, --pids=PID    Process IDs. Use comma to separate different PIDs.
                        Example: -p 26,22
  -i NIC, --nic=NIC     Network interface card (NIC). Default is 'eth0'.

Operation modes
---------------
There are two operation modes:
(a)  Using an already existing process (or a comma-separated set of processes) 
with the '-p' option.
(b)  Creating the program to monitor (by passing the program as an argument)

Mode (a) has precedence over (b), i.e. if '-p' is used, a program passed as
an argument will not be created (and it will only monitor the processes with the
PID passed with '-p'). If '-p' is not used, you must pass the program to monitor
as an argument. See the example below for mode (b).

About options
-------------
The sampling rate should be a minimum of 1 (second); the default is 5 seconds.
If the output directory is not specified, the script will save measurements by
default in the 'output' directory in the current path; this directory will be
created if it does not exist. The number of samples will depend on the amount of
time your program runs and the sampling rate. If the run time not known in 
advance, simply specify a large number of samples; the script should end when 
the main program terminates.

Output
------
The script will save measurements in files with the following name:
[timestamp]_[pid]_[hostname].dat
where 'timestamp' is the current time and 'pid' is the process id of the
executed program. If the program creates child processes, their measurements
will be saved in different files (with different pid).

Example
-------
In the following example we use the profiler to get measurements of the 'find'
program in Linux:

$ ./mprofiler.py -s 1 -n 1000 "find /usr -name data.txt"
[PROFILER]: Calling main program...
[PROFILER]: Main program has PID 3484
......
[PROFILER]: Main program is done.
[PROFILER]: Saving data in files...

Now we list the saved files:

$ ls ./output/
1338240428_3484.dat

Then we can see some of the saved data:

$ head ./output/1338240428_3484.dat 
Timestamp,PID,minflt,majflt,utime,stime,num_threads,vsize,rss,processor,
stack_size,rchar,wchar,read_bytes,write_bytes,canceled_write_bytes,
num_file_desc,nicRcvBytes,nicRcvPckts,nicSentBytes,nicSentPckts,
IPInTruncatedPkts,IPInOctets,IPOutOctets,
1345568734,6341,519,0,0,0,1,114860032,242,2,140488002908144,9027,0,0,0,0,6,
67765628,75127,5919958,29581,0,66161975,5682077,
1345568735,6341,735,0,5,6,1,114913280,309,2,140488003109355,9027,108,933888,0,0,
7,67765930,75129,5919958,29581,0,66162249,5682077,
1345568736,6341,760,0,7,12,1,115015680,324,2,140488003109355,9027,108,3342336,0,
0,7,67766736,75133,5923862,29586,0,66162999,5685911,


Notice that data is comma separated and that the first row corresponds to the
name of the performance metrics.

Metric descriptions
-------------------
Timestamp:  timestamp of the measurement
PID: process id
Program: program name (as it is stored in the /proc/PID/stat file)
minflt: minor page faults
majflt: major page faults
utime: user-level CPU time
stime: system-level CPU time
num_threads: number of threads
vsize: virtual memory size
rss: RAM memory
processor: CPU number last executed on
stack_size: size of the stack
rchar: read characters
wchar: written characters
read_bytes: read bytes
write_bytes: written bytes
canceled_write_bytes: canceled written bytes
num_file_desc: number of open file descriptors (including sockets)
nicRcvBytes: received bytes from NIC
nicRcvPckts: received packets from NIC
nicSentBytes: sent bytes from NIC
nicSentPckts: sent packets from NIC
IPInTruncatedPkts: truncated IP packets
IPInOctets: received IP octets
IPOutOctets: sent IP octets

Bugs
----
Please report bugs to ilagunap@gmail.com