distribution

Short, simple, direct scripts for creating character-based graphs in a command terminal. Status: stable. Features added very rarely.

Purpose

To generate graphs directly in the (ASCII-based) terminal. Most common use-case: if you type

long | list | of | commands | sort | uniq -c | sort -rn

| sort | uniq -c | sort -rn

| distribution

The tool is mis-named. It was originally for generating histograms (a distribution of the frequency of input tokens) but it has since been expanded to generate time-series graphs (or, in fact, graphs with any arbitrary "x-axis") as well.

At first, there will be only two scripts, the originals written in Perl and Python by Tim Ellis. Any other versions people are willing to create will be placed here. The next likely candidate language is C++.

There are a few typical use cases for graphs in a terminal as we'll lay out here:

Tokenize and Graph

A stream of ASCII bytes, tokenize it, tally the matching tokens, and graph the result. For this example, assume "file" is a list of words with one word per line, so passing it to "xargs" makes it all-one-line.

$ cat file | xargs #put all words on one line
this is an arbitrary stream of tokens. this will be graphed with tokens pulled out. this is the first use case.
$ cat file | xargs | distribution --tokenize=word --match=word --size=small -v
tokens/lines examined: 25
 tokens/lines matched: 21
       histogram keys: 17
              runtime: 8.00ms
      Key|Ct (Pct)    Histogram
     this|3 (14.29%) -----------------------------------------------------------------------o
       is|2  (9.52%) ---------------------------------------o
   tokens|2  (9.52%) ---------------------------------------o
  graphed|1  (4.76%) -------o
     will|1  (4.76%) -------o

Aggregate and Graph

An already-tokenised input, one-per-line, tally and graph them.

$ cat file | distribution -s=small -v
tokens/lines examined: 21
 tokens/lines matched: 21
       histogram keys: 18
              runtime: 14.00ms
      Key|Ct (Pct)    Histogram
     this|3 (14.29%) -----------------------------------------------------------------------o
       is|2  (9.52%) ---------------------------------------o
  graphed|1  (4.76%) -------o
       be|1  (4.76%) -------o

Graph Already-Aggregated/Counted Tokens

A list of tallies + tokens, one-per-line. Create a graph with labels. This matches the typical output of several Unix commands such as "du."

$ du -s /etc/*/ 2>/dev/null | distribution -g -v
tokens/lines examined: 107
 tokens/lines matched: 5,176
       histogram keys: 107
              runtime: 2.00ms
                    Key|Ct  (Pct)    Histogram
              /etc/ssl/|920 (17.77%) -------------------------------------------
           /etc/init.d/|396  (7.65%) -------------------
              /etc/apt/|284  (5.49%) -------------
   /etc/nagios-plugins/|224  (4.33%) -----------
              /etc/cis/|188  (3.63%) ---------
           /etc/nagios/|180  (3.48%) ---------
            /etc/fonts/|172  (3.32%) --------
              /etc/ssh/|172  (3.32%) --------
          /etc/default/|164  (3.17%) --------
    /etc/console-setup/|132  (2.55%) -------

Graph a List of Integers

A list of tallies only. Create a graph without labels. This is typical if you just have a stream of numbers and wonder what they look like. The

--numonly

There is a different project: https://github.com/holman/spark that will produce simpler, more-compact graphs. By contrast, this project will produce rather lengthy and verbose graphs with far more resolution, which you may prefer.

Features

1. Configurable colourised output.
2. rcfile for your own preferred default commandline options.
3. Full Perl tokenising and regexp matching.
4. Partial-width Unicode characters for high-resolution charts.
5. Configurable chart sizes including "fill up my whole screen."

Installation

If you use homebrew,

brew install distribution

To put the script into your homedir on the machine you plan to run the script:

$ wget https://raw.githubusercontent.com/philovivero/distribution/master/distribution.py
$ sudo mv distribution.py /usr/local/bin/distribution
$ alias worddist="distribution -t=word"

It is fine to place the script anywhere in your

$PATH

ls -alR | worddist

Options

  --keys=K       periodically prune hash to K keys (default 5000)
  --char=C       character(s) to use for histogram character, some substitutions follow:
        pl       Use 1/3-width unicode partial lines to simulate 3x actual terminal width
        pb       Use 1/8-width unicode partial blocks to simulate 8x actual terminal width
        ba       (▬) Bar
        bl       (Ξ) Building
        em       (—) Emdash
        me       (⋯) Mid-Elipses
        di       (♦) Diamond
        dt       (•) Dot
        sq       (□) Square
  --color        colourise the output
  --graph[=G]    input is already key/value pairs. vk is default:
        kv       input is ordered key then value
        vk       input is ordered value then key
  --height=N     height of histogram, headers non-inclusive, overrides --size
  --help         get help
  --logarithmic  logarithmic graph
  --match=RE     only match lines (or tokens) that match this regexp, some substitutions follow:
        word     ^[A-Z,a-z]+$ - tokens/lines must be entirely alphabetic
        num      ^\d+$        - tokens/lines must be entirely numeric
  --numonly[=N]  input is numerics, simply graph values without labels
        actual   input is just values (default - abs, absolute are synonymous to actual)
        diff     input monotonically-increasing, graph differences (of 2nd and later values)
  --palette=P    comma-separated list of ANSI colour values for portions of the output
                 in this order: regular, key, count, percent, graph. implies --color.
  --rcfile=F     use this rcfile instead of $HOME/.distributionrc - must be first argument!
  --size=S       size of histogram, can abbreviate to single character, overridden by --width/--height
        small    40x10
        medium   80x20
        large    120x30
        full     terminal width x terminal height (approximately)
  --tokenize=RE  split input on regexp RE and make histogram of all resulting tokens
        word     [^\w] - split on non-word characters like colons, brackets, commas, etc
        white    \s    - split on whitespace
  --width=N      width of the histogram report, N characters, overrides --size
  --verbose      be verbose

Syslog Analysis Example

You can grab out parts of your syslog ask the script to tokenize on non-word delimiters, then only match words. The verbosity gives you some stats as it works and right before it prints the histogram.

$ zcat /var/log/syslog*gz \
    | awk '{print $5" "$6}' \
    | distribution --tokenize=word --match=word --height=10 --verbose --char=o
tokens/lines examined: 16,645    
 tokens/lines matched: 5,843
       histogram keys: 92
              runtime: 10.75ms
Val       |Ct (Pct)      Histogram
ntop      |1818 (31.11%) ooooooooooooooooooooooooooooooooooooooooooooooooooooooo
WARNING   |1619 (27.71%) ooooooooooooooooooooooooooooooooooooooooooooooooo
kernel    |1146 (19.61%) ooooooooooooooooooooooooooooooooooo
CRON      |153 (2.62%)   ooooo
root      |147 (2.52%)   ooooo
message   |99 (1.69%)    ooo
last      |99 (1.69%)    ooo
ntpd      |99 (1.69%)    ooo
dhclient  |88 (1.51%)    ooo
THREADMGMT|52 (0.89%)    oo

Process List Example

You can start thinking of normal commands in new ways. For example, you can take your "ps ax" output, get just the command portion, and do a word-analysis on it. You might find some words are rather interesting. In this case, it appears Chrome is doing some sort of A/B testing and their commandline exposes that.

$ ps axww \
    | cut -c 28- \
    | distribution --tokenize=word --match=word --char='|' --width=90 --height=25
Val                     |Ct (Pct)    Histogram
usr                     |100 (6.17%) |||||||||||||||||||||||||||||||||||||||||||||||||||||
lib                     |73 (4.51%)  ||||||||||||||||||||||||||||||||||||||
browser                 |38 (2.35%)  ||||||||||||||||||||
chromium                |38 (2.35%)  ||||||||||||||||||||
P                       |32 (1.98%)  |||||||||||||||||
daemon                  |31 (1.91%)  |||||||||||||||||
sbin                    |26 (1.60%)  ||||||||||||||
gnome                   |23 (1.42%)  ||||||||||||
bin                     |22 (1.36%)  ||||||||||||
kworker                 |21 (1.30%)  |||||||||||
type                    |19 (1.17%)  ||||||||||
gvfs                    |17 (1.05%)  |||||||||
no                      |17 (1.05%)  |||||||||
en                      |16 (0.99%)  |||||||||
indicator               |15 (0.93%)  ||||||||
channel                 |14 (0.86%)  ||||||||
bash                    |14 (0.86%)  ||||||||
US                      |14 (0.86%)  ||||||||
lang                    |14 (0.86%)  ||||||||
force                   |12 (0.74%)  |||||||
pluto                   |12 (0.74%)  |||||||
ProxyConnectionImpact   |12 (0.74%)  |||||||
HiddenExperimentB       |12 (0.74%)  |||||||
ConnectBackupJobsEnabled|12 (0.74%)  |||||||
session                 |12 (0.74%)  |||||||

Graphing Pre-Tallied Tokens Example

Sometimes the output you have is already some keys with their counts. For example the output of "du" or "command | uniq -c". In these cases, use the --graph (-g) option, which skips the parsing and tokenizing of the input.

Further, you can use very short versions of the options in case you don't like typing a lot. The default character is "+" because it creates a type of grid system which makes it easy for the eye to trace right/left or up/down.

$ sudo du -sb /etc/* | distribution -w=90 -h=15 -g
Val                   |Ct (Pct)         Histogram
/etc/mateconf         |7780758 (44.60%) +++++++++++++++++++++++++++++++++++++++++++++++++
/etc/brltty           |3143272 (18.02%) ++++++++++++++++++++
/etc/apparmor.d       |1597915 (9.16%)  ++++++++++
/etc/bash_completion.d|597836 (3.43%)   ++++
/etc/mono             |535352 (3.07%)   ++++
/etc/ssl              |465414 (2.67%)   +++
/etc/ardour2          |362303 (2.08%)   +++
/etc/X11              |226309 (1.30%)   ++
/etc/ImageMagick      |202358 (1.16%)   ++
/etc/init.d           |143281 (0.82%)   +
/etc/ssh              |138042 (0.79%)   +
/etc/fonts            |119862 (0.69%)   +
/etc/sound            |112051 (0.64%)   +
/etc/xdg              |111971 (0.64%)   +
/etc/java-7-openjdk   |100414 (0.58%)   +

Keys in Natural Order Examples

The output is separated between STDOUT and STDERR so you can sort the resulting histogram by values. This is useful for time series or other cases where the keys you're graphing are in some natural order. Note how the "-v" output still appears at the top.

$ cat NotServingRegionException-DateHour.txt \
    | distribution -v \
    | sort -n
tokens/lines examined: 1,414,196    
 tokens/lines matched: 1,414,196
       histogram keys: 453
              runtime: 1279.30ms
Val             |Ct (Pct)      Histogram
   2012-07-13 03|38360 (2.71%) ++++++++++++++++++++++++
   2012-07-28 21|18293 (1.29%) ++++++++++++
   2012-07-28 23|20748 (1.47%) +++++++++++++
   2012-07-29 06|15692 (1.11%) ++++++++++
   2012-07-29 07|30432 (2.15%) +++++++++++++++++++
   2012-07-29 08|76943 (5.44%) ++++++++++++++++++++++++++++++++++++++++++++++++
   2012-07-29 09|54955 (3.89%) ++++++++++++++++++++++++++++++++++
   2012-07-30 05|15652 (1.11%) ++++++++++
   2012-07-30 09|40102 (2.84%) +++++++++++++++++++++++++
   2012-07-30 10|21718 (1.54%) ++++++++++++++
   2012-07-30 16|16041 (1.13%) ++++++++++
   2012-08-01 09|22740 (1.61%) ++++++++++++++
   2012-08-02 04|31851 (2.25%) ++++++++++++++++++++
   2012-08-02 06|28748 (2.03%) ++++++++++++++++++
   2012-08-02 07|18062 (1.28%) ++++++++++++
   2012-08-02 20|23519 (1.66%) +++++++++++++++
   2012-08-03 03|21587 (1.53%) ++++++++++++++
   2012-08-03 08|33409 (2.36%) +++++++++++++++++++++
   2012-08-03 10|15854 (1.12%) ++++++++++
   2012-08-03 15|29828 (2.11%) +++++++++++++++++++
   2012-08-03 16|20478 (1.45%) +++++++++++++
   2012-08-03 17|39758 (2.81%) +++++++++++++++++++++++++
   2012-08-03 18|19514 (1.38%) ++++++++++++
   2012-08-03 19|18353 (1.30%) ++++++++++++
   2012-08-03 22|18726 (1.32%) ++++++++++++
__________________

$ cat /usr/share/dict/words 
    | awk '{print length($1)}' 
    | distribution -c=: -w=90 -h=16 
    | sort -n
Val|Ct (Pct)       Histogram
2 |182 (0.18%)    :
3 |845 (0.85%)    ::::
4 |3346 (3.37%)   ::::::::::::::::
5 |6788 (6.84%)   :::::::::::::::::::::::::::::::
6 |11278 (11.37%) ::::::::::::::::::::::::::::::::::::::::::::::::::::
7 |14787 (14.91%) :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
8 |15674 (15.81%) ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
9 |14262 (14.38%) :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
10|11546 (11.64%) :::::::::::::::::::::::::::::::::::::::::::::::::::::
11|8415 (8.49%)   :::::::::::::::::::::::::::::::::::::::
12|5508 (5.55%)   :::::::::::::::::::::::::
13|3236 (3.26%)   :::::::::::::::
14|1679 (1.69%)   ::::::::
15|893 (0.90%)    :::::
16|382 (0.39%)    ::
17|176 (0.18%)    :

MySQL Slow Query Log Analysis Examples

You can sometimes gain interesting insights just by measuring the size of files on your filesystem. Someone had captured slow-query-logs for every hour for most of a day. Assuming they all compressed the same (a proper analysis would be on uncompressed files - uncompressing them would have caused server impact - this is good enough for illustration's sake), we can determine how much slow query traffic appeared during a given hour of the day.

Something happened around 8am but otherwise the server seems to follow a normal sinusoidal pattern. But note because we're only analysing the file size, it could be that 8am had the same number of slow queries, but that the queries themselves were larger in byte-count. Or that the queries didn't compress as well.

Also note that we aren't seeing every histogram entry here. Always take care to remember the tool is hiding low-frequency data from you unless you ask it to draw uncommonly-tall histograms.

$ du -sb mysql-slow.log.*.gz | ~/distribution -g | sort -n
Val                 |Ct (Pct)         Histogram
mysql-slow.log.01.gz|1426694 (5.38%)  ++++++++++++++++++++
mysql-slow.log.02.gz|1499467 (5.65%)  +++++++++++++++++++++
mysql-slow.log.03.gz|1840727 (6.94%)  ++++++++++++++++++++++++++
mysql-slow.log.04.gz|1570131 (5.92%)  ++++++++++++++++++++++
mysql-slow.log.05.gz|1439021 (5.42%)  ++++++++++++++++++++
mysql-slow.log.07.gz|859939 (3.24%)   ++++++++++++
mysql-slow.log.08.gz|2976177 (11.21%) ++++++++++++++++++++++++++++++++++++++++++
mysql-slow.log.09.gz|792269 (2.99%)   +++++++++++
mysql-slow.log.11.gz|722148 (2.72%)   ++++++++++
mysql-slow.log.12.gz|825731 (3.11%)   ++++++++++++
mysql-slow.log.14.gz|1476023 (5.56%)  +++++++++++++++++++++
mysql-slow.log.15.gz|2087129 (7.86%)  +++++++++++++++++++++++++++++
mysql-slow.log.16.gz|1905867 (7.18%)  +++++++++++++++++++++++++++
mysql-slow.log.19.gz|1314297 (4.95%)  +++++++++++++++++++
mysql-slow.log.20.gz|802212 (3.02%)   ++++++++++++

A more-proper analysis on another set of slow logs involved actually getting the time the query ran, pulling out the date/hour portion of the timestamp, and graphing the result.

At first blush, it might appear someone had captured logs for various hours of one day and at 10am for several days in a row. However, note that the Pct column shows this is only about 20% of all data, which we can also conclude because there are 964 histogram entries, of which we're only seeing a couple dozen. This means something happened on July 31st that caused slow queries all day, and then 10am is a time of day when slow queries tend to happen. To test this theory, we might re-run this with a "--height=600" (or even 900) to see nearly all the entries to get a more precise idea of what's going on.

$ zcat mysql-slow.log.*.gz \
    | fgrep Time: \
    | cut -c 9-17 \
    | ~/distribution --width=90 --verbose \
    | sort -n
tokens/lines examined: 30,027    
 tokens/lines matched: 30,027
       histogram keys: 964
              runtime: 1224.58ms
Val      |Ct (Pct)    Histogram
120731 03|274 (0.91%) ++++++++++++++++++++++++++++++++++
120731 04|210 (0.70%) ++++++++++++++++++++++++++
120731 07|208 (0.69%) ++++++++++++++++++++++++++
120731 08|271 (0.90%) +++++++++++++++++++++++++++++++++
120731 09|403 (1.34%) +++++++++++++++++++++++++++++++++++++++++++++++++
120731 10|556 (1.85%) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
120731 11|421 (1.40%) +++++++++++++++++++++++++++++++++++++++++++++++++++
120731 12|293 (0.98%) ++++++++++++++++++++++++++++++++++++
120731 13|327 (1.09%) ++++++++++++++++++++++++++++++++++++++++
120731 14|318 (1.06%) +++++++++++++++++++++++++++++++++++++++
120731 15|446 (1.49%) ++++++++++++++++++++++++++++++++++++++++++++++++++++++
120731 16|397 (1.32%) ++++++++++++++++++++++++++++++++++++++++++++++++
120731 17|228 (0.76%) ++++++++++++++++++++++++++++
120801 10|515 (1.72%) +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
120803 10|223 (0.74%) +++++++++++++++++++++++++++
120809 10|215 (0.72%) ++++++++++++++++++++++++++
120810 10|210 (0.70%) ++++++++++++++++++++++++++
120814 10|193 (0.64%) ++++++++++++++++++++++++
120815 10|205 (0.68%) +++++++++++++++++++++++++
120816 10|207 (0.69%) +++++++++++++++++++++++++
120817 10|226 (0.75%) ++++++++++++++++++++++++++++
120819 10|197 (0.66%) ++++++++++++++++++++++++

A typical problem for MySQL administrators is figuring out how many slow queries are taking how long. The slow query log can be quite verbose. Analysing it in a visual nature can help. For example, there is a line that looks like this in the slow query log:

 # Query_time: 5.260353  Lock_time: 0.000052  Rows_sent: 0  Rows_examined: 2414  Rows_affected: 1108  Rows_read: 2

It might be useful to see how many queries ran for how long in increments of tenths of seconds. You can grab that third field and get tenth-second precision with a simple awk command, then graph the result.

It seems interesting that there are spikes at 3.2, 3.5, 4, 4.3, 4.5 seconds. One hypothesis might be that those are individual queries, each warranting its own analysis.

$ head -90000 mysql-slow.log.20120710 \
    | fgrep Query_time: \
    | awk '{print int($3 * 10)/10}' \
    | ~/distribution --verbose --height=30 --char='|o' \
    | sort -n
tokens/lines examined: 12,269    
 tokens/lines matched: 12,269
       histogram keys: 481
              runtime: 12.53ms
Val|Ct (Pct)     Histogram
0  |1090 (8.88%) ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||o
2  |1018 (8.30%) |||||||||||||||||||||||||||||||||||||||||||||||||||||||||o
2.1|949 (7.73%)  |||||||||||||||||||||||||||||||||||||||||||||||||||||o
2.2|653 (5.32%)  |||||||||||||||||||||||||||||||||||||o
2.3|552 (4.50%)  |||||||||||||||||||||||||||||||o
2.4|554 (4.52%)  |||||||||||||||||||||||||||||||o
2.5|473 (3.86%)  ||||||||||||||||||||||||||o
2.6|423 (3.45%)  ||||||||||||||||||||||||o
2.7|394 (3.21%)  ||||||||||||||||||||||o
2.8|278 (2.27%)  |||||||||||||||o
2.9|189 (1.54%)  ||||||||||o
3  |173 (1.41%)  |||||||||o
3.1|193 (1.57%)  ||||||||||o
3.2|200 (1.63%)  |||||||||||o
3.3|138 (1.12%)  |||||||o
3.4|176 (1.43%)  ||||||||||o
3.5|213 (1.74%)  ||||||||||||o
3.6|157 (1.28%)  ||||||||o
3.7|134 (1.09%)  |||||||o
3.8|121 (0.99%)  ||||||o
3.9|96 (0.78%)   |||||o
4  |110 (0.90%)  ||||||o
4.1|80 (0.65%)   ||||o
4.2|84 (0.68%)   ||||o
4.3|90 (0.73%)   |||||o
4.4|76 (0.62%)   ||||o
4.5|93 (0.76%)   |||||o
4.6|79 (0.64%)   ||||o
4.7|71 (0.58%)   ||||o
5.1|70 (0.57%)   |||o

Apache Logs Analysis Example

Even if you know sed/awk/grep, the built-in tokenizing/matching can be less verbose. Say you want to look at all the URLs in your Apache logs. People will be doing GET /a/b/c /a/c/f q/r/s q/n/p. A and Q are the most common, so you can tokenize on / and the latter parts of the URL will be buried, statistically.

By tokenizing and matching using the script, you may also find unexpected common portions of the URL that don't show up in the prefix.

$ zcat access.log*gz \
    | awk '{print $7}' \
    | distribution -t=/ -h=15
Val            |Ct (Pct)      Histogram
Art            |1839 (16.58%) +++++++++++++++++++++++++++++++++++++++++++++++++
Rendered       |1596 (14.39%) ++++++++++++++++++++++++++++++++++++++++++
Blender        |1499 (13.52%) ++++++++++++++++++++++++++++++++++++++++
AznRigging     |760 (6.85%)   ++++++++++++++++++++
Music          |457 (4.12%)   ++++++++++++
Ringtones      |388 (3.50%)   +++++++++++
CuteStance     |280 (2.52%)   ++++++++
Traditional    |197 (1.78%)   ++++++
Technology     |171 (1.54%)   +++++
CreativeExhaust|134 (1.21%)   ++++
Fractals       |127 (1.15%)   ++++
robots.txt     |125 (1.13%)   ++++
RingtoneEP1.mp3|125 (1.13%)   ++++
Poetry         |108 (0.97%)   +++
RingtoneEP2.mp3|95 (0.86%)    +++

Here we had pulled apart our access logs and put them in TSV format for input into Hive. The user agent string was in the 13th position. I wanted to just get an overall idea of what sort of user agents were coming to the site. I'm using the minimal argument size and my favorite "character" combo of "|o". I find it interesting that there were only 474 unique word-based tokens in the input. Also, it's clear that a large percentage of the visitors come with mobile devices now.

$ zcat weblog-2014-05.tsv.gz \
  | awk -F '\t' '{print $13}' \
  | distribution -t=word -m=word -c='|o' -s=m -v
tokens/lines examined: 28,062,913    
 tokens/lines matched: 11,507,407
       histogram keys: 474
              runtime: 15659.97ms
Val        |Ct (Pct)       Histogram
Mozilla    |912852 (7.93%) ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||o
like       |722945 (6.28%) |||||||||||||||||||||||||||||||||||||||||||||||||||||||||o
OS         |611503 (5.31%) ||||||||||||||||||||||||||||||||||||||||||||||||o
AppleWebKit|605618 (5.26%) |||||||||||||||||||||||||||||||||||||||||||||||o
Gecko      |535620 (4.65%) ||||||||||||||||||||||||||||||||||||||||||o
Windows    |484056 (4.21%) ||||||||||||||||||||||||||||||||||||||o
NT         |483085 (4.20%) ||||||||||||||||||||||||||||||||||||||o
KHTML      |356730 (3.10%) ||||||||||||||||||||||||||||o
Safari     |355400 (3.09%) ||||||||||||||||||||||||||||o
X          |347033 (3.02%) |||||||||||||||||||||||||||o
Mac        |344205 (2.99%) |||||||||||||||||||||||||||o
appversion |300816 (2.61%) |||||||||||||||||||||||o
Type       |299085 (2.60%) |||||||||||||||||||||||o
Connection |299085 (2.60%) |||||||||||||||||||||||o
Mobile     |282759 (2.46%) ||||||||||||||||||||||o
CPU        |266837 (2.32%) |||||||||||||||||||||o
NET        |247418 (2.15%) |||||||||||||||||||o
CLR        |247418 (2.15%) |||||||||||||||||||o
Aspect     |242566 (2.11%) |||||||||||||||||||o
Ratio      |242566 (2.11%) |||||||||||||||||||o

And here we had a list of referrers in "referrer [count]" format. They were done one per day, but I wanted a count for January through September, so I used a shell glob to specify all those files for my 'cat'. Distribution will notice that it's getting the same key as previously and just add the new value, so the key "x1" can come in many times and we'll get the aggregate in the output. The referrers have been anonymized here since they are very specific to the company.

$ cat referrers-20140* | distribution -v -g=kv -s=m
tokens/lines examined: 133,564    
 tokens/lines matched: 31,498,986
       histogram keys: 14,882
              runtime: 453.45ms
Val    |Ct (Pct)          Histogram
x1     |24313595 (77.19%) ++++++++++++++++++++++++++++++++++++++++++++++++++++
x2     |3430278 (10.89%)  ++++++++
x3     |1049996 (3.33%)   +++
x4     |210083 (0.67%)    +
x5     |179554 (0.57%)    +
x6     |163158 (0.52%)    +
x7     |129997 (0.41%)    +
x8     |122725 (0.39%)    +
x9     |120487 (0.38%)    +
xa     |109085 (0.35%)    +
xb     |99956 (0.32%)     +
xc     |92208 (0.29%)     +
xd     |90017 (0.29%)     +
xe     |79416 (0.25%)     +
xf     |70094 (0.22%)     +
xg     |58089 (0.18%)     +
xh     |52349 (0.17%)     +
xi     |37002 (0.12%)     +
xj     |36651 (0.12%)     +
xk     |32860 (0.10%)     +

This seems a really good time to use the --logarithmic option, since that top referrer is causing a loss of resolution on the following ones! I'll re-run this for one month.

$ cat referrers-201402* | distribution -v -g=kv -s=m -l
tokens/lines examined: 23,517    
 tokens/lines matched: 5,908,765 
       histogram keys: 5,888
              runtime: 78.28ms
Val  |Ct (Pct)         Histogram
x1   |4471708 (75.68%) +++++++++++++++++++++++++++++++++++++++++++++++++++++
x2   |670703 (11.35%)  ++++++++++++++++++++++++++++++++++++++++++++++
x3   |203489 (3.44%)   ++++++++++++++++++++++++++++++++++++++++++
x4   |43751 (0.74%)    +++++++++++++++++++++++++++++++++++++
x5   |36211 (0.61%)    ++++++++++++++++++++++++++++++++++++
x6   |34589 (0.59%)    ++++++++++++++++++++++++++++++++++++
x7   |31279 (0.53%)    ++++++++++++++++++++++++++++++++++++
x8   |29596 (0.50%)    +++++++++++++++++++++++++++++++++++
x9   |23125 (0.39%)    +++++++++++++++++++++++++++++++++++
xa   |21429 (0.36%)    ++++++++++++++++++++++++++++++++++
xb   |19670 (0.33%)    ++++++++++++++++++++++++++++++++++
xc   |19057 (0.32%)    ++++++++++++++++++++++++++++++++++
xd   |18945 (0.32%)    ++++++++++++++++++++++++++++++++++
xe   |18936 (0.32%)    ++++++++++++++++++++++++++++++++++
xf   |16015 (0.27%)    +++++++++++++++++++++++++++++++++
xg   |13115 (0.22%)    +++++++++++++++++++++++++++++++++
xh   |12067 (0.20%)    ++++++++++++++++++++++++++++++++
xi   |8485 (0.14%)     +++++++++++++++++++++++++++++++
xj   |7694 (0.13%)     +++++++++++++++++++++++++++++++
xk   |7199 (0.12%)     +++++++++++++++++++++++++++++++

Graphing a Series of Numbers Example

Suppose you just have a list of integers you want to graph. For example, you've captured a "show global status" for every second for 5 minutes, and you want to grep out just one stat for the five-minute sample and graph it.

Or, slightly more-difficult, you want to pull out the series of numbers and only graph the difference between each pair (as in a monotonically-increasing counter). The

--numonly=

Therefore there's a lot of output, which is snipped in the example output that follows. The "val" column is simply an ascending list of integers, so you can tell where output was snipped by the jumps in those values.

$ grep ^Innodb_data_reads globalStatus*.txt \
    | awk '{print $2}' \
    | distribution --numonly=mon --char='|+'
Val|Ct (Pct)     Histogram
1  |0 (0.00%)    +
91 |15 (0.05%)   +
92 |14 (0.04%)   +
93 |30 (0.10%)   |+
94 |11 (0.03%)   +
95 |922 (2.93%)  |||||||||||||||||||||||||||||||||||||||||||||||||||||||||+
96 |372 (1.18%)  |||||||||||||||||||||||+
97 |44 (0.14%)   ||+
98 |37 (0.12%)   ||+
99 |110 (0.35%)  ||||||+
100|18 (0.06%)   |+
101|12 (0.04%)   +
102|19 (0.06%)   |+
103|164 (0.52%)  ||||||||||+
200|62 (0.20%)   |||+
201|372 (1.18%)  |||||||||||||||||||||||+
202|228 (0.72%)  ||||||||||||||+
203|43 (0.14%)   ||+
204|917 (2.91%)  ||||||||||||||||||||||||||||||||||||||||||||||||||||||||+
205|64 (0.20%)   |||+
206|178 (0.57%)  |||||||||||+
207|90 (0.29%)   |||||+
208|90 (0.29%)   |||||+
209|101 (0.32%)  ||||||+
453|0 (0.00%)    +
454|0 (0.00%)    +

The Python version eschews the header and superfluous "key" as the Perl version will probably also soon do:

$ cat ~/tmp/numberSeries.txt | xargs
01 05 06 09 12 22 28 32 34 30 37 44 48 54 63 70 78 82 85 88 89 89 90 92 95
$ cat ~/tmp/numberSeries.txt \
    | ~/Dev/distribution/distribution.py --numonly -c='|o' -s=s
   5  (0.39%) ||o
   6  (0.47%) ||o
   9  (0.70%) ||||o
  12  (0.94%) |||||o
  22  (1.71%) ||||||||||o
  28  (2.18%) |||||||||||||o
  32  (2.49%) |||||||||||||||o
  34  (2.65%) ||||||||||||||||o
  30  (2.34%) ||||||||||||||o
  37  (2.88%) |||||||||||||||||o
  44  (3.43%) ||||||||||||||||||||o
  48  (3.74%) ||||||||||||||||||||||o
  54  (4.21%) |||||||||||||||||||||||||o
  63  (4.91%) |||||||||||||||||||||||||||||o
  70  (5.46%) |||||||||||||||||||||||||||||||||o
  78  (6.08%) ||||||||||||||||||||||||||||||||||||o
  82  (6.39%) ||||||||||||||||||||||||||||||||||||||o
  85  (6.63%) ||||||||||||||||||||||||||||||||||||||||o
  88  (6.86%) |||||||||||||||||||||||||||||||||||||||||o
  89  (6.94%) ||||||||||||||||||||||||||||||||||||||||||o
  89  (6.94%) ||||||||||||||||||||||||||||||||||||||||||o
  90  (7.01%) ||||||||||||||||||||||||||||||||||||||||||o
  92  (7.17%) |||||||||||||||||||||||||||||||||||||||||||o
  95  (7.40%) |||||||||||||||||||||||||||||||||||||||||||||o
$ cat ~/tmp/numberSeries.txt \
    | ~/Dev/distribution/distribution.py --numonly=diff -c='|o' -s=s
   4  (4.26%) ||||||||||||||||||o
   1  (1.06%) ||||o
   3  (3.19%) |||||||||||||o
   3  (3.19%) |||||||||||||o
  10 (10.64%) |||||||||||||||||||||||||||||||||||||||||||||o
   6  (6.38%) |||||||||||||||||||||||||||o
   4  (4.26%) ||||||||||||||||||o
   2  (2.13%) |||||||||o
  -4 (-4.26%) o
   7  (7.45%) |||||||||||||||||||||||||||||||o
   7  (7.45%) |||||||||||||||||||||||||||||||o
   4  (4.26%) ||||||||||||||||||o
   6  (6.38%) |||||||||||||||||||||||||||o
   9  (9.57%) ||||||||||||||||||||||||||||||||||||||||o
   7  (7.45%) |||||||||||||||||||||||||||||||o
   8  (8.51%) ||||||||||||||||||||||||||||||||||||o
   4  (4.26%) ||||||||||||||||||o
   3  (3.19%) |||||||||||||o
   3  (3.19%) |||||||||||||o
   1  (1.06%) ||||o
   0  (0.00%) o
   1  (1.06%) ||||o
   2  (2.13%) |||||||||o
   3  (3.19%) |||||||||||||o

HDFS DU Example

HDFS files are often rather large, so I first change the numeric file size to megabytes by dividing by 1048576. I must also change it to an int value, since distribution doesn't currently deal with non-integer counts.

Also, we are pre-parsing the du output to give us only the megabytes count and the final entry in the filename.

awk -F /

$ hdfs dfs -du /apps/hive/warehouse/aedb/hitcounts_byday/cookie_type=shopper \
    | awk -F / '{print int($1/1048576) " " $8}' \
    | distribution -g -c='-~' --height=20 \
    | sort
          Key|Ct      (Pct)   Histogram
dt=2014-11-15|3265438 (2.53%) ----------------------------------------------~
dt=2014-11-16|3241614 (2.51%) ----------------------------------------------~
dt=2014-11-20|2964636 (2.29%) ------------------------------------------~
dt=2014-11-21|3049912 (2.36%) -------------------------------------------~
dt=2014-11-22|3292621 (2.55%) -----------------------------------------------~
dt=2014-11-23|3319538 (2.57%) -----------------------------------------------~
dt=2014-11-24|3070654 (2.38%) -------------------------------------------~
dt=2014-11-25|3086090 (2.39%) --------------------------------------------~
dt=2014-11-27|3113888 (2.41%) --------------------------------------------~
dt=2014-11-28|3124426 (2.42%) --------------------------------------------~
dt=2014-11-29|3431859 (2.66%) -------------------------------------------------~
dt=2014-11-30|3391117 (2.62%) ------------------------------------------------~
dt=2014-12-01|3167744 (2.45%) ---------------------------------------------~
dt=2014-12-02|3134248 (2.43%) --------------------------------------------~
dt=2014-12-03|3023733 (2.34%) -------------------------------------------~
dt=2014-12-04|3022274 (2.34%) -------------------------------------------~
dt=2014-12-05|3040776 (2.35%) -------------------------------------------~
dt=2014-12-06|3054159 (2.36%) -------------------------------------------~
dt=2014-12-09|3065252 (2.37%) -------------------------------------------~
dt=2014-12-10|3316703 (2.57%) -----------------------------------------------~

Running Tests

The

tests

distribution

tests

cd tests/
distribution=../distribution ./runTests.sh
distribution=../distribution.py ./runTests.sh

The

runTests.sh

-v

To-Do List

New features are unlikely to be added, as the existing functionality is already arguably a superset of what's necessary. Still, there are some things that need to be done.

• No Time::HiRes Perl module? Don't die. Much harder than it should be. Negated by next to-do.
• Get scripts into package managers.

Porting

Perl and Python are fairly common, but I'm not sure 100% of systems out there have them. A C/C++ port would be most welcome.

If you write a port, send me a pull request so I can include it in this repo.

Port requirements: from the user's point of view, it's the exact same script. They pass in the same options in the same way, and get the same output, byte-for-byte if possible. This means you'll need (Perl) regexp support in your language of choice. Also a hash map structure makes the implementation simple, but more-efficient methods are welcome.

I imagine, in order of nice-to-haveness:

• C or C++
• Go
• Lisp
• Ocaml
• Java
• Ruby

List of known ports

1. go-distribution

Authors

• Tim Ellis
• Philo Vivero
• Taylor Stearns