Simple example of aggregated logging using log_by_lua hooks
This is an example for embedding Lua into the logging phase of Nginx and building a simple log aggregation service.
One of the only dependencies not directly included in OpenResty is libpcre. You can either install it from source, or use your distribution. A modern version (>8.21) is recommended to leverage PCRE_JIT (faster regex), but it will still work with an older version.
In my case on Mac OS using Homebrew :
$ brew install pcre
Then download the latest OpenResty from this page and compile it. More up do date installation intruction can be found on the Openresty install page
$ wget http://agentzh.org/misc/nginx/ngx_openresty-1.2.4.11.tar.gz $ tar xvzf ngx_openresty-1.2.4.11.tar.gz $ cd ngx_openresty-1.2.4.11
Now let's finally compile it. I found a slight issue that only happens on Mac OS while using hombrew, and I had to the configure line where to find my pcre library (
--with-ld-opt="-L/usr/local/lib") This should get fixed in the next releases
$ ./configure --with-luajit --with-ld-opt="-L/usr/local/lib" $ make
This will create a /usr/local/openresty/ directory that contains everything you need
$ make install
Let's check that the Nginx we just compiled has all the necessary to run this example
$ /usr/local/openresty/nginx/sbin/nginx -V
This will display all the module that Nginx has been built with, including ngxlua-0.7.5 here (the dev version of ngxlua can be found here)
configure arguments: --prefix=/usr/local/openresty/nginx --add-module=../ngx_devel_kit-0.2.17 --add-module=../echo-nginx-module-0.41 --add-module=../xss-nginx-module-0.03rc9 --add-module=../ngx_coolkit-0.2rc1 --add-module=../set-misc-nginx-module-0.22rc8 --add-module=../form-input-nginx-module-0.07rc5 --add-module=../encrypted-session-nginx-module-0.02 --add-module=../srcache-nginx-module-0.16 --add-module=../ngx_lua-0.7.5 --add-module=../headers-more-nginx-module-0.19 --add-module=../array-var-nginx-module-0.03rc1 --add-module=../memc-nginx-module-0.13rc3 --add-module=../redis2-nginx-module-0.09 --add-module=../redis-nginx-module-0.3.6 --add-module=../auth-request-nginx-module-0.2 --add-module=../rds-json-nginx-module-0.12rc10 --add-module=../rds-csv-nginx-module-0.05rc2 --with-http_ssl_module
Now let's run Nginx with the configuration example. (You don't need to run it as root, as the example doesn't listen on port 80)
$ git clone https://github.com/mtourne/nginx_log_by_lua.git
The server will run until you send you hit Control+C
$ /usr/local/openresty/nginx/sbin/nginx -p nginx_log_by_lua
$ curl localhost:8080
If everything went right, you should see the sever response :
Hello
Now, let's query our log aggregation port :
$ curl localhost:6080
And here is the result :
Since last measure: 1.0729999542236 secs Request Count: 1 Average req time: 0 secs Requests per Secs: 0.93196648896742
As a final test you can try hammering your web application and see the log aggregation in action :
ab -n 1000 -c 100 http://localhost:8080/ > /dev/null && echo && curl http://localhost:6080/Completed 100 requests Completed 200 requests Completed 300 requests Completed 400 requests Completed 500 requests Completed 600 requests Completed 700 requests Completed 800 requests Completed 900 requests Completed 1000 requests Finished 1000 requests
Since last measure: 0.10899996757507 secs Request Count: 1000 Average req time: 0.0012160038948059 secs Requests per Secs: 9174.3146557475
For the purpose of the example the code is very simple, but with very simple additions a json printer could be added to make the output easier to consume by other services (OpenResty does include the package lua-cjson)
We could also push the example further and aggregate logs to calculate timings similarly to
ab, which displays what is the average request_time for the 90, 95, 99 percentile.
Happy Hacking!