nginx_log_by_lua

by mtourne

Simple example of aggregated logging using log_by_lua hooks

128 Stars 23 Forks Last release: Not found Other 3 Commits 0 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:

This is an example for embedding Lua into the logging phase of Nginx and building a simple log aggregation service.

Installation

One of the only dependencies not directly included in OpenResty is libpcre. You can either install it from source, or use your distribution. A modern version (>8.21) is recommended to leverage PCRE_JIT (faster regex), but it will still work with an older version.

In my case on Mac OS using Homebrew :

$ brew install pcre

Then download the latest OpenResty from this page and compile it. More up do date installation intruction can be found on the Openresty install page

$ wget http://agentzh.org/misc/nginx/ngx_openresty-1.2.4.11.tar.gz
$ tar xvzf ngx_openresty-1.2.4.11.tar.gz
$ cd ngx_openresty-1.2.4.11

Now let's finally compile it. I found a slight issue that only happens on Mac OS while using hombrew, and I had to the configure line where to find my pcre library (

--with-ld-opt="-L/usr/local/lib"
) This should get fixed in the next releases
$ ./configure --with-luajit --with-ld-opt="-L/usr/local/lib"
$ make

This will create a /usr/local/openresty/ directory that contains everything you need

$ make install

Launching the demo server

Let's check that the Nginx we just compiled has all the necessary to run this example

$ /usr/local/openresty/nginx/sbin/nginx -V

This will display all the module that Nginx has been built with, including ngxlua-0.7.5 here (the dev version of ngxlua can be found here)

configure arguments: --prefix=/usr/local/openresty/nginx --add-module=../ngx_devel_kit-0.2.17 --add-module=../echo-nginx-module-0.41 --add-module=../xss-nginx-module-0.03rc9 --add-module=../ngx_coolkit-0.2rc1 --add-module=../set-misc-nginx-module-0.22rc8 --add-module=../form-input-nginx-module-0.07rc5 --add-module=../encrypted-session-nginx-module-0.02 --add-module=../srcache-nginx-module-0.16 --add-module=../ngx_lua-0.7.5 --add-module=../headers-more-nginx-module-0.19 --add-module=../array-var-nginx-module-0.03rc1 --add-module=../memc-nginx-module-0.13rc3 --add-module=../redis2-nginx-module-0.09 --add-module=../redis-nginx-module-0.3.6 --add-module=../auth-request-nginx-module-0.2 --add-module=../rds-json-nginx-module-0.12rc10 --add-module=../rds-csv-nginx-module-0.05rc2 --with-http_ssl_module

Now let's run Nginx with the configuration example. (You don't need to run it as root, as the example doesn't listen on port 80)

$ git clone https://github.com/mtourne/nginx_log_by_lua.git

The server will run until you send you hit Control+C

$ /usr/local/openresty/nginx/sbin/nginx -p nginx_log_by_lua

Testing it

$ curl localhost:8080

If everything went right, you should see the sever response :

Hello

Now, let's query our log aggregation port :

$ curl localhost:6080

And here is the result :

Since last measure: 1.0729999542236 secs
Request Count:      1
Average req time:   0 secs
Requests per Secs:  0.93196648896742

As a final test you can try hammering your web application and see the log aggregation in action :

ab -n 1000 -c 100 http://localhost:8080/ > /dev/null && echo && curl http://localhost:6080/

Completed 100 requests Completed 200 requests Completed 300 requests Completed 400 requests Completed 500 requests Completed 600 requests Completed 700 requests Completed 800 requests Completed 900 requests Completed 1000 requests Finished 1000 requests

Since last measure: 0.10899996757507 secs Request Count: 1000 Average req time: 0.0012160038948059 secs Requests per Secs: 9174.3146557475

Beyond the example

For the purpose of the example the code is very simple, but with very simple additions a json printer could be added to make the output easier to consume by other services (OpenResty does include the package lua-cjson)

We could also push the example further and aggregate logs to calculate timings similarly to

ab
, which displays what is the average request_time for the 90, 95, 99 percentile.

Happy Hacking!

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.