This article will discuss the nginx-log-collector project, which will read nginx logs and send them to the Clickhouse cluster. Usually for logs we use ElasticSearch. Clickhouse requires less resources (disk space, RAM, CPU). Clickhouse records data faster. Clickhouse compresses data, making disk data even more compact. 

  • To view log analytics, create a dashboard for Grafana.
  • Install nginx, grafana in the standard way
  • Install the clickhouse cluster using ansible-playbook 

Creating Databases and Tables in Clickhouse

This file describes SQL queries for creating a database and tables for nginx-log-collector in Clickhouse.

We make each request in turn on each server of the Clickhouse cluster.

Important note. On this line, logs_cluster should be replaced with your cluster name from the clickhouse_remote_servers.xml file between “remote_servers” and “shard”.

Installing and Configuring nginx-log-collector-rpm

Nginx-log-collector does not have rpm. Here https://github.com/patsevanton/nginx-log-collector-rpm create rpm for it. Rpm will be collected using Fedora Copr.

Install the rpm package nginx-log-collector-rpm

Edit the config /etc/nginx-log-collector/config.yaml:

Nginx setup

 

General nginx config:

vhost1.conf:

Add virtual hosts to the / etc / hosts file:

HTTP server emulator

As an HTTP server emulator we will use nodejs-stub-server.

Nodejs-stub-server does not have rpm. Here https://github.com/patsevanton/nodejs-stub-server create rpm for it. Rpm will be collected using Fedora Copr

Install nodejs-stub-server package on upstream nginx rpm

Stress Testing

Testing done using Apache benchmark.

Install it:

We start testing using Apache benchmark from 5 different servers:

Grafana Setup

You also need to create a table variable with the contents of nginx.access_log.

Singlestat Total Requests:

Singlestat Failed Requests:

Singlestat Failing Percent:

Singlestat Avg Response Time:

Singlestat Max Response Time:

Count Status:

To output data, you need to install the plugin and restart grafana.

Pie TOP 5 Status:

Count http_user_agent:

GoodRate/BadRate:

Response Timing:

Upstream response time (response time of the 1st upstream):

Conclusion: 

Hopefully the community will get involved in developing / testing and using nginx-log-collector.

And someone, when he implements nginx-log-collector, will tell you how much he saved the disk, RAM, CPU.