Skip to content

logs: Implement a parser for haproxy log formats #6

@fionera

Description

@fionera

A fast reader implementation without regex would be cool.

// none
// Connect from 127.0.0.1:57765 to 127.0.0.1:8080 (test/HTTP)

// tcplog
// "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq"
// 127.0.0.1:57843 [16/Oct/2023:02:58:03.240] test app_backend/<NOSRV> -1/-1/0 84 LR 1/1/0/0/0 0/0

// httplog
// "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"
// 127.0.0.1:54848 [16/Oct/2023:02:29:52.943] test app_backend/<NOSRV> 0/-1/-1/-1/0 200 84 - - LR-- 1/1/0/0/0 0/0 "GET / HTTP/1.1"

// httplog clf
// "%{+Q}o %{-Q}ci - - [%trg] %r %ST %B \"\" \"\" %cp %ms %ft %b %s %TR %Tw %Tc %Tr %Ta %tsc %ac %fc %bc %sc %rc %sq %bq %CC %CS %hrl %hsl
// 127.0.0.1 - - [16/Oct/2023:01:05:56 +0000] "GET / HTTP/1.1" 200 84 "" "" 58628 319 "test" "app_backend" "<NOSRV>" 0 -1 -1 -1 0 LR-- 1 1 0 0 0 0 0 "" ""

// httpslog
// "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %[fc_err]/%[ssl_fc_err,hex]/%[ssl_c_err]/%[ssl_c_ca_err]/%[ssl_fc_is_resumed] %[ssl_fc_sni]/%sslv/%sslc"
// 127.0.0.1:57881 [16/Oct/2023:02:58:32.647] test app_backend/<NOSRV> 0/-1/-1/-1/0 200 84 - - LR-- 1/1/0/0/0 0/0 "GET / HTTP/1.1" 0/-/-/-/0 -/-/-

// Flags are :
//  * Q: quote a string
//  * X: hexadecimal representation (IPs, Ports, %Ts, %rt, %pid)
//  * E: escape characters '"', '\' and ']' in a string with '\' as prefix
//       (intended purpose is for the RFC5424 structured-data log formats)
//Example:
//log-format %T\ %t\ Some\ Text
//log-format %{+Q}o\ %t\ %s\ %{-Q}r
//
//log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
//Please refer to the table below for currently defined variables :
//
//  +---+------+------------------------------------------------------+---------+
//  | R | var  | field name (8.2.2 and 8.2.3 for description)         | type    |
//  |   |      | sample fetch alternative                             |         |
//  +===+======+======================================================+=========+
//  |   | %o   | special variable, apply flags on all next var        |         |
//  +---+------+------------------------------------------------------+---------+
//  |                          date formats                                     |
//  +---+------+------------------------------------------------------+---------+
//  |   | %T   | Accept date UTC + timezone                           |         |
//  |   |      | %[accept_date,utime("%d/%b/%Y:%H:%M:%S %z")]         | date    |
//  +---+------+------------------------------------------------------+---------+
//  |   | %Tl  | Accept date local + timezone                         |         |
//  |   |      | %[accept_date,ltime("%d/%b/%Y:%H:%M:%S %z")]         | date    |
//  +---+------+------------------------------------------------------+---------+
//  |   | %Ts  | Accept date as a UNIX timestamp                      | numeric |
//  +---+------+------------------------------------------------------+---------+
//  |   | %t   | Accept date local (with millisecond resolution)      |         |
//  |   |      | %[accept_date(ms),ms_ltime("%d/%b/%Y:%H:%M:%S.%3N")] | date    |
//  +---+------+------------------------------------------------------+---------+
//  |   | %ms  | Accept date milliseconds                             |         |
//  |   |      | %[accept_date(ms),ms_utime("%3N")                    | numeric |
//  +---+------+------------------------------------------------------+---------+
//  | H | %tr  | Request date local (with millisecond resolution)     |         |
//  |   |      | %[request_date(ms),ms_ltime("%d/%b/%Y:%H:%M:%S.%3N")]| date    |
//  +---+------+------------------------------------------------------+---------+
//  | H | %trg | Request date UTC + timezone                          |         |
//  |   |      | %[request_date,utime("%d/%b/%Y:%H:%M:%S %z")]        | date    |
//  +---+------+------------------------------------------------------+---------+
//  | H | %trl | Request date local + timezone                        |         |
//  |   |      | %[request_date,ltime("%d/%b/%Y:%H:%M:%S %z")]        | date    |
//  +---+------+------------------------------------------------------+---------+
//  |                          Timing events                                    |
//  +---+------+------------------------------------------------------+---------+
//  | H | %Ta  | Active time of the request (from TR to end)          |         |
//  |   |      | %[txn.timer.total]                                   | numeric |
//  +---+------+------------------------------------------------------+---------+
//  |   | %Tc  | Tc                                                   |         |
//  |   |      | %[bc.timer.connect]                                  | numeric |
//  +---+------+------------------------------------------------------+---------+
//  |   | %Td  | Td = Tt - (Tq + Tw + Tc + Tr)                        |         |
//  |   |      | %[res.timer.data]                                    | numeric |
//  +---+------+------------------------------------------------------+---------+
//  |   | %Th  | connection handshake time (SSL, PROXY proto)         |         |
//  |   |      | %[fc.timer.handshake]                                | numeric |
//  +---+------+------------------------------------------------------+---------+
//  | H | %Ti  | idle time before the HTTP request                    |         |
//  |   |      | %[req.timer.idle]                                    | numeric |
//  +---+------+------------------------------------------------------+---------+
//  | H | %Tq  | Th + Ti + TR                                         |         |
//  |   |      | %[req.timer.tq]                                      | numeric |
//  +---+------+------------------------------------------------------+---------+
//  | H | %TR  | time to receive the full request from 1st byte       |         |
//  |   |      | %[req.timer.hdr]                                     | numeric |
//  +---+------+------------------------------------------------------+---------+
//  | H | %Tr  | Tr (response time)                                   |         |
//  |   |      | %[res.timer.hdr]                                     | numeric |
//  +---+------+------------------------------------------------------+---------+
//  |   | %Tt  | Tt                                                   |         |
//  |   |      | %[fc.timer.total]                                    | numeric |
//  +---+------+------------------------------------------------------+---------+
//  |   | %Tu  | Tu                                                   |         |
//  |   |      | %[txn.timer.user]                                    | numeric |
//  +---+------+------------------------------------------------------+---------+
//  |   | %Tw  | Tw                                                   |         |
//  |   |      | %[req.timer.queue]                                   | numeric |
//  +---+------+------------------------------------------------------+---------+
//  |                          Others                                           |
//  +---+------+------------------------------------------------------+---------+
//  |   | %B   | bytes_read           (from server to client)         | numeric |
//  |   |      | %[bytes_out]                                         |         |
//  +---+------+------------------------------------------------------+---------+
//  | H | %CC  | captured_request_cookie                              | string  |
//  +---+------+------------------------------------------------------+---------+
//  | H | %CS  | captured_response_cookie                             | string  |
//  +---+------+------------------------------------------------------+---------+
//  |   | %H   | hostname                                             | string  |
//  +---+------+------------------------------------------------------+---------+
//  | H | %HM  | HTTP method (ex: POST)                               | string  |
//  +---+------+------------------------------------------------------+---------+
//  | H | %HP  | HTTP request URI without query string                | string  |
//  +---+------+------------------------------------------------------+---------+
//  | H | %HPO | HTTP path only (without host nor query string)       | string  |
//  +---+------+------------------------------------------------------+---------+
//  | H | %HQ  | HTTP request URI query string (ex: ?bar=baz)         | string  |
//  |   |      | ?%[query]                                            |         |
//  +---+------+------------------------------------------------------+---------+
//  | H | %HU  | HTTP request URI (ex: /foo?bar=baz)                  | string  |
//  +---+------+------------------------------------------------------+---------+
//  | H | %HV  | HTTP version (ex: HTTP/1.0)                          | string  |
//  |   |      | HTTP/%[req.ver]                                      |         |
//  +---+------+------------------------------------------------------+---------+
//  |   | %ID  | unique-id                                            | string  |
//  +---+------+------------------------------------------------------+---------+
//  |   | %ST  | status_code                                          | numeric |
//  +---+------+------------------------------------------------------+---------+
//  |   | %U   | bytes_uploaded       (from client to server)         | numeric |
//  |   |      | %[bytese]                                          |         |
//  +---+------+------------------------------------------------------+---------+
//  |   | %ac  | actconn                                              |         |
//  |   |      | %[act_conn]                                          | numeric |
//  +---+------+------------------------------------------------------+---------+
//  |   | %b   | backend_name                                         |         |
//  |   |      | %[be_name]                                           | string  |
//  +---+------+------------------------------------------------------+---------+
//  |   | %bc  | beconn      (backend concurrent connections)         | numeric |
//  +---+------+------------------------------------------------------+---------+
//  |   | %bi  | backend_source_ip       (connecting address)         |         |
//  |   |      | %[bc_src]                                            | IP      |
//  +---+------+------------------------------------------------------+---------+
//  |   | %bp  | backend_source_port     (connecting address)         |         |
//  |   |      | %[bc_src_port]                                       | numeric |
//  +---+------+------------------------------------------------------+---------+
//  |   | %bq  | backend_queue                                        | numeric |
//  +---+------+------------------------------------------------------+---------+
//  |   | %ci  | client_ip                 (accepted address)         |         |
//  |   |      | %[src]                                               | IP      |
//  +---+------+------------------------------------------------------+---------+
//  |   | %cp  | client_port               (accepted address)         |         |
//  |   |      | %[src_port]                                          | numeric |
//  +---+------+------------------------------------------------------+---------+
//  |   | %f   | frontend_name                                        | string  |
//  +---+------+------------------------------------------------------+---------+
//  |   | %fc  | feconn     (frontend concurrent connections)         | numeric |
//  +---+------+------------------------------------------------------+---------+
//  |   | %fi  | frontend_ip              (accepting address)         |         |
//  |   |      | %[dst]                                               | IP      |
//  +---+------+------------------------------------------------------+---------+
//  |   | %fp  | frontend_port            (accepting address)         |         |
//  |   |      | %[dst_port]                                          | numeric |
//  +---+------+------------------------------------------------------+---------+
//  |   | %ft  | frontend_name_transport ('~' suffix for SSL)         | string  |
//  +---+------+------------------------------------------------------+---------+
//  |   | %lc  | frontend_log_counter                                 | numeric |
//  +---+------+------------------------------------------------------+---------+
//  |   | %hr  | captured_request_headers default style               | string  |
//  +---+------+------------------------------------------------------+---------+
//  |   | %hrl | captured_request_headers CLF style                   | string  |
//  |   |      |                                                      | list    |
//  +---+------+------------------------------------------------------+---------+
//  |   | %hs  | captured_response_headers default style              | string  |
//  +---+------+------------------------------------------------------+---------+
//  |   | %hsl | captured_response_headers CLF style                  | string  |
//  |   |      |                                                      | list    |
//  +---+------+------------------------------------------------------+---------+
//  |   | %pid | PID                                                  |         |
//  |   |      | %[pid]                                               | numeric |
//  +---+------+------------------------------------------------------+---------+
//  | H | %r   | http_request                                         | string  |
//  +---+------+------------------------------------------------------+---------+
//  |   | %rc  | retries                                              | numeric |
//  +---+------+------------------------------------------------------+---------+
//  |   | %rt  | request_counter (HTTP req or TCP session)            | numeric |
//  +---+------+------------------------------------------------------+---------+
//  |   | %s   | server_name                                          | string  |
//  +---+------+------------------------------------------------------+---------+
//  |   | %sc  | srv_conn     (server concurrent connections)         | numeric |
//  +---+------+------------------------------------------------------+---------+
//  |   | %si  | server_IP                   (target address)         |         |
//  |   |      | %[bc_dst]                                            | IP      |
//  +---+------+------------------------------------------------------+---------+
//  |   | %sp  | server_port                 (target address)         |         |
//  |   |      | %[bc_dst_port]                                       | numeric |
//  +---+------+------------------------------------------------------+---------+
//  |   | %sq  | srv_queue                                            | numeric |
//  +---+------+------------------------------------------------------+---------+
//  | S | %sslc| ssl_ciphers (ex: AES-SHA)                            |         |
//  |   |      | %[ssl_fc_cipher]                                     | string  |
//  +---+------+------------------------------------------------------+---------+
//  | S | %sslv| ssl_version (ex: TLSv1)                              |         |
//  |   |      | %[ssl_fc_protocol]                                   | string  |
//  +---+------+------------------------------------------------------+---------+
//  |   | %ts  | termination_state                                    | string  |
//  +---+------+------------------------------------------------------+---------+
//  | H | %tsc | termination_state with cookie status                 | string  |
//  +---+------+------------------------------------------------------+---------+
//
//    R = Restrictions : H = mode http only ; S = SSL only

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions