diff --git a/CHANGELOG.md b/CHANGELOG.md index 72adff8..cf5c5e7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,30 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## Unreleased + +### Added +- `Zexbox.Logging.JsonFormatter.config/1` (and the + `Zexbox.Logging.json_formatter_config/1` delegate) — returns a + `:logger` formatter tuple wrapping `LoggerJSON.Formatters.Basic` with + Zappi-standard metadata and redactor defaults. Designed to be splatted + into `config :logger, :default_handler, formatter: ...` in + `runtime.exs`. Mirrors the Ruby-side opsbox `JsonFormatter` so Phoenix + logs land in Elasticsearch as one structured document per event + instead of fanning multi-line content out into many. + +### Changed +- `:elixir` constraint bumped from `~> 1.14` to `~> 1.15` to match the + `logger_json` 7.x requirement. + +### Dependencies +- Adds `logger_json ~> 7.0`. + +### Notes +- This release does not pin `:jason`. Consumers may include it directly + (the default `LoggerJSON` encoder) or, on Elixir 1.18+, set + `config :logger_json, encoder: JSON` to use the stdlib `JSON` module. + ## 1.5.1 - 2026-02-05 - Handles cases where `$callers` and `$ancestors` may not be pids to avoid crashing metric handler. diff --git a/README.md b/README.md index 7b01eab..a74f08a 100644 --- a/README.md +++ b/README.md @@ -143,6 +143,66 @@ Adding your own logs is as simple as calling the `Zexbox.Telementry.attach/4` (w Zexbox.Telemetry.attach(:my_event, [:my, :event], &MyAppHandler.my_handler/3, nil) ``` +### JSON-formatted logs for Logstash / Elasticsearch + +By default the Elixir Logger emits multi-line plain-text output. Filebeat +ships every line as a separate Elasticsearch document, so a single struct +inspection or stack trace can fan out into dozens of indexed docs. +`Zexbox.Logging.JsonFormatter` returns a `:logger` formatter tuple that +swaps the default formatter for a JSON one — every log event becomes a +single line of JSON, so multi-line content collapses into one ES +document at the ingest layer. This mirrors the behaviour of opsbox's +`JsonFormatter` on the Ruby side. + +The configuration is declarative — `:logger` is configured before +`Application.start/2` runs, so logs emitted during application boot use +the formatter that has been *configured*, not one installed at runtime. + +In `config/runtime.exs`: + +```elixir +import Config + +if config_env() == :prod do + config :logger, :default_handler, + formatter: Zexbox.Logging.json_formatter_config() +end +``` + +Output (one line per event): + +```json +{"time":"2026-05-02T01:23:45.678Z","severity":"info","message":"Hello","metadata":{...}} +``` + +To override the metadata allow-list or redactors: + +```elixir +config :logger, :default_handler, + formatter: + Zexbox.Logging.json_formatter_config( + metadata: [:request_id, :trace_id, :user_id], + redactors: [] + ) +``` + +The defaults include a `RedactKeys` redactor stripping common credential +metadata keys; see `Zexbox.Logging.JsonFormatter.default_metadata/0` and +`default_redactors/0` for the full lists. + +### Encoder choice + +`logger_json` uses Jason as the JSON encoder by default. On Elixir 1.18+ +you can opt into the stdlib `JSON` module — set this in +`config/config.exs` (compile-time): + +```elixir +config :logger_json, encoder: JSON +``` + +Otherwise add `{:jason, "~> 1.4"}` to your application's deps. Zexbox +intentionally does not pin an encoder so consumers can choose. + ## Metrics In order to setup metrics with InfluxDB you'll need to add the following configuration: diff --git a/lib/zexbox/logging.ex b/lib/zexbox/logging.ex index 758b508..a589405 100644 --- a/lib/zexbox/logging.ex +++ b/lib/zexbox/logging.ex @@ -3,9 +3,29 @@ defmodule Zexbox.Logging do Module for logging events in Zexbox. """ - alias Zexbox.Logging.LogHandler + alias Zexbox.Logging.{JsonFormatter, LogHandler} alias Zexbox.Telemetry + @doc """ + Returns a `:logger` formatter tuple wrapping `LoggerJSON.Formatters.Basic` + with Zappi-standard defaults. Designed to be splatted into + `config :logger, :default_handler, formatter: ...` in `config/runtime.exs`: + + if config_env() == :prod do + config :logger, :default_handler, + formatter: Zexbox.Logging.json_formatter_config() + end + + Mirrors opsbox's Ruby-side `JsonFormatter`: every log event becomes one + JSON object on one line, so multi-line content (struct inspections, + multi-line SQL, stack traces) collapses into a single Elasticsearch + document at the ingest layer. + + See `Zexbox.Logging.JsonFormatter` for the full option list. + """ + @spec json_formatter_config(keyword()) :: {module(), map()} + defdelegate json_formatter_config(opts \\ []), to: JsonFormatter, as: :config + @doc """ Attaches Telemetry handlers for Phoenix controller events. diff --git a/lib/zexbox/logging/json_formatter.ex b/lib/zexbox/logging/json_formatter.ex new file mode 100644 index 0000000..0a903e4 --- /dev/null +++ b/lib/zexbox/logging/json_formatter.ex @@ -0,0 +1,89 @@ +defmodule Zexbox.Logging.JsonFormatter do + @moduledoc """ + Returns a `:logger` formatter tuple suitable for Logstash / Elasticsearch + ingestion, wrapping `LoggerJSON.Formatters.Basic` with Zappi-standard + defaults. + + Splat into `config/runtime.exs`: + + config :logger, :default_handler, + formatter: Zexbox.Logging.JsonFormatter.config() + + Mirrors opsbox's Ruby-side `JsonFormatter`: every log event becomes one + JSON object on one line, so multi-line content (struct inspections, + multi-line SQL, stack traces) collapses into a single Elasticsearch + document at the ingest layer. + + ## Why declarative + + `:logger` is configured before `Application.start/2` runs. Logs emitted + during application boot use the formatter that has been *configured* — + not one installed imperatively at runtime. Pure config makes JSON + active from the first log line emitted by the BEAM. + + ## Encoder + + Consumers choose the JSON encoder. The default for `LoggerJSON` is + Jason; on Elixir 1.18+ you can opt into the stdlib `JSON` module with: + + config :logger_json, encoder: JSON + + This is a compile-time setting in `config/config.exs`. + """ + + alias LoggerJSON.{Formatters.Basic, Redactors.RedactKeys} + + @default_metadata [ + :request_id, + :trace_id, + :span_id, + :user_id, + :pid, + :module, + :function, + :line, + :application + ] + + @default_redactors [ + {RedactKeys, ["password", "secret", "token", "authorization", "api_key", "session"]} + ] + + @doc """ + Returns the formatter tuple for `config :logger, :default_handler, formatter: ...`. + + ## Options + + * `:metadata` — list of `Logger` metadata keys to include in the JSON + output, or `:all` to include every key set via `Logger.metadata/1`. + Defaults to a curated Zappi list (see `default_metadata/0`). Pass + an explicit list to override; metadata is a security-adjacent + surface and `:all` is **not** recommended for production. + + * `:redactors` — list of `LoggerJSON.Redactor` tuples. Defaults to a + single `RedactKeys` redactor stripping common credential keys + (see `default_redactors/0`). Pass `[]` to disable redaction; pass + a list to extend or override. + + ## Examples + + iex> {mod, _config} = Zexbox.Logging.JsonFormatter.config() + iex> mod + LoggerJSON.Formatters.Basic + """ + @spec config(keyword()) :: {module(), map()} + def config(opts \\ []) do + Basic.new( + metadata: Keyword.get(opts, :metadata, @default_metadata), + redactors: Keyword.get(opts, :redactors, @default_redactors) + ) + end + + @doc "The default metadata allow-list, exposed for inspection or extension." + @spec default_metadata() :: list(atom()) + def default_metadata, do: @default_metadata + + @doc "The default redactor list, exposed for inspection or extension." + @spec default_redactors() :: list({module(), term()}) + def default_redactors, do: @default_redactors +end diff --git a/mix.exs b/mix.exs index 5e5dcb2..ee3a439 100644 --- a/mix.exs +++ b/mix.exs @@ -5,7 +5,7 @@ defmodule Zexbox.MixProject do [ app: :zexbox, version: "1.5.1", - elixir: "~> 1.14", + elixir: "~> 1.15", start_permanent: Mix.env() == :prod, dialyzer: [plt_add_apps: [:mix, :ex_unit]], description: description(), @@ -38,6 +38,7 @@ defmodule Zexbox.MixProject do {:ex_doc, "~> 0.35.1", only: :dev, runtime: false}, {:instream, "~> 2.2"}, {:ldclient, "~> 3.8.0", hex: :launchdarkly_server_sdk}, + {:logger_json, "~> 7.0"}, {:mix_audit, "~> 2.0", only: [:dev, :test], runtime: false}, {:mock, "~> 0.3.0", only: :test}, {:sobelow, "~> 0.8", only: [:dev, :test]}, diff --git a/mix.lock b/mix.lock index f3472fc..f5170de 100644 --- a/mix.lock +++ b/mix.lock @@ -19,6 +19,7 @@ "jason": {:hex, :jason, "1.4.4", "b9226785a9aa77b6857ca22832cffa5d5011a667207eb2a0ad56adb5db443b8a", [:mix], [{:decimal, "~> 1.0 or ~> 2.0", [hex: :decimal, repo: "hexpm", optional: true]}], "hexpm", "c5eb0cab91f094599f94d55bc63409236a8ec69a21a67814529e8d5f6cc90b3b"}, "jsx": {:hex, :jsx, "3.1.0", "d12516baa0bb23a59bb35dccaf02a1bd08243fcbb9efe24f2d9d056ccff71268", [:rebar3], [], "hexpm", "0c5cc8fdc11b53cc25cf65ac6705ad39e54ecc56d1c22e4adb8f5a53fb9427f3"}, "ldclient": {:hex, :launchdarkly_server_sdk, "3.8.0", "4e900c33cfa9fdcd80f01de371d1108eb8f3834f41a1c56d1f739597a6d8bf57", [:rebar3], [{:certifi, "~> 2.14", [hex: :certifi, repo: "hexpm", optional: false]}, {:eredis, "1.7.1", [hex: :eredis, repo: "hexpm", optional: false]}, {:jsx, "3.1.0", [hex: :jsx, repo: "hexpm", optional: false]}, {:lru, "2.4.0", [hex: :lru, repo: "hexpm", optional: false]}, {:shotgun, "1.2.1", [hex: :shotgun, repo: "hexpm", optional: false]}, {:uuid, "~> 2.0.2", [hex: :uuid_erl, repo: "hexpm", optional: false]}, {:verl, "1.0.1", [hex: :verl, repo: "hexpm", optional: false]}, {:yamerl, "0.10.0", [hex: :yamerl, repo: "hexpm", optional: false]}], "hexpm", "015deb283e9e964ebd02cc3e95c21828ff41ce62b6115793d4e93c01cd6af661"}, + "logger_json": {:hex, :logger_json, "7.0.4", "e315f2b9a755504658a745f3eab90d88d2cd7ac2ecfd08c8da94d8893965ab5c", [:mix], [{:decimal, ">= 0.0.0", [hex: :decimal, repo: "hexpm", optional: true]}, {:ecto, "~> 3.11", [hex: :ecto, repo: "hexpm", optional: true]}, {:jason, "~> 1.4", [hex: :jason, repo: "hexpm", optional: true]}, {:plug, "~> 1.15", [hex: :plug, repo: "hexpm", optional: true]}, {:telemetry, "~> 1.0", [hex: :telemetry, repo: "hexpm", optional: true]}], "hexpm", "d1369f8094e372db45d50672c3b91e8888bcd695fdc444a37a0734e96717c45c"}, "lru": {:hex, :lru, "2.4.0", "a8f9967ca9b6f260baa19e2efb2aeb3853a3f5bd5f8416f537a672294b38c1bc", [:rebar3], [], "hexpm", "4fcf77e882b5e57eca068999acba4386a20dbce8e446c98c6a0f8fb3d170afeb"}, "makeup": {:hex, :makeup, "1.2.1", "e90ac1c65589ef354378def3ba19d401e739ee7ee06fb47f94c687016e3713d1", [:mix], [{:nimble_parsec, "~> 1.4", [hex: :nimble_parsec, repo: "hexpm", optional: false]}], "hexpm", "d36484867b0bae0fea568d10131197a4c2e47056a6fbe84922bf6ba71c8d17ce"}, "makeup_elixir": {:hex, :makeup_elixir, "1.0.1", "e928a4f984e795e41e3abd27bfc09f51db16ab8ba1aebdba2b3a575437efafc2", [:mix], [{:makeup, "~> 1.0", [hex: :makeup, repo: "hexpm", optional: false]}, {:nimble_parsec, "~> 1.2.3 or ~> 1.3", [hex: :nimble_parsec, repo: "hexpm", optional: false]}], "hexpm", "7284900d412a3e5cfd97fdaed4f5ed389b8f2b4cb49efc0eb3bd10e2febf9507"}, diff --git a/test/zexbox/logging/json_formatter_test.exs b/test/zexbox/logging/json_formatter_test.exs new file mode 100644 index 0000000..2de2950 --- /dev/null +++ b/test/zexbox/logging/json_formatter_test.exs @@ -0,0 +1,34 @@ +defmodule Zexbox.Logging.JsonFormatterTest do + use ExUnit.Case, async: true + + alias Zexbox.Logging.JsonFormatter + + describe "config/1" do + test "returns LoggerJSON.Formatters.Basic with sensible defaults" do + assert {LoggerJSON.Formatters.Basic, opts} = JsonFormatter.config() + assert :request_id in opts.metadata + assert :user_id in opts.metadata + refute opts.redactors == [] + end + + test "metadata override is honoured" do + assert {_module, %{metadata: [:foo, :bar]}} = + JsonFormatter.config(metadata: [:foo, :bar]) + end + + test "redactors override is honoured" do + assert {_module, %{redactors: []}} = JsonFormatter.config(redactors: []) + end + + test "default lists are exposed for inspection" do + assert :request_id in JsonFormatter.default_metadata() + assert [{LoggerJSON.Redactors.RedactKeys, _keys}] = JsonFormatter.default_redactors() + end + end + + describe "Zexbox.Logging.json_formatter_config/1 delegate" do + test "returns the same config" do + assert Zexbox.Logging.json_formatter_config() == JsonFormatter.config() + end + end +end