egress-auditor

egress-auditor will monitor new outbound connections and generate appropriate iptables rules (or logs, or ... depending on output plugin).

Connections can be detected using several methods.

The goal is to audit what connections your system need to establish outside, and this tool can help in several scenarios:

• audit egress connections and automatically display required iptables rules to allow these connections;

• when rules are in place, log all egress connections just before they are dropped by the chain (explicitly or by the default policy) and send alerts based on loki queries for instance

• let only connections be established if they are initiated by specific processes (TDB, requires nfqueue)

This is alpha stuff.

Quick start

# add iptable rules on OUTPUT to send new connections to NFLOG
sudo iptables -I OUTPUT -m state --state NEW -p tcp -j NFLOG --nflog-group 100
sudo iptables -I OUTPUT -m state --state NEW -p udp -j NFLOG --nflog-group 100
go build . 
# start egress-auditor using the nflog input and the same group id used in iptables
sudo ./egress-auditor -i nflog -I nflog:group:100 -o iptables -O iptables:verbose:2
egress-auditor is running... press ctrl-c to stop
new TCP connection 192.168.1.229:60166 -> 146.148.13.123:443(https) by curl
^C # <- Ctrl+C pressed here
# [nflog] Line generated for curl running as ubuntu with command "curl https://www.devops.works"
# [nflog] Parent of this process was bash running as ubuntu
iptables -I OUTPUT -d 146.148.13.123 -p tcp -m tcp --dport 443 -j ACCEPT -m comment --comment "curl"

If you use nftables, you can set-up nflog target like so:

nft add rule filter output ct state new log group 100 accept
nft add rule ip6 filter output ct state new log group 100 accept

or:

nft add rule ip filter OUTPUT ct state new log group 100 accept
nft add rule ip6 filter OUTPUT ct state new log group 100 accept

depending on your tables setup.

Usage

See -h for help, and -l for the list of input/output plugins.

In a nutshell, inputs are added using -i, outputs using -o.

If a plugin needs an option, they are passed using -I for inputs and -O for outputs. For those options, the required format is pluginame:optionname:optionvalue.

For instance, to set verbosity to 2 for the iptables output plugin, the proper invocation is:

... -O iptables:verbose:2

Of course, this implies the iptables output module has been loaded using -o iptables in the same CLI.

The -R option can be used to hide egress-auditor and it's arguments from ps output. This allows for more sneaky auditing, preventing someone to spot the program too easily and kill it.

For instance, when running:

sudo ./egress-auditor -R '[loop25]' ...

a call to ps auwx | grep egress | grep -v grep won't return any results, since the process has been renamed to [loop25] (and hangs out with its other loop kernel-threads friends).

Building

go build .

If you're lazy and do not want to type sudo when running egress-auditor, you can give it some capabilities:

sudo setcap 'cap_net_admin=+ep' ./egress-auditor 

Loki stack

If you want to play with egress captured logs in loki, you can start a docker-compose stack in the _misc directory, then point egress-auditor at loki.

cd _misc
docker-compose up -d
cd ..
sudo iptables -I OUTPUT -m state --state NEW -p tcp -j NFLOG --nflog-group 100
sudo iptables -I OUTPUT -m state --state NEW -p udp -j NFLOG --nflog-group 100
sudo ./egress-auditor -i nflog -I nflog:group:100 -o loki -O loki:url:http://127.0.0.1:3100 -O loki:labels:test=true,lokirules=yes,fizz=buzz

Then :

• login with admin:admin,
• create a datasource with type 'Loki' and URL http://loki:3100
• click save and test, and got to the Explore panel to start playing, using this query for instance: sum by (process) (rate({destport="443"} [1m]))

Please note that loging to a remote Loki server might generate log lines for Loki itself. You might want to prevent this trafic to be matched in the NFLOG rule.

Available modules

Run egress-auditor -l to get an up to date list and their options.

Inputs

• nflog: captures using nflog iptable target
• ebpf: captures using kprobes on tcp_*_connect / udp[v6]_sendmsg
• nfqueue (+ auto-allow using process filters)
• pcap (device + file, no proc info for the latter)

Outputs

• iptables
• loki
• logfmt (stdout or file, with SIGHUP support for log rotation)

eBPF input

The ebpf input is an alternative to nflog that does not require any iptables/nftables rules. It attaches kprobes to tcp_v4_connect, tcp_v6_connect, udp_sendmsg, and udpv6_sendmsg, capturing the originating process PID directly from the kernel context. This eliminates the race against /proc/net/tcp that nflog suffers from for short-lived processes.

sudo ./egress-auditor -i ebpf -o logfmt

Requirements:

• Linux kernel with kprobe support (any 4.x+)
• CAP_BPF and CAP_PERFMON (or root) at runtime
• To build the eBPF object code: clang and libbpf-dev. After cloning, run go generate ./internal/inputs/ebpf/ to compile the eBPF program and generate the Go bindings. The resulting bpf_bpfel.go, bpf_bpfeb.go, bpf_bpfel.o, bpf_bpfeb.o files are committed in the repo, so plain go build works without clang.

Options:

• -I ebpf:quiet:true — suppress per-connection messages on stderr
• -I ebpf:allow-loopback:true — include loopback traffic
• -I ebpf:ignore-cidr:<CIDR> — drop events whose dest IP is in this network (may be repeated, IPv4 or IPv6)
• -I ebpf:ignore-port:<port> — drop events with this dest port (may be repeated)
• -I ebpf:ignore-comm:<name> — drop events from this process name (matched against the resolved /proc name; may be repeated; supports glob wildcards *, ?, [...] — e.g. chrome*, *-worker)
• -I ebpf:ignore-cmdline:<pattern> — drop events whose full command line matches this pattern (substring match by default; with glob wildcards * crosses any character including / — e.g. */unbound*, syncthing)
• -I ebpf:ignore-parent:<name> — drop events whose parent process name matches (same syntax as ignore-comm: exact or glob)

Filtering: nflog vs ebpf

With nflog, you bypass uninteresting traffic by simply not matching it in iptables (the kernel's packet classifier handles it for free):

iptables -I OUTPUT -d 10.0.0.0/8 -j ACCEPT
iptables -A OUTPUT -m state --state NEW -p tcp -j NFLOG --nflog-group 100

With ebpf there is no equivalent kernel classifier — kprobes fire for every outgoing connection — so the same intent is expressed via ignore-* options:

sudo ./egress-auditor -i ebpf -o logfmt \
    -I ebpf:ignore-cidr:10.0.0.0/8 \
    -I ebpf:ignore-cidr:192.168.0.0/16 \
    -I ebpf:ignore-cidr:fe80::/10 \
    -I ebpf:ignore-port:53 \
    -I ebpf:ignore-comm:chronyd

Filtering happens in Go userspace after the eBPF event is read. For an audit tool this is fine; if you need wire-speed filtering for very high event rates, the natural next step is to push these rules into the eBPF program via BPF maps (LPM trie for CIDRs, hash for ports). Not implemented yet.

Logfmt output and log rotation

The logfmt output writes one line per connection in logfmt format. By default it writes to stdout, but can be directed to a file:

sudo ./egress-auditor -i nflog -I nflog:group:100 -I nflog:quiet:true -o logfmt -O logfmt:file:/var/log/egress.log

Use -I nflog:quiet:true to suppress the per-connection messages that nflog prints to stderr, since the logfmt output already captures all connection details.

When writing to a file, the logfmt output handles SIGHUP by closing and reopening the file. This makes it compatible with logrotate. Example logrotate configuration:

/var/log/egress.log {
    daily
    rotate 7
    compress
    missingok
    notifempty
    postrotate
        kill -HUP $(pidof egress-auditor) 2>/dev/null || true
    endscript
}

Caveats

• use -I nflog:allow-loopback:true to consider loopback directed traffic
• when using nflog, originating process might not be found for really short lived connections

TODO

• PTR lookups on destination ?
• pass down a logger to prevent logging mess
• -C : how many cnx to capture before bailing out
• -t: duration to capture before exiting
• -debug

Licence

MIT

Contributions welcome.