HDDS-11234. Manage Netty native memory consumption

[smengcl]

What changes were proposed in this pull request?

Adds two opt-in env vars in ozone_java_setup to cap Netty's pooled direct memory:

• OZONE_NETTY_MAX_DIRECT_MEMORY sets -Dio.netty.maxDirectMemory=<bytes> (unshaded io.netty, used by S3G/gRPC).
• OZONE_RATIS_NETTY_MAX_DIRECT_MEMORY sets -Dorg.apache.ratis.thirdparty.io.netty.maxDirectMemory=<bytes> (Ratis-shaded copy, used by DN write/replication).

Two variables are needed because the unshaded and Ratis-shaded Netty classes have independent allocators, each defaulting to ≈ -Xmx — so the implicit per-process ceiling is roughly 2 × -Xmx of direct memory.

Both env vars are unset by default. No behavior change unless operators set them.

Companion docs PR in apache/ozone-site adds docs/06-troubleshooting/17-netty-direct-memory.md: apache/ozone-site#448

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11234

How was this patch tested?

• Manually confirmed each -D lands in the correct OPTS var.

[jojochuang]

we might be able to simplify it after #10030 which supposedly migrates gRPC usage to Ratis-shaded gRPC.

[adoroszlai]

#10030 is too big and should be split

#10030 (review)
#10030 (comment)

[smengcl]

Ah I misunderstood earlier. Let me reopen this PR

[smengcl]

I'm closing this one atm to wait for #10030 to land then. cc @yandrey321

will land this first, then tweak when #10030 is done

[Copilot]

Pull request overview

Adds opt-in configuration to cap Netty pooled direct (native) memory usage in Ozone daemons by wiring environment variables into JVM -D...maxDirectMemory flags (both unshaded Netty and the Ratis-shaded Netty copy).

Changes:

• Add OZONE_NETTY_MAX_DIRECT_MEMORY and OZONE_RATIS_NETTY_MAX_DIRECT_MEMORY handling in ozone_java_setup to set the corresponding Netty system properties.
• Document the new env vars and expected byte-value format in ozone-env.sh.

Reviewed changes

Copilot reviewed 1 out of 2 changed files in this pull request and generated no comments.

File	Description
hadoop-ozone/dist/src/shell/ozone/ozone-functions.sh	Appends Netty max direct memory system properties to OZONE_OPTS / RATIS_OPTS based on new env vars.
hadoop-hdds/common/src/main/conf/ozone-env.sh	Adds operator-facing documentation for the new Netty direct memory cap env vars.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[yandrey321]

We need to document these params and have a guide for how to use them.

[adoroszlai]

We need to document these params and have a guide for how to use them.

@yandrey321 see https://github.com/apache/ozone-site/pull/448/files

[adoroszlai]

Thanks @smengcl for the patch, @jojochuang, @yandrey321 for the review.

[smengcl]

Thanks @jojochuang @adoroszlai @yandrey321 for reviewing this.