NUT swarm stress testing: revise cleanup of disconnected sockets

[jimklimov]

In testing I also saw something concerning:

• in at least Linux, clients (upslog here) were apparently piling up and exceeding the MAXCONN value embedded into the generated tests/NIT/tmp/etc/upsd.conf file (two swarm sizes plus 30 to feel generous). Maybe they were retrying connections as their earlier attempts failed (due to initial bug, may still be possible when upsd is overwhelmed), compounding the issue: upsd remembers older connection attempts, earlier in its list, and until they time out, it has little chance of seeing newer ones. Not sure OTOH if "connection closed/timed out" is evaluated at that time, or it is only fast-tracked for drivers.

Originally posted by @jimklimov in #3302

Upon a quick look at upsd.c mainloop(), there are checks for FD validity when adding them or not to the list of sockets to check for activity. This seems to be a "static" check based on a previously known disconnection, and so a saved "invalid" FD/handle value. Clients are not looked at after 60 seconds of inactivity (hard-coded).

Maybe the fixes done for issue #3302 and/or #3365 would cover this more timely detection of disconnections well enough to make the point moot, but if not (so such "leak" or piling-up remains), keep this in mind. Maybe detection of disconnections should be more aggressive somehow.

[jimklimov]

Investigated, not a leak but overlooked behavior in test: upslog is used with * to monitor all UPSes on a server, so each client process established that many libupsclient connections.

It may be or not be reasonable to have an upslog monitoring multiple devices from the same server re-use the connection (see #1639, #2756 etc.), but that is a separate issue. => #3374

[jimklimov]

Still, it may make sense to split the walk over sockets highlighted by poll()/WaitForMultipleObjects() to first close all that are easy to identify as disconnected, so as to invalidate associated file descriptors and avoid checking them next time, and then re-walk the active FDs where some time-consuming work with dialog and remote processes may be involved.

With work on PR #3368 there still are cases when the swarm activity of ~1k connections precludes new clients from connecting in practice, despite there being available slots under the "MAXCONN" allowance on upsd.

It was also noted that when upsc timeout was set to 60 seconds (via envvar or -W 60 on command line, see #2847) it was waiting for some time but 50/50 if it would spend a minute or abort the initial connection attempt earlier and log it at 5-something seconds.