Add behaviors to independent shellcommand to allow for status of 'error'

[vanderpol]

In the current design of the independent shellcommand, any errors that occur when running arbitrary system commands are captured as part of standard out, but have no impact the status of the collected item. This is by design, as certain commands, such as a find command may return back a bit of noise.

Proposal: Add a behavior to give the content author control of what standard errors mean, and if they should be displayed

For the behavior, some options are

• Include_stderror (include standard error in the system data item, but it has no impact on the item's status, default)
• Ignore_stderror (exclude all stderror messages, same as if command redirected stderror /dev/null)
• Item_error_if_stderror (include standard error messages in the system data item, AND mark the system data item status as 'error')

We could also give the content author complete control and put in a stderror pattern in the object to allow content authors to filter and only see certain errors as they see fit.

Thoughts?

[vanderpol]

@solind and @A-Biggs - Some backstory on this discussion. We are writing Exchange 2019 content right now, and the cmdlet for Exchange does not function remotely, even when via remote WMI spawned application, or via WindowsRM, Powershell Remote etc.. they all end up in an error "Active Directory operation failed on. The supplied credential for <domain\user> is invalid>

So remotely all of the these shell command return 'false', as no data is collected from the system. We would like to figure out a way to get this to 'error', but to be consistent across OVAL interpreters, it seems a schema change is required.

[solind]

I recall feeling strongly that the item needed to have stderr lines that were distinct from the stdout lines, because they are completely different character streams and where one result will appear to be interleaved within another is completely the result of non-deterministic race conditions. I'm guessing that's not what was decided, and now everyone can appreciate my wisdom LOL

[vanderpol]

We went with probe design by committee on this, and did not go with our initial design. One item is always created, with 0-N stdout_line and 0-N stderr_line, so the error is not specifically related to the std_out. The question at hand here, is how to get the status to 'error' for the collected item. The current design just logs all std_error as elements of the item, but there is no way to know if the existence of stderr_line data should warrant the item having a status of 'error'. The goal here is more control to the content author to more accurately report on what happened.

[solind]

Oh, ok. One way to force an error to happen is to create a var_ref that points to a non-existing object. So, you could use a variable and a variable_object (or a var_ref in some object or state) to generate an error condition.

Is there a particular error code that is returned as the command's exit code (and is the exit code captured)?

[vanderpol]

Powershell has an annoying way of not returning back exit_status either when it's successful or failed... We get back 9 lines of long winded stderor text, which are captured as 9 stderr_line elements.

As right now our application is the only product that fully supports this test, it seems like the perfect time to make any minor changes to the spec, that prevents making complex content that forces something that we could just mandate in the applications when they complete SCAP 1.4 self assertion.

[solind]

Is it the only product that supports the ntuser_test? I'm pretty sure Joval supports it in all its problematic glory. The difference being it's OVAL 5.11.2 and not 5.12, but this would not really be in the spirit of a minor change.

[vanderpol]

I think you might be crossing wires on different issues, this one is about the new independent shellcommand test, of which SCC might be the only implementation so far. As for the ntuser test, (the other issue), all tools that support scap 1.3 support it, and there have only been minor changes since oval 5.11.2.

For the shellcommand, I'll propose a behavior that leaves the default like it currently is, with the option to interpret stderror to be critical enough to error the item, or expected to the point of ignoring it entirely.

[solind]

Ugh, you're right. Wrong thread, or wrong headspace!

There's got to be a way to get the exit code. What if you echo %ERRORLEVEL% or something like that?

[vanderpol]

I'm including @wilsonbryanl on this conversation. I'm pretty sure this a well documented issue online, and from my vague recollection it has something to do with cmdlets vis native executable calls. Here's some documentation: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_error_handling?view=powershell-7.6

And a key snippet:
$LASTEXITCODE
Contains the exit code of the last native program that ran. A value of 0 conventionally indicates success. Any non-zero value indicates failure. This variable isn't affected by PowerShell cmdlet errors.

Given that the exit status is not reliable, the existence of stderror is all we can trust at this point, and why this proposed behavior is linked to stderr vs exit_status. I'm by no means set on the names/values for the behaviors and @solind you tend to offer up good suggestions, feel free to propose some.

[vanderpol]

In internal discussions, we likely will need to add two behaviors

error_if_exit_status_not_0: Boolean, default = false
error_if_stderr_exists: Boolean, default = false

This will allow the content author to control when (or if) the shellcommand item ever returns a status of error. We decided to make 2 Booleans as a content author may want to error if either criteria is met, which would make a single behavior more complex.

[wilsonbryanl]

BW - A few notes on PowerShell's Exit Status. The shell command test does gather the value of the $LASTEXITCODE after the content defined command is run. However this value is often either not defined or simply set to 0 which represents success. One prime example is if a cmdlet doesn't exist, no $LASTEXITCODE is set meaning an exist exit_status of null is reported. Please read the following:

  Does PowerShell Set an Error Level When a Cmdlet Doesn’t Exist?
  When you try to run a cmdlet that doesn’t exist in PowerShell, the engine generates a statement‑terminating error of type CommandNotFoundException [Microsoft Learn](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_error_handling?view=powershell-7.6). This is a terminating error, which means it stops the current statement from running, but execution can continue at the next statement in the script.
  
  Key points:
  
  Error level (exit code): PowerShell itself does not automatically set an exit code for this type of error unless you explicitly tell it to. By default, PowerShell’s error handling is internal and doesn’t return a numeric exit code to the calling process

BW - The error message is reported to Stderr which is captured in the shellcommand_test stderr_line output but again no data is reported to the exit_status.