Document NaN Comparison

[ntjohnson1]

Describe the bug
NaN < <float> and NaN > <float> should both yield false.
However I don't get that behavior in datafusion.

To Reproduce
I reproduced this on 48 and 49

import numpy as np
import pyarrow as pa
import pyarrow.compute as pc
import datafusion as dfn
from datafusion import col


def py_arrow_example() -> None:
    nan = pa.array([np.nan], type=pa.float64())
    not_nan = pa.array([1.0], type=pa.float64())
    print(f"Less: {pc.less(nan, not_nan)}\nGreater: {pc.greater(nan, not_nan)}")


def datafusion_example() -> None:
    table = pa.table({"a": [np.nan], "b": [1.0]})
    ctx = dfn.SessionContext()
    df = ctx.from_arrow(table)
    result = df.select(
        (col("a") < col("b")).alias("less"),
        (col("a") > col("b")).alias("greater"),
    )
    print(result)


if __name__ == "__main__":
    py_arrow_example()
    datafusion_example()

Output

Less: [
  false
]
Greater: [
  false
]
DataFrame()
+-------+---------+
| less  | greater |
+-------+---------+
| false | true    |
+-------+---------+

Expected behavior
False for both comparisons or clearer documentation for the handling.

[kosiew]

@ntjohnson1

Thanks for reporting this.
This looks like something to be fixed in the upstream repo.

[kosiew]

hi @ntjohnson1

I did more research on NaN comparison.

The values are ordered in the following sequence:

---

• Negative quiet NaN
• Negative signaling NaN
• Negative infinity
• Negative numbers
• Negative subnormal numbers
• Negative zero
• Positive zero
• Positive subnormal numbers
• Positive numbers
• Positive infinity
• Positive signaling NaN
• Positive quiet NaN

---

[kosiew]

@ntjohnson1
Can we close this issue?

[ntjohnson1]

I was on vacation last week and just catching up. Should have a little more time to look at this later this week. This seems a little footgunish so I at least updated the issue to document it.

On a quick search it looks like all the export methods to_<...> yield a different behavior except polars which also deviates from the standard for nans.

My motivating use case was comparing two data frames that had struct column with nans and getting weird behavior, I may file a different issue now that I understand that nans are supposed to be handled this way.

[ntjohnson1]

Ok I didn't have a chance to circle back to this this week. I'll just close this for now and open a new issue when I circle back to my motivating use case. I'll probably file a ticket to document this nan treatment as well for clarity.