Different behavior in datafusion 35.0.0 in reading hive-partitioned parquet data

[jwimberl]

Describe the bug
pip recently switched to installing datafusion with version string '35.0.0'. Compared to a previous installation of version '34.0.0', creating an external table from hive-partitioned parquet data following the [https://arrow.apache.org/datafusion/user-guide/sql/ddl.html](documented instructions) does not work. While all the partition columns show up as columns of the table, the columns from the parquet data themselves do not appear.

To Reproduce

# prepare fake data
import pyarrow as pa
import pyarrow.parquet as pq
import pandas as pd
data = pd.DataFrame(data={'col1': [1, 2], 'col2': [3, 4]})
table = pa.Table.from_pandas(data)
import os
os.mkdir("fake=0")
pq.write_table(table,"./fake=0/data.parquet")

# load into datafusion
import datafusion as df
ctx = df.SessionContext()
ctx.sql("""
CREATE EXTERNAL TABLE data
STORED AS PARQUET
PARTITIONED BY (fake)
LOCATION './*/data.parquet'
""")

The loaded data is missing col1 and col2:

>>> ctx.sql("SELECT * FROM data")
DataFrame()
+------+
| fake |
+------+
| 0    |
| 0    |
+------+
>>> ctx.sql("SELECT table_name, column_name FROM information_schema.columns")
DataFrame()
+------------+-------------+
| table_name | column_name |
+------------+-------------+
| data       | fake        |
+------------+-------------+

Expected behavior
The same steps with DataFusion 34.0.0 produce the following output:

>>> ctx.sql("SELECT * FROM data");
DataFrame()
+------+------+------+
| col1 | col2 | fake |
+------+------+------+
| 1    | 3    | 0    |
| 2    | 4    | 0    |
+------+------+------+
>>> ctx.sql("SELECT table_name, column_name FROM information_schema.columns")
DataFrame()
+------------+-------------+
| table_name | column_name |
+------------+-------------+
| data       | col1        |
| data       | col2        |
| data       | fake        |
+------------+-------------+

Additional context
Operating system: Rocky 8
Python version: 3.10.11
DataFusion version: 35.0.0, recently installed via pip
pyarrow version: 15.0.0

[jwimberl]

This issue reproduces using the newest datafusion release, 43.1.0.

[jwimberl]

I realize now this issue might instead belong in https://github.com/apache/datafusion/issues and will move it there if so.

[kylebarron]

This is probably a duplicate of #957

[ozankabak]

This still seems to be a valid issue that reproduces on vanilla install of DataFusion