fix(human_labeled_dataset): preserve # inside unquoted CSV cells

[Chen Yufeiyang (feiiiiii5)]

Fixes #2974

HumanLabeledDataset.from_csv passes comment="#" to pandas, which treats every # anywhere in the row as the start of a comment — so a hash inside an unquoted cell ("mentions C#", # Heading, a leading # in an "assistant_response") silently drops fields or the entire row. csv.writer/DataFrame.to_csv do not quote fields just because they contain #.

Skip only the leading # comment lines (already parsed for metadata) with skiprows and remove the pandas comment argument, so the rest of each row is parsed verbatim.

Test:

pytest tests/unit/score/test_human_labeled_dataset.py -q

Fails on main (1 entry, objective mentions C in the issue's repro), passes on this branch (74 passed). ruff check / ruff format --check clean on the touched files.

[hannahwestra25]

Suggested change

	with open(csv_path, encoding="utf-8", errors="ignore") as f:
	for line in f:
	if line.lstrip().startswith("#"):
	skiprows += 1
	else:
	break
	with open(csv_path, encoding="utf-8-sig", errors="ignore") as f:
	for line in f:
	stripped = line.strip()
	if stripped.startswith("#") or not stripped:
	skiprows += 1
	else:
	break

lstrip() doesn't strip U+FEFF, so a UTF-8-BOM file (Excel's default) leaves skiprows=0 and the metadata line becomes the header — ValueError: Required column 'assistant_response' is missing. Worked before, since comment="#" ran after BOM handling. A leading blank line fails the same way. plus adding unit test

[hannahwestra25]

Suggested change

	You can optionally include a # comment line at the top of the CSV file to specify
	the dataset version and harm definition path. Only leading # lines are treated as
	comments; a # anywhere else is preserved as literal data. The format is:

[Chen Yufeiyang (feiiiiii5)]

Good catch on both, and reproduced before changing anything — you were right about the cause.

The BOM/blank-line bug. Confirmed on the branch head: a file written as utf-8-sig (Excel's default) with the metadata line, and the same file with a blank first line, both raised ValueError: Version not specified and not found in CSV file. Two places read the file as plain utf-8: the metadata parse at the top of from_csv, and my skiprows scan. lstrip() does not strip U+FEFF, so the comment line never looked like one.

Fixed in dc504802: both reads now use utf-8-sig, which drops a BOM when present and behaves like utf-8 otherwise, and the scan skips leading blank lines as you suggested:

        skiprows = 0
        with open(csv_path, encoding="utf-8-sig", errors="ignore") as f:
            for line in f:
                stripped = line.strip()
                if stripped.startswith("#") or not stripped:
                    skiprows += 1
                else:
                    break

Two unit tests: test_from_csv_reads_metadata_comment_after_byte_order_mark (fails on main — the BOM bug predates this PR, since the version parse had the same encoding) and test_from_csv_skips_blank_line_before_metadata_comment. The full tests/unit/score run is the same 32 pre-existing failures as main, and ruff check / ruff format are clean.

The docstring. Added your wording: "Only leading # lines are treated as comments; a # anywhere else is preserved as literal data."