Incorrect PreImages result for elements not in image

[ttiiddee]

Incorrect PreImages result for elements not in image.

Steps to Reproduce

G := Group((1,2));;
f := GroupHomomorphismByImages(G, G, [(1,2)], [()]);;
# The map f sends (1,2) -> () and () -> ().
# The set [(1,2)] contains no elements from the image of f.
PreImages(f, [(1,2)]);

Expected behavior
The output should be an empty list [ ].

Actual behavior
GAP returns the whole group: [ (), (1,2) ]

[stertooy]

Related: #4088, #4088, #5073 and #5173.

What happens here is that PreImages(f, [(1,2)]) calls PreImagesElm(f, (1,2)), and that returns RightCoset(Group([ (1,2) ]), ()) using this code:

gap/lib/mapphomo.gi

Lines 331 to 349 in f6c2800

	#############################################################################
	##
	#M PreImagesElm( <map>, <elm> ) . for s.p. gen. mapping resp. mult. & inv.
	##
	InstallMethod( PreImagesElm,
	"method for s.p. general mapping respecting mult. & inv., and element",
	FamRangeEqFamElm,
	[ IsSPGeneralMapping and RespectsMultiplication and RespectsInverses,
	IsObject ],
	function( map, elm )
	local pre;
	pre:= PreImagesRepresentative( map, elm );
	if pre = fail then
	return [];
	else
	return RightCoset( KernelOfMultiplicativeGeneralMapping( map ), pre );
	fi;
	end );

If we believe the documentation, PreImagesRepresentative(f, (1,2)) would return fail and hence PreImagesElm(f, (1,2)) would return the empty set [].

However, as was discussed in the issues I linked, PreImagesRepresentative(f, x) currently does not necessarily return fail if x does not lie in Image(f), and may instead return an incorrect element or throw an error. IIRC adding a check like x in Image(f) would work, but would also ruin the performance of functions where we already know this to be true.

There is work underway to resolve this, by splitting this function into PreImagesRepresentative and PreImagesRepresentativeNC, where the former should correctly handle the case where x does not lie in Image(f) and the latter doesn't have to. But this isn't finished.

Currently, I think the safest thing to do is just assume any preimage-related function only works for things actually contained in the image.