It returns the inverted transform, which is always the same as the input, besides for TRANSFORM_90 and TRANSFORM_270. https://gitlab.gnome.org/GNOME/mutter/merge_requests/366
The existing ones are statically inlined, so there is no .c file right now. https://gitlab.gnome.org/GNOME/mutter/merge_requests/366