The graphene functions used by clutter for picking assume that boxes are
inclusive in both there start and end coordinates, so picking at y
coordinate 32 for an actor with the height 32 placed at y coordinate 0
would still be considered a hit. This however is wrong as 32 is the
first position that is not in the actor anymore.
Usually this would not be much of a problem, because motion events are
rarely ever at exactly these borders and even if they are there will be
another motion event soon after. But since actors in gnome-shell usually
are aligned with the pixel grid and on X11 enter/leave events are
generated by the X server at integer coordinates, this case is much
more likely for those.
This can cause issues with Firefox which when using client side
decorations, still requests MWM_DECOR_BORDER via _MOTIF_WM_HINTS to have
mutter draw a border + shadow. This means that the Firefox window even
when using CSD is still reparented. For such windows we receive among
others XI_RawMotion and XI_Enter events, but no XI_Motion events. And
the raw motion events are discarded after an enter event, because that
sets has_pointer_focus to TRUE in MetaSeatX11. So when moving the cursor
from the panel to a maximized Firefox window the last event clutter
receives is the enter event at exactly integer coordinates. Since the
panel is 32px tall and the generated enter event is at y position 32,
the picking code will pick a panel actor and the focus will remain on it
as long as the cursor does not leave the Firefox window.
Fix this by excluding the bottom and right border of a box when picking.
Fixes https://gitlab.gnome.org/GNOME/gnome-shell/-/issues/4041
Part-of: <https://gitlab.gnome.org/GNOME/mutter/-/merge_requests/1842>
a7c4e8cefa