You’re right - it was gradient filter.
I did try it with the auto mask like you did but I also got that thick area around the bird that isn’t masked. So if I really darken it that area looks “off”. I could go in and fine tune it with the brush though.
In fact the reversed Auto Mask does have an effect right up to the boundary it detected. It’s just that the mask doesn’t get drawn correctly.
like @Joanna mentioned “It’s just that the mask doesn’t get drawn correctly.”
with Auto mask try not to get ‘overboard’
– otherwise use the Eraser [ALT+ … in the Windows version]
or use Mask …
and then after copy / reverse the mask for the background