Nah, it's cool. That's the first thing I thought of too, but with some experimenting found it to be a non-issue. Firstly, the sleeve really won't come out of the union once installed properly until you retrieve it with w stopper knot, Secondly, it retains a bit of that U shape which creates friction on the rope slowing or stopping a rapid slide. Thirdly, if it did somehow contact your kit then the impact would be minimal - not enough to cause you to plummet.
In fact at times when it's too stubborn to go inti the union from the ground I've just let it ride above my hitch until I can get there and force the issue - you've got to watch your hitch of course to make sure it's setting but then you watch all your gear all the time anyway right?
Enjoy the sleeve and proceed with confidence.