I don't think it's our exclusive right, altho we certainly do it most of the time and it would normally fall under our purview. If the set shifts are being "choreographed" by the AD or the director, then hey, go for it, I will still be standing there actively helping negotiate what needs to live/move/end up where and make suggestions. But I have worked w directors who love doing this stuff (using actors, actors/crew or just crew) and as long as my needs are satisfied, it's all good. One director just does it cuz he's never had a real SM before and he kinda loves doing it. And recently for IS HE DEAD we had a completely choreographed intermission shift, all done by crew, which the director, sound and light designer worked out based on the cues they wanted to interpolate. I put things in order and figured out where they needed to live off stage and made some suggestions throughout, but it was a group effort.
Yes, I usually assume it's my job, but if someone else really wants to do it and/or wants to do something creative and specific with it that they want to orchestrate, well then, no skin off my nose!