For me, it usually depends on my board op. If I have a very trustworthy/experienced board op, I'll be fine letting them take cues on their own. I'll still give them standbys, but I'll say, "Standby Light Q 22 to take on your own visual when such and such happens" to remind them what they're watching out for.
You could get theatre administration involved here, since the integrity of the show is being compromised. It would be perfectly legit to submit something in writing saying that you feel, as a crew member on the show, that the stage manager is not upholding their contractual obligation to maintain the artistic integrity of the show.