In this business, trust me, if you quit for the right reasons, the only people that should against you is the producer you left - and you probably don't want to work for them again anyways.
Although we are taught over and over it's a small world, but I know stage managers that have been fired multiple times from shows, and yet they still get work, all the way up to Broadway . . .
Again, if you are good, you should be able to find the work.
To get back on my soapbox a bit about the state of our industry - the market is flooded with stage managers (not particularly good stage managers, just a lot of them). This leads to the devaluation of our job (come do my show for the resume credit, come do this show for 300.00 for all eight weeks of the run, come do this show for a metro card and a case of beer) - along with some dreadful working conditions - ask any young or not-so-young New Yorl stage manager about their shared space rehearsal or performance venue, where they are expected to do 12 other things other then stage management, often without an assistant, and do it all for less the unemployment.
There are things we can do - one, we can stop the high output of stage managers from our educational system (or at least hope the educational system raises the bar on the students they are putting out). Two, we can stop accepting these no-pay or low-pay jobs. Three, we can stop taking jobs with these horrific working conditions. Four, we can have the guts to quit jobs that become unbearable or abusive. (I know that many producers will just turn around and pluck someone else from the over populated work force to replace us . . . but if people just refused to work under these conditions, hopefully in the long run the working environments will change).
Off of soap box.