It's probably more of an issue between you and your mom, than between you and the cast. Make sure you have a very professional, non-familial relationship with her at work. Even if everyone knows she's your mom - you don't have to act like it in public. You are both there to get a job done. Ask her to treat you like you're in charge - because you are. If she's a high-tier ballet dancer, she knows a bit about professional relationships, and she knows the role of a choreographer and the role of a stage manager. She shouldn't call you "sweetie" or any other pet name or family nickname, or talk about "what are we going to have for dinner tonight?" because those aren't conversations that need to be had in the workplace. And by that same token, you shouldn't have those conversations with her either.
Maybe it would be helpful for you to not call her "mom." This suggestion is probably going to be hard for you - I know it is hard for me, and I'm in my mid-to-late 20s. If you call her by the same name everyone else is calling her (whether it's "Mrs. [Lastname]" or just her first name), people will understand that this is a business relationship. I'm not saying you must not call her "mom," it's just a thing that I think would really help me if I was in that position.
I recently stage managed a professonal show that had a voiceover-only character and after the original actor kept canceling on our recording sessions, we brought in my dad to do it. Luckily I didn't have to interact with him too much because he was only in for one day, but while he was there I just treated him like one of my actors. He didn't get to sit at the SM table with me or anything like that. We were both there to do our own jobs, and doing our jobs involved not interfering with each other.
Don't panic. That's the key to everything in your job, is to maintain a level head. So even if you're frustrated with your mom (because that's bound to happen), it doesn't matter. You're there to get the job done. You're there to create art. Let go of personal issues, at least in the rehearsal room. Don't let any of it get under your skin.
If you give the cast and crew the impression that you are in control and you are deserving of respect... then you will get that respect. If they see you accidentally slip into childish mode to bicker with your mother, you're going to lose their respect.
Just have a chat with your mother before rehearsals begin. Explain that you need to distance yourself from the mother-daughter relationship while you're in the rehearsal room, because you need to be able to run the process effectively, which means you need to have the respect of the entire cast and crew, which will be hard to keep if you guys act too much like mother and daughter in public. She will probably also have some ideas of how to help with that.
