Well I grew up with a speech impediment of my own (medial and final R) so I wouldn't have a problem with it!
(Mine is gone now after 3 years of speech therapy around middle school... it still happens once in a blue moon but not enough for anyone else to notice, but I still feel a sense of identity with it even though it no longer affects me)
But I feel like... when a unique speech pattern is part of your everyday life, you make subconscious 'fixes' for it. You're always compensating. So I would guess your calling has a slightly different rhythm than mine does just so you can have a little "just in case" wiggle room. I can imagine that being on book could be frustrating both for you and the actors... but I don't think it should be a huge problem? Unless you get an actor who is super impatient. But I suspect that if they've ever had an actual conversation with you before the moment when they call line... then they will probably be aware of your stutter. Like, it shouldn't be a
surprise to them. So yes, it may slightly throw off the rhythm of the scene, but... eh, I feel like most people aren't going to be shocked by it so they should be able to take it in stride.
I feel like it's not that big of a deal, in the big scheme of things. Speaking clearly is part of the job, but certainly not the only or even necessarily most important. So no, I definitely do not think a stutter is inherently a problem. If, on an average day, it doesn't hurt your ability to do your job, then no I don't think it matters.
