As a PA and intern, I did not worry about dressing casually. I figured I was the lowest man on the totem pole, so to speak, and therefore not obligated to dress well. I also used the others around me as the clue for appropriate dress, but I paid most careful attention to the other stage managers, rather than artistic or administrative staff. After all, they are the ones who I worked with daily, and who had to do the type of work we do, namely, crawling around the floor with spike tape, hauling rehearsal furniture into studios, etc. My standard uniform then was chinos and a t-shirt or polo shirt. Casual, yes, but not ratty.
After getting my card, I made a conscience effort to "dress the part" of a professional stage manager. What this translated into: same chinos, nonathletic looking sneakers, and more button-down shirts and sweaters. I also shy away from anything too clingy or revealing. The last thing I want is to do is make the crew (esp!) view me as a dopey girl who gets by on her looks. The t-shirt and polos are still a part of the repertory, but usually later in the process, after I've established myself within a company.
For me, the clothes are a visual clue to the cast, director, designers, and crew take I have my act together. I'm often much younger than most of these people, and I want to convey a sense of responsibility and diligence to them from the outset.
All this being said, I never wear clothes to work that I would hesitate to get dirty, and I must be able to move easily in them.