I am going to be the jerk here.
You said " I am doing my job of making tech go as smoothly as possible", but you are not - obviously there is still a major issue.
You know, you may not appreciate being walked over, but you know what, as stage managers, that is what happens to us all the time.
The givens of the situation are
1) You are the SM
2) He is the ASM
3) Nothing is going to change that fact - unless you quit, because they are firing him
4) You have to tech the show and open it up.
The fact that an outsider is seeing the problems is just proving the the situation is not being managed well; and that's sad. Management is our job, managing creative, quirky, technical people. I think it's impossible for one of us to offer advice on what you can do because at the end of the day, this job is about a personal style that you manage with - to give specific advice I would need to your style, observe your ASM, have some history, etc, etc.
We have all had bad assistants - assistants that have a different agenda, assistants that bad, assistants that are lazy, assistants that just don't know their job. We all have different techniques to either address the issues face out and solve the problem, or work around the issue.
I think at this point, you have to figure out how to do the show without him - which you already are - having actors help with quick changes, you may need to do the intermission shift, whatever.
Maybe he will go away on his own, if you can't fire him.
I wish you well, it's a bad situation that has just got worse.