There are excellent books on acting, and a few good ones on directing actors, such as I'll Be In My Trailer
by John Badham and Craig Modderno. There isn't room in this blog for everything you want to know about directing actors, and I'm not the best person to tell you. Nor can anyone tell you everything you need to know -- probably the best thing is to take an acting class. I learned quite a bit about writing and directing from my two years at the Joanne Baron Studio training in, and observing, Meisner technique. An acting class gives you a sense of what actors are going through, and a vocabulary for speaking to actors, that will separate you from the "shooters."
But I can tell you a few things about auditions.
First of all, put your actors at ease. You may be nervous. You haven't directed before, perhaps. You might be feeling bad that you're having 27 professional actors show up to audition for 5 parts you're paying bupkis for. (In Canada, ACTRA will cut its scale way down for low-budget independent productions; SAG probably has similar reules.) You might be inclined to overcompensate by being all high-and-mighty with the actors, to establish your position. On my first student film, I wanted to see what the actors would look like in studio lighting, so I basically had lamps shining in their faces during the audition. Don't do that. Be warm and friendly and put them at their ease.
Have your auditions in a neutral place. Don't ask actors to come to your house if you can avoid it. We were fortunate to have gracious permission from Galafilm's head to audition in his conference room. No actor's going to mind coming to a production company's offices, y'know?
Tape your auditions. Get a cheap consumer mini-DV camera. What the actor looks like in person, how his or her performance reads in the room, may not translate to the screen. They may be better or worse on camera. Some actors look younger on screen. Some look older. Someone who seemed to be underplaying in the room may turn out to have perfectly calibrated his performance for the screen.
More importantly, you can't remember everyone's performance. You'll want to go back and look at the auditions several times, as you narrow down your choices. Can't do that if you haven't taped it.
Be sure to put the camera where you can see the actor's whole face! No use having it off to one side!
But, don't you be the person running the camera. You should be watching the actor. Have a friend keep the actor in frame. Also, don't be the one running lines. Have a second friend read lines with the actor. You want to be totally focused on the actor's performance, not worried about what your next line is. I had great help from our associate producer, Laurie Nyveen, and my wife, Lisa.
Don't give any direction for the first take. You want to see what the actor's instincts bring you. They may surprise you. They may get it completely wrong. They may nail it.
Compliment every take! Auditions make everyone nervous. A nervous actor isn't showing you what he can do. Compliments help make the nerves go away.
Now give the actor an adjustment. Don't ask for a result -- "faster!" or "snottier" -- but give the actor an imaginative "as if" adjustment for his next performance: "try this one as if you're illegally parked" or "as if you think she's an idiot." Try not to overload too many adjustments at once.
Try giving the actor a completely different adjustment. I had a scene where the character apologizes. I asked my actors "try this one as if you think you really were wrong," then "try this one as if you think she's overreacting." See how well they integrate your directions.
A cold read can tell you about actors' instincts, and that's valuable. But you need to know how well they take direction. Skilled actors are like dancers, but their footwork is emotional. It's always amazing to me to see a choreographer give dancers some footwork, and the dancers replicate it right away. I couldn't even master The Hustle, myself. Forget the Macarena. Good actors will take your direction and give it back to you beautifully integrated into their performance.
You can add adjustments one at a time: "try it as if you think she's an idiot." "Okay, now try it as if you think she's an idiot, but she's hot
Casting isn't just a cold reading. I once had the pleasure to direct Mariska Hargitay in a scene for class. I've directed actors who came in with better first readings. But I've never seen anyone who could interpret and integrate a direction more naturally. You need to know what they're going to be like when you direct them. There was at least one actor I thought was very good, but I felt my directions weren't reaching him. I opted to go with someone else, because I need to know I can direct my cast.
I try to leave enough time to take everyone to the point where I feel they've given it their best shot, or I'm sure I don't want them. On one of our auditions, I let someone go but I had a nagging sense I hadn't got everything he could give. So I opened the window and shouted down to him on the street to come back in. Don't be embarrassed -- what actor doesn't want to be called back in? His second performance (after I asked him to run up and down some stairs) was much more effective. We wound up casting him -- which I couldn't have done based on his first audition. Don't be embarrassed to ask, "How do you feel? Was that good for you?" Actors often know if they can do better. Sometimes they're wrong, but it's usually worth another take to find out.
Take your time casting! Casting is easily half the job of directing your actors. And casting is free! There's no crew. You have no obligation to pay or even feed your cast during auditions. I've never heard of anyone spending too long auditioning their cast. But there are any number of disaster stories about directors who hired actor friends without auditions and regretted it.
And finally, remember: have fun! You get to hear your script come to life a bunch of different ways. What's more fun than that?