If they don't live up to your roleplaying expectations, it's always fine to say no.
Nontheless, you should be respectful about it.
I won't mention some things because many of the wonderful players have already contributed, so I'll keep it short and sweet.
~In my opinion, I would make sure you use the word "no" or at least express the intention very clearly. I cannot mention how many times I've been confused by the roleplaying captain (is that what it's called?) because they were so terribly afraid of hurting my feelings.
~Only offer them advice if they ask for it. Or at least say, "Would you like some tips to help make your rp samples better?" If you go shouting things at them, or even politely explaining to them what they did "wrong," they're going to feel like they're breaking some sort of rules. They're going to feel pressured into it (even if it makes them a better rper), and they might honestly get a bit annoyed.
~Say your please's and thank you's. Even if their sample wasn't much to talk about, they still put an effort into it- even if you had more trouble deciphering it than they did writing it.
~Wish them good luck. Everyone likes to hear that.
~You should explain why you won't accept them. This is very different from telling them what they did wrong, or giving them advice. You're not scolding them. You're not telling them what they should do. You're simply explaining why you don't want them in your rp.
~Give them a second chance. Everyone deserves one; but if the second chance wasn't good either, it's still fine to say no.
~If you have a realistic rp, and they only want to be bright, neon characters, don't scold them for it, but instead, point them in the direction of another rp that accepts colorful animals. (:
~TIP: This is a personal thing. All rp samples, aren't like this, but this is just personal input. Tell them BEFORE what you're looking for- not in a sense so much as telling them specific details you need in order for them to join. I mean this- Tell them if your rp takes place in third person, first person, etc. Make sure they know what form they should be writing in. You don't want someone sending this--> "I looked up at the sky with a dreamy daze across my face. Oh, how the wind feels nice against my face!" If you're looking for this, "The wolf sat at the edge of the cliff, gently lifting her face into the cool breeze. The wind rustled through her fur, and she closed her eyes in bliss."
Well, I guess this wasn't so short-and-sweet after all...Oh well. I hope my input counts for something. ^.^
