Don't feel bad for wanting to say 'no' to your friend, Rustys. I'm sure you're not the only one who has been in the same position as you. Saying 'no' or accepting a 'no' is one of the many things that friends have to do. Either in your favor or theirs.
Yes, there will and are going to be times where either you or your friend are not going to like a 'no.' However, instead of disliking it, respect it. Your friend doesn't know what you feel is not a good time to do something and vise versa. So, I hope your friend never takes offense to you saying 'no' to wanting someone to play with. It's just sometimes, at the very moment is not really a good time or you just don't want to play.