10 points for honest answer?

You need to tell him what you just said up there. You need to tell him that, because if you want to keep him around, you need to explain why. I'm sure he will understand! Just be honest. If you feel the need to save face, just send him a text message or an email. Then, when the air is clear, you both can meet up with again.