You have asked for my opinion on casual sex and what I am going to say is going to sound conservative and out of keeping with my usual liberal views.
I think that casual sex doesn't come naturally to women--no pun intended. If you think about the act of intercourse in which a woman takes a man inside her body, you can see that it is a much more physically intimate and vulnerable act for a woman than it is for a man. What more, a woman risks becoming pregnant each time she has intercourse.
Even though birth control makes it possible for a woman to have intercourse and not become pregnant, our biological programming doesn't understand this. Our programming, which dates back to the origin of the species, urges a woman to hold off on having intercourse until she is sure that her man is going to protect and provide for the offspring. The institution marriage has evolved as a way of providing this assurance. Women may feel that they can deny their biological programming, but one way or another, the programming rears its head.
In your case, your programming has left you yearning for more connection with him and a feeling of regret that you gave him your body so soon. He, on the other hand, hasn't looked back since he had sex with you because he, like all men, has been wired for variety--to impregnate as many women as they can in order to insure the survival of the species.
So, in answer to your question, I believe that casual sex is in keeping with a man's biological programming, but not a woman's. Trying to separate your body from your heart is not a natural operation and one way or another I think you will pay for trying to go against your biology. I think you also need to look at your tendency to act on impulse. True, acting on sexual or aggressive urges does relieve the tension. The problem with acting on impulse is that you end up saying or doing things without proper reflection and this often gets you into trouble.
Had you thought it over, I think you might have decided to hold off on sex, not to give him the impression that you were just a one night stand. I think you need to working on slowing yourself down and sitting with the discomfort that comes from not acting until you've had a chance to think things through. You are now being offered a good opportunity to practice the skill of not acting on impulse by resisting the urge to call him.
The ball is in his court. Sit back and wait. If you chase him, you will be conveying a sense of hunger and neediness that is going to put you at a disadvantage. While you are waiting, this about this man's behavior. Think about his obvious reluctance to get too close. He doesn't call when he says he will and he has let a lot of time pass since you last got together. He is playing it very cool, which means that you need to be even cooler still.
While you're chilling, examine why you want to chase after a guy who is so distant. Learn from the experience and learn to live with the discomfort and frustration that comes from not acting on your initial impulse. Ideally, you should be having sex with someone who is able and willing to give you back the connection you desire.