Oh, your relationship is anything but physical. If you were only a sex kitten to him, then he wouldn't be scared by how much he misses you. He is clearly deeply attached to you.
In answer to your question, why is he scared to have missed you so much. There are a couple of possibilities. To love is to lose, and the more you love someone the more frightened you are to lose that person. Your leaving made him aware, more or less consciously, of how scared he is to lose you, either through death, a break up or through your abandoning him.
nner selves, we all have to admit that it is scary to get close to another person. The more you care, the more you suffer when the relationship ends. And, since all relationships come to an end, it is only a matter of time before the pain comes.
See if you can get him talking. That is the best medicine for his fear. We need to dilute his feelings so that he doesn't become so terrified that he takes a hike before you can leave him due to natural or unnatural causes.