How to Let Him Know You Care, Without Running Him Off

It's a delicate balancing act so far.    There is a reason that your instincts are warning you not to tell him that you care. You must be receiving signals that say, "get to close and you'll run me off." 

He has surely been treating your relationship in a casual way. He doesn't date you often because he's busy with work.  A man who is interested usually pursues more than he is pursuing. Keep in mind that his distance may have nothing to do with whether or not he finds you attractive. He may simply be scared of intimacy and hiding behind his busy schedule. 

The only way you are going to know where he stands is to talk to him. Yes,  you could hang back and wait for him to call, and surely this wouldn't threaten him in the least. However, hanging back is a very passive approach. Ideally, you want to act upon your environment, not be acted upon. Waiting for him to call is to be acted upon. 

I think that life is too short to play games. If he's too terrified of intimacy to handle your feelings for him, then he isn't ready for a relationship and you might as well find out sooner rather than later.  If you decide to deal directly, you could say, "I have very much enjoyed the time we've spent together and would like to see more of you." Then see what he says.  If he gives you the work excuse, you might say, "It's not clear to me whether you are saying that work is particularly busy right now or whether you are telling me that work will always prevent us from seeing more of each other." 

It's best that you find out exactly where you stand. If he's too afraid to come out from behind his desk, then that's his loss.  You deserve a man who can return your love.