You are in a bind. If you allow him to control you, he will become worse and worse. The obvious course of action is to put your foot down and not permit him to control you.
However, you said that you are afraid that he might hurt you physically, so I am concerned that your standing your ground will get you hurt. I think you need to go to a battered women's support group. Even though he hasn't hurt you yet, the fact that you sense that he will should be taken seriously.
Trust your instincts. Don't wait until you get hurt to seek help. In group work on understanding why you are drawn to a controlling man and how this relationship recreates what you experienced as a kid. When you make the links and talk about how you felt way back when you will be on the path to freeing yourself from his tyranny.
I want to hear back from you to know what is happening.