I hear how tormented you are and I think the reason is because you haven't learned a basic fact about the human psyche: That is, we humans are wild beasts filled with all kinds of outrageous thoughts and feelings that we can't control.
Repeat after me: Feelings are like the wind; sometimes they blow east, sometimes west, sometimes north, and sometimes south. No one ever says that the west wind is good and the east wind is bad. The same is true for feelings. Feelings aren't wrong or right, they simply are what they are and they need to be accepted without judgment.
The reason why most people find it difficult to accept their feelings and even condemn themselves for their them is because all too often feelings and actions are one in the same. If they feel angry, they act in damaging and destructive ways; if they feel sexual desire, they act out sexually.
So, you can only allow yourself to feel all your feelings without reproach when you make the separation between feelings and actions. To separate feelings from actions, we must harness our raw emotions and then subject them to the scrutiny of the observing part of our psyches (called the observing ego).
It's the observing egos job to consciously choose a way of handling our feelings that 's in the best interest of all parties involved. If your intended words or actions will feel good for you (kind of like taking an emotional dump on someone) but are bad for the other person, then it's bad for you in the long run to say what you have the urge to say.
The key is to transform these raw feelings into words or a course of actions that's good for everyone. Many people say that what separates us from the animals is our capacity to control our actions and our behavior. We shouldn't dis animals since I've seen animals resist the temptation to act on their impulses.
I had a little canary who was furious at me one day. He began to charge toward me with an attacking posture, and then I saw him stop dead in his tracks, shake his head, then shift gears. By the time he reached me he kissed me instead. That's impulse control!
We all should strive to be more like my little canary was. All this being said, I want you to know that you are a dream husband. You should praise, not condemn yourself for feeling tempted and still choosing to behave in a faithful way. That's commendable!
Instead of torturing yourself for your feelings, study them and try to understand why they are arising. What is this obsession with your first girlfriend telling you about yourself and/or your relationship with your wife. Is there something that existed in your relationship with your first girlfriend that you wish you could experience with your wife? Is your pull toward your ex. an expression of your wish to return to your youth and perhaps a symptom of a mid-life crisis in the making?
When you stop beating on yourself, you'll be able to understand the whys of this obsession and then take steps to actually resolve whatever emotional or relational issues you uncover.
At the same time, keep working on your tendency to self-attack. It's not healthy and it makes your life a misery.