When you ask me what you should feel, it's the same as saying, when am I allowed to feel hungry or thirsty? In the feeling department, no one can tell you how you should feel. I suspect that you were raised by parents you told you that your feelings were wrong or not allowed, which explains why you are asking me to tell you how to feel.
As I see it, your first problem is that you don't allow yourself to feel what you feel. You treat yourself like an unentitled citizen. I don't think you wanted to go along with the swap in the first place, but, it sounds like you are so accustomed to swallowing your true feelings, that you went along with your husband's plan despite your misgivings.
I think you must be hurt and furious with him for what he's done. I'm not telling you to feel this way, if you don't. However, I am quite sure that if you allowed yourself to feel, you would be really hurt and pissed.
On the subject of denying feelings. I said that I think you learned to stifle yourself in your first family. Not surprisingly, this pattern has extended to this marriage. You described your marriage as perfect, no arguments, etc. Where are all the normal feelings that arise in even the best of marriages? Where was the conflict, the anger, the hurt. All buried under a shroud of perfection.
The problem with emotional cut-off of this magnitude is that issues go underground and feelings of resentment are buried. Buried resentment transforms into boredom (we need a little excitement, let's try a partner swap). In other words, the desire for a swap is a symptom of a deeper issue--that the marriage is dying because feelings are not being addressed and resolved.
So, when you ask me how you should feel. If you allow yourself to feel all your feelings, the natural reaction to your husband's 'request' would be a cornucopia of emotion--hurt, rage, betrayal, etc..
What's more, if you allowed yourself to feel everything, I am sure you would feel utterly outraged that your husband displayed feelings for this other woman right under your nose. When he made declarations to her in bed that he never said to you. He dumped right on you!
I want you to know that your husband's request for wife swapping and the words he told this other woman, expressed a great deal of anger on his part. Rage and anger is normal in any relationship. But, the goal is to talk and not enact these feelings. When enactments occur, every gets very damaged.
Before we go any further, I think you need to find out why you are so willing to wipe out your feelings and go along with a plan that goes against your grain. In other words, why are you so compliant and willing to swallow your true feelings. Am I correct that your parents stifled you?
If I were you, I would find a good therapist and begin talking about your history, and explore your willingness to swallow feelings and comply, even when it goes against your grain.
Once you have allowed yourself to own all your feelings, I would tell your husband that you are not willing to swap any longer (because this is how you feel, not because you think it's what I am telling you). I would tell him that you are not willing to share him and that the problem with your marriage is not sexual boredom but a refusal to deal properly with negative feelings. I would also tell him that you expect him to go into marriage counseling so that you both can learn to discuss all your feelings, so that you don't enact them in ways that are damaging and hurtful to each other.
I want you to know that when you become an entitled citizen, and put your foot down, he will stand up and take notice. He isn't going to respect you and your feelings until you respect them first.
My best wishes to you. You will need courage and strength to find your voice. Your goal: Never allow the voice in your head, or those around you, to silence your true feelings ever again.