You asked me if you should pursue this relationship. You really need to ask your lover this question. (Should I feel like giving up on us. Is that what you want me to feel?). In answer to your other question, yes, a relationship with a large age difference can work out fine, provided the couple is compatible (see my Dr. Love's Ten Tips hyperlink: Is it Love or Lust).
Let's backtrack a second and make sure that your lover isn't using this age issue as a smoke screen for another issue. In other words, would he be frightened of commitment if you were his age? Would he be afraid of your leaving him for another man if you were his age? I would ask him these questions and see if he can discover whether he actually suffers from a fear of abandonment. I would also ask him to be more specific about why he thinks the age difference could result in your leaving him one day. Make him talk openly. Is he afraid that he wouldn't be able to perform sexually and that you would be forced to seek satisfaction elsewhere? (Let him know that there are plenty of ways to correct impotence in older men. Remind him that many men have erections until their death. Also, point out that there are many ways to make love besides intercourse. You love him, not just his penis. )
Find out what other fears he has and talk them through. This should help. I would also ask him if there is anything that you could say or do to convince him of your complete devotion. After you complete all these discussions, he should feel better and more able to commit. If he doesn't feel better, try some couples counseling. If he is still stuck, then and only then will you be faced with the decision about whether to stay or move on. Lots of luck. Let me know how it goes.