Love Or Selfishness

Years ago, when my husband and I were first married, we discussed what would happen if one of us died.
We were both in our 30’s at the time, so I didn’t consider this is frivolous chat. We both promised never to marry if one of us should pass away. I felt greatly relieved because I loved him so much I couldn’t bear the thought of another woman taking my place.
We are now pushing 50, and I am seriously ill. Last night as we were sitting by the fireplace my husband brought up our promise and assured that it still stands. My question is this: Am I being unrealistic and selfish to find comfort in this promise? I don’t think it would be too hard on him, because he’s workaholic anyhow and didn’t marry me until he was in his mid-30’s. We have no children and are devoted to each other. I can honestly say that if he died tomorrow I could easily live a purposeful life without ever having another man.
Are there other couples who have made this promise and been able to keep it? Are there couples who made this promise and later regretted it? I am torn between the guilt I feel that I may be depriving him for years of future happiness and yet wanting so much to know that I will have been his only true love.