Worth comes from within and you are powerless over others' opinions. So meet with this person if you think it's the right thing to do and then let it be what it is.

I completely respect your hard work. I used to do something similar — took care of old people in a long-term care facility -- and it was the hardest thing I ever did. It made it slightly easier to think of the patients as my little old children. What I hope you remind yourself all the time is how very respectable nursing-type work is. Besides being utterly necessary to everyone, it makes you stronger and can make the rest of your life seem very easy by comparison. Like boot camp, for a soldier. Sorry, I get carried away but I used to worry about how awful it would be for the wrong kind of people to be the only ones taking care of us when we’re all old.

Don't go there. You are a unique human being and so is the person you
are meeting. Your self worth comes from having a good family and a
rewarding job that you do well.

If this person was a friend, that should help you through. Don't compare circumstances -- I know wealthy people who are awful and don't lead such great lives. Be happy to know you are helping others. This is the stuff that makes us worthwhile.