How would you address this worry?

I think I can understand how you feel. It's a very unpleasant thing to know that others have portrayed you as someone you are not or have been dishonest about your character. We have all suffered this at one time or another, some of us worse than others. Unfortunately, there will always be people around who have nothing better to do than to try to destroy other people's reputations. As there will, unfortunately, always be people who have nothing better to do than listen to their gossip. Such is the way with too many people in my opinion. If you are like me, then you detest the injustice of being treated in such a manner. You need to tell yourself that for such people they will always be that way and are not worth bothering about. Anything they don't know, or think they know, they will invent anyway as they have petty lives. You know you are better than that, so the best tactic is not to let them see it bothers you. Anyone who knows you are a decent person will not be persuaded by gossip and anyone who doesn't know you but chooses to judge you is not worth knowing. The best thing is just to keep away from negative people or people who have injured you in the past. Hold your head up high and ignore them. You will never be able to change them, only they can do that. If you know you are okay with the way you conduct yourself what does the opinion of Joe Bloggs mean? Absolutely nothing. There are far more important issues to worry about in this life. go out and enjoy your life and don't fret about others opinions. Chin up.