End of the day what matters is your Conscience. If your conscience is clear it doesn't matter what others think or say about you. People talk bad about others sometimes simply because of jealousy or some misunderstandings. 

Best way handle such awkward situations is to ignore it. Be polite and respectful when you are face to face with them but carve them out of your inner circle.  

Relatives are tricky I personally think that if God had relatives even he might not be able to be on good terms with all of them...