This is so common in Bengali families and it is very disgusting. Like you, I also feel uncomfortable around my father/brother if I don't cover up, its very sad. It even goes to the extent where I never actually hugged my brothers because we shouldn't be touching, I think I only hugged them on my wedding day.
That's just awful, Pariah! I briefly went through a phase when I first started getting serious about the hijab where I felt uncomfortable around my stepdad, who had known me since I was little. It took my Pakistani Muslim friend telling me that since he is my family I could still touch and hug him.
Upon recollection, I never once saw any physical sign of affection between my father-in-law or any of his children, let alone myself.