I feel like "afraid" would be the wrong word, but I definitely avoid starting a talk about it. It's either immediately a heated discussion that I - obviously - initiated, just before talking bad about the other person's mother, or it's people givinge the "I don't eat that much meat anyways" talk. I'm so tired of both.
Also I feel like when people find out, many lose the ability to see beyond the label. Like I suddenly loose every bit of personality I ever had. That's why I especially avoid it in a professional environment. My boss can trust a professional, experienced worker, but can they trust a vegan? Don't wanna find out.