I think your hesitance is key here: seems in your gut you feel the risk is likely.
If you do not feel inherently safe disclosing to your manager and/or HR, then I personally wouldn't disclose. That's not to say you absolutely shouldn't, because we don't know the context, the history, the location or size of your organisation.
Before I knew I was AuDHD, I saw neurodivergence weaponised a bunch in one of my previous jobs. A couple of autistic men often got away with wildly inappropriate behaviour and preferential treatment because management made sure to coddle them because they were 'different'. Briefly, an 'out' autistic woman started in the same area and she was labelled as needy, impossible to work with, constantly denigrated, and quickly dismissed. I saw a lot of myself in her and wonder if the only difference in how we were treated, was that she disclosed.
I have disclosed AuDHD to my current manager only because she makes me feel extremely safe and she's keen to learn about ND so we can work together more effectively. I would not do this if I sat under any of the other managers in our area, because I do not think they are equipped to understand, nor do I think they'd attempt to.
I appreciate how complex this is and it's something I've been thinking about a lot lately... Whether it's for some societal greater good that I be "out" about my ND to give visibility of different neurotypes. But I'm also hella tired and don't want to be the Neurodiversity Champion at this point in my journey.
Whatever decision you come to I really wish you all the best.