trans men
first, you have your terms backwards. Trans men were assigned female at birth and transitioned to male. They want to be able to use the men's restroom. You are discussing trans women's right to use the women's bathroom.
What does the physical size of some trans women have to do with anything? There are a lot of gay men who are bigger and stronger than me, if a "bad actor" wanted to sexually assault me, the men's bathroom would be the perfect place for them to do it. Should we ban gay men from straight men's restrooms because of some men would have that fear? No, and the reason is because it's not a valid fear, it's in my head (or was), it's not likely. Does that mean no gay man has ever assaulted a straight man in a restroom? Of course not, but it just not a realistic fear. It's in their head. It's a phobia.
Cis women who are uncomfortable sharing a bathroom with trans women are uncomfortable not because they have seen big hairy men in dresses eyeballing them in bathrooms, but because they are irrationally afraid of that. It's not a valid fear, because it doesn't happen. It's a phobia. Cis women are just as safe with a trans woman in the bathroom than they are with other cis women in a bathroom. That is an indisputable fact based on all actual evidence. Anything else is an irrational fear based on prejudice and ignorance and you are validating that. That means you are supporting transphobia right now, whether you want to admit it or not.
Automata 2014
Oblivion 2013
Jupiter Ascending 2015
Evolution 2001
There might be a pattern in there somewhere...