As others have said, it's all about your threat model.
For me, I keep mic and camera off at all times until I need it. Its a slight nuisance, but I'm nearly sure that it covers all the bases.
I could go hardcore and physically disable the mic in my phone, then only use earbuds for my mic. That's a bit much for me, and maybe my situation is a bit much for you.