Oxygen is reactive enough that free O2 doesn't tend to stay that way. Even to us, too much oxygen is toxic because it starts reacting with things we don't want it to inside our bodies (which is why antioxidants are beneficial). The presence of oxygen in an atmosphere implies there's life producing it (or maybe some other process we aren't yet aware of). It can also imply that there's also life using it to keep it in a balance that doesn't freeze the planet or allow it to destroy any life because oxygen producers keep pumping more of it into the atmosphere.

And the danger part comes from a combination of alien life maybe having microbes our immune systems can't defend against and the possibly of picking one of those up and bringing it back to other human settlements before the carriers realize they have a fatal infection.

That second part is what makes it more dangerous than a ship accidentally flying into a star or trying to land on a gas giant, which would probably be fatal to all aboard the ship but not dangerous to humanity as a whole. Even just entering an oxygen atmosphere with a ship that never lands could be bad if that ship must later return home.