Personally I think it's only much of a problem when it's two languages from a language branch other than my native language.

So, for example, as my native language is Portuguese (from the Romance branch) I have no trouble telling French from Portuguese, Italian or Spanish (even when words are the same the accent is different) and whilst I might on occasion mix Spanish words into Italian or vice-versa when speaking, it's unsual, but when I learned German after having learned Dutch it was very confusing and almost felt like the German language knowledge was eating up the Dutch language knowledge in my mind, because one so often polluted the other one (more in the thinking and talking in that language, rather than spotting if the language spoken was Dutch or German, since the accents still give it away, to the point that I can tell Swiss German from that from Germany even though my German language knowledge is still pretty basic).

Meanwhile back when I first I learned French after having learned English I never confused one with the other.

I think that if you're intimately familiar with a language branch you know enough to spot even small differences and know which is which or at least it's a lot easier (hence I might confuse Spanish words with Italian ones - both foreign languages to me - but it's unusual) but in a totally different language branch the "distance" from what is familiar is a lot larger and words from multiple languages from that branch which you're not sure of just sound like they might be from any of those languages (or even multiple of them, which they sometimes are).