Prion diseases are not as simple as coming into contact with a protein composed of D-amino acids.

While it's still not fully known, it's generally one specific type of protein PrP^C that gets transformed by PrP^Sc, converting the C to Sc, and so on.

D-amino acids also occur naturally on earth.

As for B vitamins, none include proteins as far as I remember, they're all small(ish) molecules like biotin and folate. Also having a difference in stereochemistry with most of these vitamins, wouldn't cause toxicity.

Obviously there are some cases of small stereochemical changes being toxic (thalidomide). But generally small changes won't automatically make something toxic.