Downtown Detroit is perfectly safe and filled with lots of fun things to do both during the day and at night. The worst that will likely happen is that a homeless person will ask you for money while sitting harmlessly on the street. This can happen in any major city, tho. Stick to downtown and stay out of the Detroit neighborhoods and you'll be fine. There is nothing touristy to do in the neighborhoods so you'll have no reason to go into them.
Pretty much everything negative people say about Detroit is the typical overblown pearl clutching. As for your family that lives in Detroit... Do they actually live in Detroit or one of the suburbs? Because that will be most telling. Most people that live in the metro Detroit area will say they're from Detroit because it's easier to explain than being from Clinton Township or Troy.. And suburbanites can be snobby about Detroit in general. I fully agree with your gut feelings and they are spot on.
As for a comment for your first time traveling to the US.. I'm delighted you chose Detroit and I'm hopeful you will have a blast while here! Detroit has awesome culinary, music, and art scenes and there is something here for everyone!
Alright, so I love your attitude and your going to fit in just fine in the D. As for your wife's family, Canton is closer to Ann Arbor than it is Detroit so I chuckled a bit when you named their city. Have a great time and let us know how you end up spending your time in our fair city!