Indeed! 2, as always, is a distraction. As the only even prime, it's the oddest of them all.
The important thing is that -1 is a square modulo p. Once it is you can use the (relatively fun and easy to do) Hensel's Lemma to compute exactly what the sqrt of -1 is in that particular p-adic ring.
So for which primes is -1 a square? A classic result of number theory states that these are the primes p such that p = 1 modulo 4. The primes like 5, 13, and 29 will have square roots of -1 in their p-adic rings, while 7, 11, and 31 will not.
Lots of rewarding math to explore around this question.