Yes, that is a concern. Pictures of a new battery shows it to normally be flat or slightly indented in that area.
Spicy Pillows
Bulging batteries are bad news, always replace
It’s not a concern until it is. Are you OK with it starting a difficult to extinguish fire in its current location? If not, you might want to get rid of it.
Just poke a hole in it to release the extra pressure! (Don't do this)
Don't say that
I love me a spicy pillow!
You can have mine. I'm done using it.
I mean, it's not going to start bulging less from here on out. Unlikely to burst, but should be replaced for safety and performance. You've already done most of the work by opening it up!
The top-right part looks suspicious, but I don't know if you should be concerned about it or not. Once you put your computer back together, try to monitor how hot this location gets while charging, compare to the other sections of the battery. If you can't tell a difference to the touch, it's probably fine. If it gets noticeably hotter, I recommend to change the battery asap.
A little? Both cells on the right are bulging pretty badly. Even the one on the left is starting to bulge.