The important thing is that the high albedo material has to convert incoming light into a particular wavelength of infrared light that is not absorbed by the atmosphere and so it flies straight out into space. If it's just white (high albedo) but doesn't convert light into the right wavelengths of infrared light, then yes, it will warm the surrounding areas because a lot of the light it's reflecting will get reflected off the atmosphere back down somewhere else.
Wikipedia has a thorough article on it: https://en.wikipedia.org/wiki/Radiative_cooling
Here's a video by a channel called NighthawkInLight: https://youtu.be/KDRnEm-B3AI?t=78
And here is a 2 part series by TechIngredients: https://www.youtube.com/watch?v=5zW9_ztTiw8 https://www.youtube.com/watch?v=dNs_kNilSjk