Putting this here because it's the top comment. I'm an Optical Engineer in the Automotive sector. I design headlights and taillights. If you're in the US you may be familiar with my work on the latest version of the Ford Expedition headlamp. So AMA but I'll give a rundown of what's up with those bright headlights below.
Alright so to start out with, there ARE requirements for maximum brightness. In the US it is regulated by a set of regulations called FMVSS 108. Canada's CMVSS 108 is nearly identical and there's a different but similar set of regulations in the EU. Almost all countries reference or copy one of these. But in any case, there are strict rules about what you can and cannot do. We'll just talk about the US though because our headlamp design tends have more glare for oncoming cars.
Right, so long story short when we measure these lamps we map out how bright they are based on the angle from directly forward, called HV. for oncoming traffic the important points are a 2 lines starting 1.5° left of center and continuing out to 90° left. One is at 0.5° up and the other is 1° up. The limits for them are 1000, and 700 candela respectively.
Firstly, these points have actually gotten DIMMER in the last few years, particularly the 6 or 7 since IIHS started rating headlamps. Overall the lamps are MUCH brighter but LEDs allow for substantially better control. So 20 years ago your Low beam would be maybe 15 m wide and light your lane 80-90 m. The latest requests from OEMs are 30 m wide and 130m range. Meanwhile the glare has been getting lower and lower. 20 years ago you would often see glare at 0.5° up of 700+ candela, maybe 500 in a really good system. Now I'd expect more like 200.
So if lamps are objectively better what's the problem? Color and Size. You see, the regulations were all written for halogen bulbs which all burn at ~3000K, a nice warm white. They also have a pretty hard limit on how bright they can be and still have a decent lifetime. All this means that to put 500 lumens on the road and hit the required candela values you knew how big a lamp was going to be. Unfortunately our eyes don't care about lumens or candela, they care about candela per square meter. This means that if that if in two identical systems, both putting out the same candela values if one is half the size, out eye will interpret it as 4 times as bright. This is REALLY easy to see in changes from old 7 inch rounds to even older 80 mm projectors. The same or lower candela values out of a much smaller area. Think of it as a big drain pipe and a power washer having the same gallons per minute flow but one is a gentle flow, the other will take your skin off.
Second issue is color. All the units I've mentioned, candela, lumens, etc acount for color and the sensitivity of your eye. BUT what they don't account for is pupil response. Basically, when you see bright light, your pupil constricts so you aren't blinded. But the pupil response for equal amounts of blue and red light are quire different, the blue is much slower. So, in comes HID and LED lamps which often sit around 5000K, closer to true white. But they aren't the SAME white as a bulb at that color would do. A white LED is just a blue LED with a yellow phosphor over it. The phosphor absorbs some blue light and emits yellow. These are balanced to trick your eye into seeing white but there's VERY little red content. I read a paper stating that the red content for an white LED the same color as a bulb is often less than 10%, for the more bluish end of the allowed values it's even worse. So what happens is you get hit in the eye with this light and instead of immediately constricting, your pupil stay much more dialated allowing more light in and glaring.
These are known issues and there ARE people working to resolve them but regulations are slow to change, especially with so many stakeholders involved. Anyhow, yeah AMA.