A recurring theme in Skyrim is decline. The Empire has collapsed since the times of Oblivion, Morrowind hasn't caught a break ever since the end of the Tribunal, Cyrodiil and Hammerfell have been ravaged by war, the Thalmor seemed to be thriving as they have managed to place themselves as a rival to the Empire and have a couple of provinces following their lead, but even they are considerably weakened. If you go to Skyrim itself, both the Companions, the College, the Thieves Guild and the Dark Brotherhood are crumbling apart and that only changes if the Dragonborn decides to join them, they've suffered a civil war, gotten divided in religion (is Talos a legitimate god?), allegiances (independence or Empire?) and culture (cosmopolitan or traditional values?), all of which are challenges that may last for generations.

If the designers of TES VI want to be coherent, the next game has to accept that almost all old institutions must either die or reinvent themselves. This means that, if the Empire survives, it must be a radically different one.

Of course, it's a game in the hands of Bethesda/Microsoft and they may just decide to go with whatever is the easiest to do marketing with, but it would be dissapointing given how Bethesda is one of the few large companies that have historically taken worldbuilding seriously.