I'd say your biggest issue is likely to be power. The CM4 isn't exactly a power hungry beast like a desktop mobo/cpu combo would be, but it's also pretty far from the power sipping abilities of most MCUs like an Arduino or Pi Pico. Some very quick napkin math from its data sheet suggests it needs a minimum of a bit under 2W at idle, and under load can consume something in the ballpark of 7 or 8W and that's using the datasheets own "typical" values, I'm sure there are situations that could very likely double those figures. While Linux can suspend and resume to drastically reduce power consumption when idle, that's not exactly a speedy process, I'm not sure you'd E.G. want to do that in between page turns.
Embedded programming and micro controllers
About embedded programming, micro controllers, ARM Cortex M, RISC-V, Arduino, Micro:Bit, etc.
I thought about setting the CM4 to sleep when no input was done for ~10min and just hoped, that that would reduce the power consumption enough to last at least 1 week or up to 1 month when used for 1h a day. But it was just a thought. I have not settled to an approach yet.
As far as I know commercially available ereaders go to sleep in between page turns. After a pageturn my Kobo/Tolino stays awake for maybe a second to allow the user to rotate the device, after that it only responds to the page buttons or touch input.
Most ereaders are running on highly modified Android systems, because it is easy to write graphical applications for it and the underlying Linux kernel makes networking and the likes very easy I suppose.