It seems the GDRP does not agree with you:
To what data do the EEA GDPR and the UK GDPR apply?
The EEA GDPR and the UK GDPR apply to all "personal data,” which includes any information relating to a living, identified or identifiable person. Examples include name, SSN, other identification numbers, location data, IP addresses, online cookies, images, email addresses, and content generated by the data subject.
Not as if the GDPR cares about that specifically. Whatever excuse or justification you might have, the law still applies... Mail servers also have to comply with the law.