At the time of sending the mail I need the metadata - so offering a SMTP server implementation which keeps this in memory while forwarding is not hard. You’d lose a persistent spool in case of delivery errors - but we’ve been doing relays that keep the client connection open while trying to deliver the mail to relay errors directly to the client already 30 years ago, so that also isn’t an excuse.

For IMAP - if you don’t do serverside searching or similar it’ll work with fully encrypted mails.