When things start to fail... adding Multipath Routing to Cloudflare Email Routing with Email Workers
Thanks to Cloudflare Email Routing, I can accept and send emails in my domain, but still manage them on my ordinary Gmail account. This all is set as I wrote down in my post titled Email in your own domain with Cloudflare (and Gmail)
Over the last week, I faced an issue with emails not being delivered to my mailbox.
Firstly I thought that there was something wrong with Gmail service blocking forwarded messages. The same though had plenty of people using this service and writing their experiences all over the internet.
There is a post on Cloudflare Community website where started complaining about it and the loss in emails specifically when using Email Routing from Cloudflare.
When the message was rejected on Cloudflare it was reported as Delivery Fail and thrown below error message:
transient error (451): 4.7.650 The mail server [104.30.8.112] has been temporarily rate limited due to IP reputation.
There is no possibility to preview the message on Cloudflare or do anything with it. Because of that, it’s hard to diagnose the problem.
As much as I like Cloudflare, I, like other users were thinking about resigning from this solution with some more reliable options.
Going back to the error message, as I thought that this was a Gmail restriction that caused this, I quickly been able to change my routing rule to forward my emails to my Outlook email.
Typically, email servers that haven’t been able to deliver messages to the destination will keep trying again and again before dropping it. Redirecting from Gmail to Outlook will change the route and on the next repeat delivery attempt mail shall flow in another direction (Outlook).
This worked! Through a few days, I got most of the emails that failed to be delivered to Gmail.
Surprisingly, all of them have been delivered marked as Spam (Junk) by Outlook servers, hence there is something more here than just a grit between Cloudflare and Gmail.
However, another day came and I saw a similar issue reported by Microsoft servers:
upstream (outlook-com.olc.protection.outlook.com.) temporary error: Unknown error: transient error (451): 4.7.650 The mail server [104.30.8.115] has been temporarily rate limited due to IP reputation.
That cannot be a coincidence and something must be wrong with Cloudflare.
People start their complaints all over the place claiming that Cloudflare is not interested in fixing it. As easy as it sounds, fixing this kind of issue may not be as straightforward as can be imagined.
Because the service is supplied free of charge it is difficult to directly ask for help, but some users, who decided to subscribe to paid plans also complain about the lack of support received in that matter.
Days passed and the blocking situation on the Gmail end seemed to return to the previous rate where the email was blocked, and when re-tried after, typically, 20 minutes was accepted and forwarded.
However, the question remains about how serious consequences this unreliability can be for us and what we can do about it.
The ability to receive emails in my domain is quite critical. I have been using my email in my domain for years when I was with Google Workspace (Free version) and have been using it almost everywhere.
Independently if that causes any material loss or not, it’s hard to afford to lose some crucial communication only because some IPs on the Cloudflare end have been classed as spam.
Mitigation (temporary redirects) has been put in place, but in the meantime, I have been looking for some alternative solution in case in future this will happen again.
If you are just a single user in your domain, choosing another solution on the market (paid) may be straightforward. However, when you start building your users (family), paying extra for each user suddenly becomes very expensive unnecessarily. Choosing the right service needs to be considered at the beginning.
Having in mind my budget, I looked into the paid solution to utilise email forwarding to Gmail called Gmailify (Gmailify.com). Gmailify charges $6.99 per year per domain regardless of users. The service, similar in name to Google’s solution, but different in functionality, is created by Mercata Sagl based in Switzerland.
The author of this service creates it with Gmail in mind. The communication is only with Google servers and is done with IPv6 as a primary way, falling back to IPv4 when needed. It supports SPF, DMARC and DKIM under specified domains. Compared to Cloudflare Email Routing, where I can set SPF and DMARC, I haven’t found a way to use DKIM with Gmail and this is probably where the main issue in this forwarding service stands.
But is it?
Cloudflare signs each email, that lacks DKIM, with their key and in the report, it shows that the message originates from
email.cloudflare.com.
Somebody in the discussion said something that made me think before rushing with my decision to switch the boat.
“I worry about reliability and security with both gmailify and forewardemail…
Both feel like 1 man shows that grew out of a hobby. Which is usually great for OSS projects, but for a fundamental and critical service like email for entire domains, I dunno… Who knows if their infrastructure could crumple under attack, or if it could be an attack vector for someone to snag email 2FA codes, etc… The indispensable man could go on vacation or die…”
As a part of my utilisation of other Cloudflare services, I have been looking into their Cloudflare Radar and chart for Email Security. During the weekend when the issue was at its peak, it showed that 50% of emails worldwide had been classified as a potential threat. Coincidence?
It’s easy to jump to conclusions and start saying bad words about the service that is provided free of charge. Of course, there is room for improvement, but if something bigger happening on the horizon that nobody understands why, mitigation practices and patience shall be the first line of defence.
A discussion that was carried out in the community became, by some of the people, a rant on Cloudflare (now deleted by that person). A rant full of demands for service, that is supplied as it is, without any charge and obligations. Some people forget about it.
Cloudflare Email Routing with Email Workers - Approach 1
Spoiler alert: Approach 3 is the one I personally use.
The discussion is great when there is a conclusion that helps everybody, and this one arrived.
One user reminded, through his other post, that there is an option to supercharge Cloudflare Email Routing with Email Workers.
export default {
async email(message, env, ctx) {
//alias lists using RegEx. Examples below will allow for plus-addressing and subdomain addressing.
// (I have configured wildcard (*) DNS records to catch all sub-domains).
const user1_alias = [
"user1+.*@mydomain.net",
".*@user1.mydomain.net",
];
const user2_alias = [
"user2+.*@mydomain.net",
".*@user2.mydomain.net",
];
//primary emails
const user1_primary = 'user1@gmail.com';
const user2_primary = 'user2@gmail.com';
const user3_primary = 'user3@gmail.com';
//backup emails (still going to primary email, but via improvmx instead of directly from Cloudflare).
//you will need to create a subdomain (ex. 'imx') setup using improvmx, and improvmx aliases pointed to primary email
const user1_backup = 'user1@imx.mydomain.net';
const user2_backup = 'user2@imx.mydomain.net';
const user3_backup = 'user3@imx.mydomain.net';
//final catchall if primary and backup fails
const user3_outlook = 'user1@outlook.com';
//This tests alias against message "To:", showing the value of [true] if matching.
const is_user1_alias = RegExp(user1_alias.join("|"), "i").test(message.to);
const is_user2_alias = RegExp(user2_alias.join("|"), "i").test(message.to);
//action to take
//inside Try is to route to backup (ImprovMX to Gmail) to get by Cloudflare to Gmail issues.
//outside Try is to route to Outlook (final catchall) if both primary and backup fails.
var gmail_error;
try {
switch (true){
case (is_user1_alias):
try {await message.forward(user1_primary);}
catch(gmail_error) {await message.forward(user1_backup);}
break;
case (is_user2_alias):
try {await message.forward(user2_primary);}
catch(gmail_error) {await message.forward(user2_backup);}
break;
default:
try {await message.forward(user3_primary);}
catch(gmail_error) {await message.forward(user3_backup);}
break;
}
}
catch(improvmx_error){
try {await message.forward(user3_outlook);}
catch(outlook_error) {
message.setReject(outlook_error.message);
return message.setReject(outlook_error.message + '\n' + improvmx_error.message + '\n' + gmail_error.message);
}
}
}
}
Saving this, and the below codes for my references in case it will disappear from the community thread.
He has set email workers to route any emails that fail to be delivered through Cloudflare service to be delivered using another forwarding service, ImprovMX.
He set a subdomain to his main domain as an email service on ImrovMX where the main domain was still managed by Cloudflare.
Additionally, the Worker added the ability to catch and deliver emails to other subdomains, which is not possible directly through Cloudflare Email Routing, as well as introduced the ability to catch emails with + symbols in email addresses, which in the Gmail environment are used to categorise emails.
Cloudflare Email Routing with Email Workers - Approach 2
Another user in the same thread created another single email worker to deal with emails by incorporating catch-all technique.
export default {
async email(message, env, ctx) {
// defining all alias to user mappings we have
const users = [{
alias: [
"first@firstdomain.de",
"first@seconddomain.de",
],
primary: "first@gmail.com",
backup: "first@gmx.de",
}, {
alias: [
"second@firstdomain.de",
],
primary: "second@gmail.com",
backup: "second@gmx.de",
}, {
alias: [
"third_alias1@firstdomain.de",
"third_alias2@firstdomain.de",
],
primary: "third@gmail.com",
backup: "third@gmx.de",
}, {
alias: [
"fourth@firstdomain.de",
],
primary: "fourth@gmail.com",
backup: "fourth@gmx.de",
}, ]
// setting the target_user as our catch all user
let target_user = {
primary: "catchall@gmail.com",
backup: "catchall@gmx.de",
}
// checking if we have a user matching and overwrite the target_user with it
users.forEach(user => {
if (RegExp(user.alias.join("|"), "i").test(message.to)) {
target_user = user;
}
});
// sending mail out, first to google and on error to gmx
let reject_reason;
try {
await message.forward(target_user.primary);
} catch (gmail_error) {
reject_reason = gmail_error.message;
try {
await message.forward(target_user.backup);
} catch (gmx_error) {
reject_reason = gmx_error.message + "\n" + reject_reason;
message.setReject(reject_reason);
}
}
}
}
Both solutions are great as adding the ability to have a backup route if emails, for some reason, are rejected.
<script async delay="https://dariusz.wieckiewicz.org/g/serve.js?client=ca-pub-5380116874441486"
crossorigin="anonymous"></script>
<ins class="adsbygoogle"
style="display:block; text-align:center;"
data-ad-layout="in-article"
data-ad-format="fluid"
data-ad-client="ca-pub-5380116874441486"
data-ad-slot="9220966978"></ins>
<script>
(adsbygoogle = window.adsbygoogle || []).push({});
</script>
Cloudflare Email Routing with Email Workers - Approach 3
Going back to the main community thread the other user provided a very simplified code for email worker that utilising simple approach for a backup route.
export default {
async email(message, env, ctx) {
await message.forward("your gmail address").catch((err) => {
return message.forward("your alternative mailbox address");
});
}
}
I thought about going into a more complex approach (1 or 2) but ended up with a simplified one as it simply works.
Simplified approach in practice
Through Email > Email Routing > Destination Addresses I have added (and verified) the Gmail addresses of all users I got there as well as their alternative email addresses (Outlook).
You need to have added and verified all routes that you will specify in your Email Workers. If not the worker simply fails to deliver.
Following that, I created a few workers for each user and each scenario.
In Routing rules I changed each action for each custom address from Send to an email to Send to a Worker and specified worker.
In such a way, I utilised simple rules that stand as follows.
Anybody sending an email to my main address dariusz@ (in my domain) will be routed to my @gmail.com address. If for some reason Cloudflare will report an error in forwarding, it will try to re-route to @outlook.com.
With a little tinkering it is possible to set a 3-way approach, when the first two fail the third and last address is tried. This was initially considered in Approach 1. For most people, a single backup route shall do its job.
Successfully routed (through workers) emails will be marked in Cloudflare Activity log as Dropped. If both routes fail, it will show an error and will be marked as Delivery Failed.
Successfully forwarded emails, without workers, were marked as Forwarded in the Activity Log and are as Dropped with Email Workers. This may be a bit confusing, but, as one person on the community forum rightfully pointed out, it’s just a naming thing by Cloudflare. We just need to re-think the word Dropped as mail successfully dropped to mailbox. Happy with that interpretation :)
Managing main and backup route in one place - Google Gmailify
When you have a nicely set main address (in Gmail) along with your backup (in Outlook) it will be nice to have all email just in one place.
In the Outlook environment, we can easily set a forwarding rule, so any email landing in Inbox will be forwarded, this time by Microsoft servers, back to our main Gmail address.
I tested, and this is working well, but…
There is one issue with just enabling Forwarding in Outlook.
Only emails successfully landed in Inbox are forwarded further. If any of our emails are forwarded through Cloudflare to our alternative email will land in the Junk folder, this will stay there (only for 10 days, as per Outlook retention policy).
I don’t see myself going into Outlook from time to time just to check the Junk folder, hence I utilised Gmailify to help with that.
I mentioned Gmailify at the beginning, but this time I am referring to Gmailify, a service offered by Google. I know, it’s confusing!
Because of the same name for two different services done by two different authors, this is why did the creator of Gmailify.com put this on his website:
Not Your Google’s Gmailify
Gmail has a feature named Gmailify. It serves a different purpose though, linking Outlook and Yahoo accounts with Gmail. We link Gmail with custom domains instead. The name conflict is coincidental.
Gmailify from Google allowing to add your Outlook account in such a unique way, that it will not just fetch messages from another email like it is when setting it through POP3 or IMAP protocol. Gmailify will integrate Outlook email into your Gmail environment.
If something lands in your Junk folder on Outlook, once synced on the Gmail environment (it’s not instant), you will see this message in your Junk folder in Gmail. Great!
If you remove the message from the Junk folder from Gmail end, it will be removed from Outlook as well.
Thanks to forwarding set in Outlook for all incoming messages and Gmailify by Google, you can manage your main email and your backup email from a single point, Gmail.
Different approach
With the introduction of simple routing with Cloudflare Email Workers, I asked myself, is it possible to route to more than one email address?
In another post in the community I found out the straight and simple answer to how to achieve that.
export default {
async email(message, env, ctx) {
const forwardList = ["email1@gmail.com", "email2@gmail.com"];
for(const email of forwardList){
await message.forward(email);
}
}
}
With a small tinkering, the worker can become an interesting method of serving a purpose asked by Cloudflare users for some time, which is routing an email to more than one address from the same user (custom address).
I am not an expert but the rough idea will be:
export default {
async email(message, env, ctx) {
const forwardList = ["email1@gmail.com", "email2@gmail.com"];
for(const email of forwardList){
await message.forward(email).catch((err) => {
return message.forward("email@outlook.com");
});
}
}
}
With multiple backups:
export default {
async email(message, env, ctx) {
const forwardList = ["email1@gmail.com", "email2@gmail.com"];
const backupList = ["email1@outlook.com", "email2@outlook.com"];
for(const email of forwardList,backup of backupList){
await message.forward(email).catch((err) => {
return message.forward(backup);
});
}
}
}
I cannot guarantee that it will work, as this may have some negative effects if one email fails and another does not.
I am using a Cloudflare Email Route from my domain to a private Google Group email address from which I propagate messages to multiple users.
A last word to this post is the count for use of Cloudflare Workers.
We have a limit on free accounts of 100,000 workers’ requests per day. Interesting is that, when I set my rules, after like 50 emails forwarded (Dropped), the number of requests goes close to 600. Unsure about how exactly workers’ requests are counted, but I think I shall be well within the limit for my private use.
Regards.
