SPF records and how they work

The mysteries of the internet : SPF edition

First of all, SPF stands for “Sender Policy framework”.
Now you know.
Period.

 

 

 

 

Just kidding.

An SPF record is used as a “sort of ” security/anti-spam measure in order to protect your outgoing mail from being sent FROM a certain list of sender locations.
You read it right : “from” (not “to“).
It’s in no way a fail-safe antispam solution, but it will decrease the amount of mails sent in your name significantly.
Office 365 even requires this specific record on first setup (although you can skip this step eventually).

Who didn’t configure his Outlook Express in the old days to send mail from as his boss to fake a “you’re fired” mail to your new colleague ? Right ? Right ? N-Nobody ?
Oops. Sorry, Bjorn.

 

Comment ça marche ?

In order to understand how things work, wel’ll start out with a little example from our fake (?) company Shortstraw LLC and their Office 365.
After setting up your O365  subscription, your SPF records looks a bit like this :

Note the “subtle” reference to mxtoolbox.com

I deliberately used the expression “text record” as SPF is nothing more than a TXT record with a specific markup. Most hosting companies have a separate entry box for SPF nowadays, but in the background it’s still a TXT.

Let’s split up everyting in this text record, so we can understand completely what exactly it is that Microsoft wants us to do.

v=spf1 : 
All this does is stating the obvious.
This record is “version SPF1”.
To make things confusing, there’s an unrelated spf2.0 record, which stands for “Sender ID”

include:spf.protection.outlook.com :
The include value *drumrolls* includes the values of another SPF record, in this case a large list of IP addresses of Microsoft servers.

The very same website I’ve used in the example above (MX Toolbox) easily provided me with this information. You can see that in itself this record loops through to more SPF values in spfa.protection.outlook.com.
This has to do with the maximum accepted length for a TXT record, which is limited to 255 characters, as explained on this RFC page.

-all
The final entry in the record is also the most conclusive one.
The “-all” stands for “disallow all other entries”

Other entries

The above example was just a textbook sample provided by Microsoft to get you started. It should be sufficient for the average O365 setup.

Reality however will often slap you in the face with a more complex situation where you’ll be needing to configure all-in-one printers, old software packages with support for mail over port 25 only, websites sending from something@your_domain_name, …
They all have a couple of specific things to keep in mind when configuring your SPF record.

A handy tool for helping you configuring your SPF record is https://www.spfwizard.net/, which does exactly what the name implies.

Other possible entries you might use in your record :

A:
It’s usually followed by an actual DNS A-record
An example will make things clear.

v=spf1 include:spf.protection.outlook.com a:mail.suamae.br -all

The classic O365 SPF record was added a:mail.suamae.br, which will allow the IP resolving to this Brazilian mail server name to be able to send mails from shortstraw.be’s domain name as well.

Just adding “A” and not defining anything behind it, will result in all A-records in shortstraw.be being able to send mail from this domain name.

MX:
Works similar to the A-entry.

v=spf1 include:spf.protection.outlook.com MX:uwmoeder.com -all

The example above will allow all the mailservers in the domain uwmoeder.com to be able to send mail from shortstraw.be

Double checking on this, it will resolve to redirect.ovh.net in this case.

Just adding “MX” without a domain name, will allow all MX entries in the shortstraw.be domain to send mail.

IP4:
You may enter one specific IPv4 address or a group of IP addresses in the typical “slash” notation (it’s called “CIDR format” in big-boy-language).
Note the omission of the letter “v” in ip4. Pay attention to this common typo.

v=spf1 include:spf.protection.outlook.com ip4:194.78.56.8 ip4:37.230.164.0/22 -all

This will allow the single host “194.78.56.8” and the group of IP addresses “37.230.164.0 to 37.230.167.255” to send mail from your domain name.

IP6: 
See above.

v=spf1 include:spf.protection.outlook.com ip6:2a01:4f8:d16:1355::2 -all

Completely the same, but for IPv6 addresses.

~all
Replacing the “-all” entry with this “~all”, results in the mails from the “not-allowed” list to be marked as so-called “Soft Fail”. You can see the result of this in the headers of a received mail as something like …

Received-SPF: softfail (google.com: domain of transitioning mail@example.com does not designate 203.0.113.2 as permitted sender) client-ip=203.0.113.2;

 

+all
Stating the “+all” addendum, ignores everything you just added in the previous part of the SPF record.
Your mails might (or might not, depending on the server) be tagged with an extra header that says something in the likes of “this mail is probably not OK, but it doesn’t matter anyway”.

?all
Almost the same as “+all”, with the exception that your mails will not be flagged.
?all stands for “No policy statement” and simply does not hold the SPF record in account.
Often used when trying to delete an SPF record without actually deleting it.

Now how does this REALLY work ?

Every self-respectable and up-to-date mail server will do an SPF checkup, when receiving mail.
That’s right, even Exchange Server 2003 can do this, updated to Service Pack 2. (not sure how it’s implemented in an Exchange 2000, though  – I’ve seen software by GFI that used to be able to do this, but if you’ve come to this page looking for an antispam solution for your Exchange 2000, you can probably guess what I’m thinking right now…)
In a side-note, this function is usually referred to as “Sender ID” in an Exchange environment.

It’s usually a hard-coded feature with nothing much to tweak, except for turning it on or off.

A basic representation of SPF functionality (picture shamelessly stolen from this site)

One simple image explains how it works.
Mail server A sends mail from a certain domain.
Mail server B receives mail from this domain and does a quick DNS lookup looking for any TXT records containing “v=spf1” after which he interprets the information like I’ve explained about one page up.

That’s it !

You now know how an SPF record works !

 

More information

I’ve taken the liberty of compiling a small list of handy websites should you want to read more on the subject.

  1. Pretty hardcore and a tough cookie to read ; the official RFC for SPF records :
    https://tools.ietf.org/html/rfc7208#section-6.1
  2. SPFWizard, for creating your own custom SPF records on the fly :
    https://www.spfwizard.net
  3. DKIM, a next-level form of protection your domain.
    It’s not covered on this page, but it’s just a friendly reminder to let you know that there’s more than one way to perform mail protection :
    https://blog.returnpath.com/how-to-explain-dkim-in-plain-english-2/
  4. Mxtoolbox.
    Your trustworthy website on troubleshooting DNS and mail related stuff.
    https://www.mxtoolbox.com
  5. Microsoft Message Header Analyzer.
    Not mentioned in this page yet, but a great tool nevertheless.
    If reading a wall of text is too hard, copy your mail headers in this website and it will provide you with readable text. Huzzah !
    https://testconnectivity.microsoft.com/?tabid=mha

OFFICE 356 MIGRATION (4) – 3 steps ahead

Remember Oliver’s company (Shortstraw LLC) mail profile in one of our previous posts ?
Refresh your memory if you stumble upon this website and have forgotten / not read the previous one.
I will base this actual setup on our findings in that post.
The hardware and operating system upgrading procedures, that I spoke about, will not be handled here.

Preparing the environment

Since we’ve established our to-do list, we can now start the procedure in which the customer will experience the least downtime.
Depending on the expectations of the customer, you can either perform all these actions on-the-fly or prepare yourself thoroughly. In this case we’ll take the long(er) road.

After having created the 30-day trial (or go ahead and buy one instantly through Microsoft or a Cloud Solutions Partner)

At least, we’re welcome

You’ll be greeted with something similar to the screenshot above.
Clicking the upper left square icon will get you into the apps menu.
Depending on the user rights and licenses, you’ll see less or more icons, representing the programs and apps you’re allowed to use.

One important icon you’ll see, is the “Admin“.
Users that have administrator rights will be provided with this option.
In this example, our admin user has a fully working E3 license (you get 25 of these buggers in an O365 trial), which is something that’s “not done” in a real life situation.
Were you to upgrade this 30 day trial to a full tenant, I strongly suggest stripping the admin account of all his licenses. It’s bad practice to use your O365 admin account for anything else than … well… admin purposes.
We’ll get into licensing later.

O365 admin menu
The admin menu

Click on “Admin” and a specific administrative portal opens.
You can take a short tour of everything by clicking “Start the tour” if you want or read onward and click “Skip”.

On the left side of the admin page you’ll notice the admin menu (which is deliberately placed as a screenshot on the right side of this page, just to confuse you)

First thing you’ll be wanting to do is to create the situation with the correct internal mailflow, user rights etc.
Remember : as long as you don’t change the MX record in the customer’s DNS settings, NOTHING will happen to the existing setup.
You can safely mess around until you’ve got the flow up and running to your own standards.

Remembering our previous conclusion, we’ll start creating our users’ mailboxes first.
If you want to get this right at first try, you’ll want to include the domain first as an “inbound” domain into your O365 tenant. This will allow you to create user names ending in @yourdomain.com instead of @yourdomain-com.onmicrosoft.com.
Again, including the domain name will not change your current mail flow.
DON’T PANIC !!!

Open the Setup menu and click “Domains”

One domain will be listed by default.
This is you tenant name (in my example “shortstraw.onmicrosoft.com”) and cannot be removed.

Click “Add Domain” and fill in the desired domain name, after which you click “Next”.

In order for Microsoft’s servers to verify your identity and double checking if you’re actually the owner or admin for your added domain, you’ll be given the choice of either adding a TXT record or adding a fake MX record into your own DNS management software at your hosting company’s admin package.

In my case, the lovely French hosting company “OVH” will be my location to turn to.

A line of TXT in OVH’s DNS management

Eventually, practically every DNS hosting company allows you to manage your settings through some form of admin portal for easy setup purposes.
Once the record has been added, you can click the “Verify” button to let Microsoft doucble check the creation of the record.

Troubleshooting :

If the TXT record is not yet found, according to the O365 domain verification wizard, you can always start your troubleshooting on a global level by surfing to https://dnschecker.org/DNS Checker creates a worldwide DNS lookup, using all sorts of DNS servers to see if your DNS record has propagated already to all locations.
Usually DNS propagation for a brand new domain record will probably not take a lot of time. It’s those record changes that tend to take longer.
Successful verification will get you to this screen

From this point on, you’ll be able to pick your domain name already in the user creation wizard.
Should you choose to continue, more DNS records will be added.
Skip forward to user creation. (link not yet implemented, because too lazy)

The screenshot above will give you a sneaky Microsoft question, with the default option set to “Set up my online services for me”.
Though Microsoft might say “Recommended”, I strongly disagree here.

“Why’s that”, you say ?
In case you decide to stop your O365 adventures and want to move on to a new platform for mail, you’re going to have to go through a lot of hassle to set this straight again.

Always choose to manage your own DNS records and click “Next”. Unless you’re REALLY pissed about your current DNS provider. In that case, I still suggest just finding another one. BUT NOT MICROSOFT FOR THE LOVE OF GOD.

Choose what you need (or select all)

A step that has been neatly added in the onboarding wizard, since Q4 of 2017 is the “Choose your Online Services” wizard.
This narrows down the amount of DNS records for you to add, according to the active checkmarks.
I’m going to select all of them, because I know my end-customer Oliver Shortstraw will need toe Exchange parts as well as the Mobile Device Management.
He’s also a somebody that changes his mind in the blink of an eye, so just to be sure we won’t have to setup anything else later, I also picked “Skype for Business”.

A somewhat huge list of DNS records will appear, for you to fill into your favorite DNS hoster *cough* OVH *cough*

A wild list appears. You crit it for 9000. It was super effective

Now in order to fully understand what’s going on here, I’ll explain in detail the actual stuff that’s going on. Teach a man to fish etc.

FINAL WARNING (I won’t repeat it again) DO NOT CHANGE THE MX RECORD JUST YET (unless this is a brand new setup for a brand new domain, then go ahead and have fun)

CNAME : autodiscover > autodiscover.outlook.com
This record basically tells your Outlook client to read a pre-made config file on a Microsoft server.
Thus allowing you to just enter your e-mail address and password in the Outlook setup wizard, instead of having to go through the hassle of manually setting up your O365 config.

CNAME : sip > sipdir.online.lync.com
Refers to the actual SIP server for using Skype for Business/Lync/Teams. Your communication client will connect to this server and this server will in place patch you through to the geographically most redundant SIP server.

CNAME : lyncdiscover > webdir.online.lync.com
This server uses the same Autodiscover protocol as the Outlook one.
It patches you through to the correct Microsoft server cluster where your tenant is hosted, as well as other various kinky background processes. Dragons be here.

CNAME : enterpriseregistration > enterpriseregistration.windows.net
Basically serves as a registration server (duh), so the Microsoft servers know what mobile device was added to the tenant for so-called “conditional access”

CNAME : enterpriseenrollment > enterpriseenrollment.manage.microsoft.com
Enrolling (again, duh) Windows mobile devices and managing them through Microsoft Intune, requires these servers.

TXT : v=spf1 …
Specifies the server(s) that may send mail, originating from your domain name.
More on SPF records in an other post.
For now, follow the suggested entry, which – shortly explained – allows a group of servers that are defined in the name spf.protection.outlook.all to send mail from your domain. All others are denied.

SRV : _SIP
Together with the sipfederationtls entry, these are usually the more tricky ones to enter, depending on the DNS management tool.
[An example from the one.com hosting panel].
This specific entry provides the security layer.

SRV : _SIPFEDERATIONTLS
This entry states the TCP port 5061 is being used for everything federation-related in communicating over SIP. A Classic SIP port uses port 5060. Microsoft likes to do things in their own special way…

MX : xxxx-yy.mail.protection.outlook.com
An automatically generated server name, based on your domain name and domain extension.
MX is short for Mail Exchanger and tells other mailservers in the world where to go dump its mail for your specific domain name.
The second you change this record in your DNS management (and it gets propagated world wide, bla bla) your mail will be directed to the server(s) in this record.

<lazy mode> Let’s assume for the time being, that our test company does not care much for just a little downtime and let’s change all these records in our DNS management tool </lazy mode>

Clicking the “Verify” button at the bottom of the wizard page will get Microsoft’s O365 server to check all your entries. Depending on the DNS management tool and the hosting company, this might take a couple of seconds up to a couple of hours.

After a successful verification of all entered services, let’s move on to creating new users in the next post.
For now, pat yourself on the back for a job well done and have a refreshing beverage.

SMTP relay on your Windows server for use with Office 365

Antique software ? No TLS/SSL support for outgoing mail ?

No problem !

If you stumble upon this article through a Google search (who uses Bing, anyway…), you’re probably wondering how to solve the following issue (or something similar) :

It’s all about the Pentiums, baby !

You have this old invoicing software that doesn’t get updates anymore ever since 2008 and relies on port 25 – unauthenticated – to send mails through your ISP’s outgoing mail server.

You happen to have this beautiful product called ‘Office 365’ and use its mail functionality for your own domain name. This domain name is used as outgoing mail domain in your software.

Luckily, you still have a Windows Server randomly lying about (hopefully 2008 R2 or higher, but this trick works with older stuff as well – also : this works on a Windows 7/8/10 , even though the IIS install method will be different)

Installing SMTP in IIS

  1. 2012 R2 Server
    Install Internet Information Services (IIS)

    1. In Server Manager, select Add Roles.
    2. On the Before you begin page in the Add Roles Wizard, select Next.
    3. On the Select Installation Type page, select Role-based or Feature-based installation.
    4. On the Select destination server page, choose Select a server from the server pool, and select the server that will be running SMTP services. Select Next.
    5. On the Select Server Roles page, select Web Server (IIS), and then select Next. If a page that requests additional features is displayed, select Add Features and then select Next.
    6. On the Select Role Services page, make sure that Basic Authentication under Security is selected, and then select Next.
    7. On the Confirm Installation Steps page, select Install.

      Install SMTP

      1. Open Server Manager and select Add Roles and Features.
      2. Select Server Selection and make sure that the server that will be running the SMTP server is selected and then select Features.
      3. On the Select Features screen, choose SMTP Server. You may be prompted to install additional components. If that’s the case, select Add Required Features and select Next.
      4. Select Install. After the installation is finished, you may have to start the SMTP service by using the Services snap-in for the Microsoft Management Console (MMC).

  2. 2008 R2 Server
    Install Internet Information Services (IIS)

    1. In Server Manager, select Add Roles.
    2. On the Before you begin page in the Add Roles Wizard, select Next.
    3. On the Select Server Roles page, select Web Server (IIS) and select Install.
    4. Select Next until you get to the Select Role Services page.
    5. In addition to what is already selected, make sure that ODBC Logging, IIS Metabase Compatibility, and IIS 6 Management Console are selected and then select Next.
    6. When you’re prompted to install IIS, select Install. You may need to restart the server after the installation is finished.Install SMTP
      1. Open Server Manager and select Add Roles and Features.
      2. On the Select Features screen, choose SMTP Server. You may be prompted to install additional components. If that’s the case, select Add Required Features and select Next.
      3. Select Install. After the installation is finished, you may have to start the SMTP service by using the Services snap-in for the Microsoft Management Console (MMC).
  3. Windows 10 / 8 / 7Use the instructions on https://www.howtogeek.com/112455/how-to-install-iis-8-on-windows-8/
    You’re probably oing to need at least the ‘Professional’ version of the operating system to be able to pull this one off.

Configuring the SMTP Service for use with O365

  1. Set up SMTP
    1. Select Start > Administrative Tools > Internet Information Services (IIS) 6.0.
    2. Expand the current server, right-click the SMTP Virtual Server, and then select Properties.
    3. On the General tab, select Advanced > Add.
    4. In the IP Address box, specify the address of the server that’s hosting the SMTP server.
    5. In the Port box, enter 587 and select OK.
    6. On the Access tab, do the following:
      1. Select Authentication and make sure that Anonymous Access is selected.
      2. Select Connection > Only the List Below, and then specify the IP addresses of the devices that will be connecting to the SMTP server, such as printers.
      3. Select Relay > Only the List Below, and then specify the IP address of the devices relaying through this SMTP server
    7. On the Delivery tab, select Outbound Security, and then do the following:
      1. Select Basic Authentication.
      2. Enter the credentials of the Office 365 user who you want to use to relay SMTP mail.
      3. Select TLS Encryption.
      4. Select Outbound Connections and in the TCP Port box, enter 587 and select OK.
      5. Select Advanced and specify SMTP.office365.com as the Smart Host.

/!\ Restart the IIS service and the SMTP service.  /!\

 

Actually testing, before applying

You can test SMTP relay services without using your software that needed it in the first place.

To test SMTP relay services, use the following steps.

  1. Create a text file using Notepad or another text editor. The file should contain the following code. Replace the source and destination email addresses with the addresses you will use to relay SMTP.
    FROM: <source email address>
    TO: <destination email address>
    SUBJECT: Test email
    
    This is a test email sent from my SMTP server
    
  2. Save the text file as Email.txt.
  3. Copy the Email.txt file into the following folder: C:\InetPub\MailRoot\Pickup.
    Try to copy it instead of just moving it. The mail file will disappear.
  4. After a short time, the file should automatically be moved to the C:\InetPub\MailRoot\Queue folder. When the SMTP server delivers the mail, the file is automatically deleted from the local folder.

    Warning: If the SMTP server can’t deliver the message, a non-delivery report (NDR) is created in the C:\InetPub\MailRoot\BadMail folder. You can use this NDR to diagnose delivery issues.

     

Troubleshooting

This is where most guides fall short.

  1. Read the mails, that appear in the Badmail folder.Usually there will be a reason for refusal or non-delivery explained in these files.
    If you have ‘show file extensions’ turned on, they will appear as .BAD files.
    Open with Notepad or a similar pogram to see something like this :


    Self-explanatory, I guess.

  2. If mails were to actually arrive at their destinaton, but marked as “phishing” or appear in the spam folder of your recipient, chances are pretty high, your software package is still sending out through the wrong outgoing mailserver.
    Seeing as I’m not a psychic, I can’t know how to configure outgoing mail in every piece of software.Press F1 

    An other reason for your mails being marked as “phising” (and I deliberately left this near the end of this article), is also related to the above (still sending out through your ISP’s SMTP server using an Office 365 mail address).
    HOWEVER…

    When the mail arrives – even though its marked as spam –  this means your ISP was able to actually deliver it. In Office 365 cases, this usually means that your ISP is not allowed to send out as your O365-linked domain name.
    I’ll provide a detailed how-to on interpreting mail headers in a later post, but for the purpose of this exercise, let’s presume the mails are being marked because of the above.

    In this case, just adding/editing an SPF record that relates to your ISP will be enough.

    I hear a couple of muffled hillbilly-sounding voices in the background asking me ‘what in tarnation is an SPF record’ and how does that work ?
    Read and weep : https://blog.returnpath.com/how-to-explain-spf-in-plain-english/

    For all you Belgians out there, these are the values you need to include in your SPF record for the bigger ISP’s :

    Telenet
          include:_spf.telenet-ops.be

    Proximus
          include:ispmail.spf.secure-mail.be
          include:bgc.spf.secure-mail.be

    The somewhat attentive reader might be asking himself ‘if I could just add an SPF record using the records for my ISP, then why did I even bother reading this article ?

    I’ll keep the answer very simple : EVERY user of this ISP will have the ability to send mail as your domain name, without passing some form of verification in this case.

    AAaaaaand we’re back to the 80’s/90’s , where it was common fun and games to change your mail address in Outlook Express or other old mail software.

    Random :
    Did you know that Outlook Express’ executable file   msimn.exe was named, because it’s short for MicroSoft Internet Mail and News ?

  3. Mail does not arrive and the NDR gives an authentication error :

    Did you change your O365 password for the account that you use to authenticate for the SMTP connection ?
    Yes you did. (or you just made a typo)

  4. Mail does not arrive and the NDR gives a ‘does not permit to send as’ error :

    Most often, this occurs when not sending as the account that is the SMTP relayer.
    Your fancy 80’s software probably sends as (e.g., which is latin for exempli gratiā and is often translated as example given – just sayin’) invoicing@shortstraw.be, while your SMTP’er is oliver@shortstraw.be.

    To solve this, either change your outgoing mail address in your 80’s software, change the authenticating O365 in your SMTP relay server OR…. *drumrolls* add ‘send as’ permissions to the invoicing mailbox for Oliver’s account.

    What’s that ? invoicing@shortstraw.be does not exist in your O365 tenant ?
    Yeah… I figured as much…
    Add is as an alias to Oliver’s box or create a new box. Choice is yours.

 

Oh, and in a footnote : you will not find the mails sent through your relay’er in the resp. mailbox’ Sent Items.
Handy for troubleshooting… NOT