The great "email follow up" feature and how we find replies

Since the arrival of cold email onto the sales automation landscape, companies like mine and all my competitors have developed email automation systems that let you send a sequence of emails to your prospects until they reply, and then the sequence stops. This is a standard feature of most sales automation email systems, including this one. How do these systems detect that somebody has replied, and then stop the email sequence?

And… how do we do it in a matter of seconds?

I’ve learned over time that this is both a difficult and gravely important problem to solve. If the algorithm isn’t just right, some automatic follow-up emails that should go out don’t, causing lost sales opportunities; but an even worse scenario is when automated follow-up emails are sent to prospects, even after they’ve already replied. Some of my most mortifying, stomach-turning days in running this company have been when I’ve heard from users who experienced just that. Admittedly in our early days, back in 2015, it happened more frequently than I liked; now, however, it’s a rare occurrence.

Let’s break down the various techniques and challenges in solving this problem.

The easiest way is thread matching

Since our system is deeply integrated with only Gmail, our first step in detecting a reply is thread matching. Every email sent from your Gmail account is assigned a thread identifier. A thread ID is like a “conversation ID.” All the emails that belong to a single conversation (or “thread”) in Gmail have the same thread ID. Using the Gmail API’s threads.get method, we can programmatically fetch all the messages belonging to a thread.

So when a new message arrives at your inbox, we examine its thread ID and match it to all the thread IDs created when you sent your campaign to see if there’s a match. If there is, then we know that there’s some kind of a reply. Most bounce-backs are tied to the original thread ID, while most automatic replies like “out of office” messages arrive under a different thread ID. So bounce-backs can easily be identified by our system, but out-of-office auto-replies can’t.

Separating bounce-backs tied to the thread from real replies tied to the thread is easy, because bounce-backs have lots of indicators, including a Subject line of “Mail Delivery Failure.”

So how does Gmail decide whether to “thread” a reply email with the original sent email? Google has published a post on how it determines which emails to attach to a conversation, but they don’t go into too much detail. Their post states that the reply has to have “the same Subject,” but that’s obviously not entirely accurate. The reply can have “Re:” or “Fwd:” prepended to the Subject, and it will still work. And, as already described above, bounce-backs are still tied to the thread, and they have entirely different Subject lines.

SMTP geeks might be familiar with the References and the In-Reply-To headers. Email clients like Gmail, Outlook, and Apple Mail use these headers to determine what emails are replies to which other emails. The problem for cold email apps like mine, however, is that these headers aren’t easily exposed through the Gmail API like the thread IDs are, so we can’t use traditional SMTP headers to find messages that are replies to any of your campaign emails. For a detailed explanation of the References and In-Reply-To headers, see this article.

The harder way is Subject line matching

The danger of using only the thread ID matching system to find replies is that sometimes a person will reply, but based on the conversation matching rules from Google, Gmail will assign it its own thread, therefore breaking the entire thread ID matching system. We’ve found that this happens both in the situations Google mentions and when:

1. The recipient alters the Subject line before replying.
2. Or, the recipient’s email system alters the Subject line before replying. For example, some systems pre-paid the word “EXTERNAL” to the Subject line of an email outside the person’s own organization.
3. Or, the recipient’s email is forwarded to another email, and then that email address replies.

That’s when we have to do more work to figure out if someone replied. In this case, we rely on good old-fashioned Subject line matching. For example, if you send a campaign with a Subject of “Let’s book a meeting,” then we look for emails in your inbox that have the same Subject line of “Let’s book a meeting.” However, this presents some complexities as well. If we treat everything we find that matches the Subject as a reply, then emails with the following Subject lines would also be treated as replies when they should not:

Out of office Re: Let’s book a meeting

Auto-response Re: Let’s book a meeting

Undeliverable Re: Let’s book a meeting

Based on these examples, you might then think, “Ah ha, just look for the presence of Re: in front of the Subject line.” Well, that’s a good idea, but it doesn’t get even close to finding all the legitimate replies while weeding out the auto-replies. Some email systems add the hint “EXT” or “EXTERNAL” to the Subject line when the sender is from outside the recipient’s organization. Meaning, let’s say you send an email with Subject “Let’s book a meeting” to a prospect. The prospect’s email system might alter the Subject line to be: “[EXTERNAL]: Let’s book a meeting.” Then they reply, and the reply comes through with the Subject: “Re: [EXTERNAL]: Let’s book a meeting.” Not only does this break the thread matching technique, but also it makes it so that just looking for “Re: Let’s book a meeting” doesn’t work. Well then, the next logical step might be to look for the presence of “Re:” and the original Subject line but not necessarily together. After all, that would work for the above example with “[EXTERNAL]” in the Subject line. But that still doesn’t solve for all cases, as shown above with the “out of office” example.

Finally, you throw your hands up in the air and think, “I know! As long as it has “Re:” and the original campaign Subject somewhere in there, but there’s nothing before the “Re:” then you’re golden, and you know you have a legitimate reply. But still, you can get a Subject line like:

Rif: Re: Let’s book a meeting

What is “Rif”? It was added by a Spanish-language email system.

Unscrupulous emailers who send with “Re:” already in the Subject

We don’t advocate doing this, but many users do send campaigns where the Subject line already includes “Re:” — for example, a campaign might go out with this Subject:

Re: We’re having a sale today

Why do marketers do this even though the email isn’t a reply? To trick the recipient into thinking it’s a reply to an existing conversation, to increase the likelihood that the recipient will open the email.

I consider this practice spammy and don’t like it when users do this. But even so, we still have to detect appropriate replies if they’re using auto follow-ups. In this case, the Subject of the reply is usually the same, because most email systems won’t then add another “Re:” to the Subject upon the reply. What some email systems will do, however, is add a number to indicate how many replies there have been. Meaning, if a recipient replies, some email systems would alter the Subject to be

Re[2]: We're having a sale today

to indicate that there are now two messages in the conversation. That’s another example of why we can’t look for just “Re: Original-Subject” to detect replies.

Even more complicated is the issue of personalized Subject lines

So figuring out what’s a reply when the thread doesn’t match is complicated, and I haven’t even discussed the most important complexifier. What if you personalize your Subject?

For example:

Let’s book a meeting, {FirstName}

Now, we have to possibly look for all variants of your Subject in your Inbox and then determine which are legitimate replies.

One final complication: sending with SMTP rather than natively with Gmail

For a while now, we’ve offered the ability to create campaigns using your Gmail account, but attaching a third-party SMTP server to your account through which the emails are actually sent. We offer this so you can send cold email sequences with automated follow ups beyond Gmail’s sending limits.

Because this option bypasses Google’s entire infrastructure when sending campaigns, we offers users the choice to have their emails logged in the “Sent Mail” folder or not. When they’re not logged, the emails send much faster. If emails are logged, then when replies come in, Gmail matches those replies to the thread, so thread matching works. If, however, a user opts for the Fast SMTP option, the emails aren’t logged in “Sent Mail,” and so thread matching can’t work because there’s no original email to which the reply can be tied. In these cases, we have to rely solely on Subject line matching. However, because the Fast SMTP option presents this problem, we don’t allow users to set auto follow-ups on campaigns that use the Fast SMTP option.

So how do we do Subject line matching?

To determine our algorithm for appropriate Subject line matching, I looked at thousands of anomalies collected by our system over our history of four-plus years. I needed whatever algorithm could cover all of these scenarios.

Legitimate replies

Re[2]: Sponsored Collaboration by Bliss Body

[Walks LLC] Re: Re: [Walks LLC] Re: Host your Experiences on byo – A Global Travel App

[EXTERNAL EMAIL] Re: Band Update

[STUDENT] Re: [STAFF] Band Update

Rif: Re: Ajay, the big shift that’s happening

Re: EXT: Re: Everything that you want to know

Not legitimate replies

RE: Publishing with JoVE Re: Avoid hitting Gmail’s sending limits

NO REPLY – YOUR RESPONSE HAS NOT BEEN PROCESSED Re: Avoid hitting Gmail’s sending limits

About Contemporary Classes Re: Avoid hitting Gmail’s sending limits

Here’s our algorithm.

1. We look at all emails that hit your inbox with a Subject that contains your original campaign Subject line. If you personalized your Subject, we break your Subject apart and look at it in its individual pieces, ignoring the personalized parts, and find emails that contain all parts of your Subject, minus the mail merge variables.
2. Next, we look for the first colon in the Subject line.
3. If there’s no colon at all, it’s not a legitimate reply.
4. After finding the first colon, we remove everything to the left in square brackets.
5. Then we look at what remains to the left of the colon. If it’s more than three non-whitespace characters, it’s not a real reply. If three or fewer, it is a real reply.

Still, though, it’s not perfect and remains a work in progress.

Here’s an example of a reply to a campaign that I sent, where the reply is separate from the thread, passes all the checks in my algorithm, but still isn’t a legitimate reply.

The timing of reply detection

SendHustle detects replies within a matter of seconds of you receiving them. This is a big change to the system that happened in August 2023 — before that, SendHustle scanned every few hours.

Note: A situation where we can’t detect replies

SendHustle gives you the ability to set a Reply-To address for a campaign. When replies go to that address, SendHustle can’t detect them (unless the emails to that address also go to the email account from which you’re sending the campaign).

That can be problematic for campaigns with auto follow-up sequences that only stop on reply. If you’ve set a reply-to address on a campaign with a stop-on-reply auto follow-up sequence, SendHustle will show you a warning before you send.