Tag: Shopify

Shopify Webhooks driving AWeber

This post is a solution to a problem I had with the AWeber Shopify integration. To get the most out of this post, check out the original problem here.

…the continuation…

Being that I was spending 30+ minutes every day manually solving this problem, it was important that I had a MVP solution quickly. I took a step back to think about my immediate needs and the future direction that I would like to take this solution and came up with the following constraints:

  • Need to get something basic up and running quickly that can be easily iterated upon
  • Everything needs to be deployable to the Google Cloud Platform (and not cost a fortune to run)
  • The solution should be something that I can eventually monetize. This means a clean, UI based integration in the Shopify ecosystem (i.e. support for Node, React, Next) and the need to be able to handle many Shopify stores and scale appropriately.
  • Anything built must be easy to fit into the multichannel lead generation vision of Threddies. Eventually this would need to become the way that all leads get added to my email service provider without using any direct integrations.

Shopify Webhooks

I did a little digging and realized that I could solve just about every variant of the core problem if I were notified any time a customer was created or updated in Shopify. Conveniently enough, Shopify provides webhooks for both of these cases (in addition to many more). Webhooks are great for creating quick integrations and very easy to handle using Google Cloud Functions.

I prototyped the ‘create customer’ webhook and had something up and running in no time for my test store. I also started to think more about how I can quickly iterate on webhook based integrations in the future. The most simplistic integration using webhooks doesn’t require authentication, but it does require verifying the data sent is actually from the expected Shopify store and not just anyone on the internet. This is done using the X-Shopify-Hmac-Sha256 header. When you receive the webhook data, you need to verify the data in the body by generating this value (using a private key) and comparing it with what Shopify sends. There are two different ways to do this and it depends on how you integrate with Shopify. The preferred approach is to develop a Shopify app which has it’s own key that you can use to verify every authorized store that is using your app. The drawback of this approach is that you need a full blown Shopify app that implements the store authorization flow and requires some UI work. Since I’m not a React expert, I opted to take the second approach and avoid the UI by having each Shopify store owner that was going to be using this integration provide their store’s key to me. You can get this key by going to your store’s settings page and registering a webhook in the ‘Notifications’ area. This key is used to verify the integrity of all webhook data sent. Things I learned from this step:

  • Cloud Functions would likely not be the final way of deploying this since it did not provide a way to surface a UI in a customer’s Shopify store.
  • The Cloud Function for each webhook is going to have a lot of repeated boilerplate for verifying the integrity of the data sent and handling responses/errors. I would also need a more centralized storage location for keeping Shopify Store specific data so that it wouldn’t need to be duplicated in every Cloud Endpoint
  • Shopify webhooks require a timely response, so putting any heavy lifting in the cloud function is not going to happen. Take too long to respond, and Shopify will deregister your interest in the webhook data. This started to get me thinking about how to recover from this scenario.

Hookup to AWeber

I was getting data from Shopify and verifying the integrity of the data, but at this point nothing was happening with it. In order to get the data into AWeber, I had to again obtain some information from each admin of the Shopify stores that I was integrating with. At the bare minimum, I needed an AWeber account id and a list id to add all subscribers to. I also needed my customer to authorize my app to interact with their AWeber account. More requirements for UI, but I still wanted to put that off and focus on solving the original problem. I also didn’t want to force my users to go into AWeber in order to add my integration. I found this great NodeJs wrapper library around AWeber’s API that did everything I wanted it to do. Using this, you can enter your AWeber integration information and use it to generate an auth URL that you can send to your customer. They can then use that URL to provide your integration with the necessary permissions to their account. They send back a verifier code after authorization that you can use to get all of the necessary tokens needed for your integration to access that account. This information doesn’t change unless the user removes your integration, so it works perfectly until I actually setup the full blown authorization path in my Shopify App.

There were two AWeber issues that I discovered at this point: there is no way to add the subscriber’s location without using an originating ip address for geolocation. This means the ability to send using the subscribers local time window is not something that will work with subscribers added this way. Also, AWeber integrations are confirmed opt-in by default; AWeber would not turn this off for everyone using my integration, so I had to turn it off for each account/list I wanted to use it with and tell my customers to do the same. This was necessary since my integration already leverages Shopify’s confirmed opt in and I didn’t want to confuse my subscribers by making them do it again. The Shopify webhook payload already includes a flag for ‘accepts_marketing’ and I verify that this flag is true before attempting to add any information to AWeber. Things learned at this point:

  • I really need that UI!
  • The create customer/update customer flows look very similar from a Cloud Function perspective, so there needs to be some consolidation.
  • The AWeber deauth path needs to be handled. This can’t be done in a cloud function since it’s so far removed from the user capable of fixing this problem. For now, just alert on the error when it occurs and add it to the list of issues to handle later. This is another item that indicates the need for a systems health check in the customer’s Shopify store (and a way to recover all subscribers added between the time when a failure occurs and the Shopify store owner resolves the problem)

Ready for Production

At this point, things were working well enough that I felt confident allowing this integration to start doing my job for me. Before moving everything to production, I refactored everything to eliminate the obvious problems that I saw at this point.

Instead of Cloud Functions being the primary entry point, I created a NodeJs app to do this instead. This allowed me to setup all of the webhook routing inside this app and put all of the webhook verification and ‘health check’ code into this app. If there was a problem, I could fail fast without fear of Shopify deregistering the webhook.

This also provided a place where I could add all of the UI code for the integration and the intelligence for recovering from failures. This app can also morph into a frontend that is capable of handling and routing any future webhook integration that I want to create.

I then deployed the NodeJS app to Google App Engine. At first it wasn’t working and the errors indicated that the Next.js build step wasn’t occurring on deployment. I solved this by adding a custom build step that is automatically run on deployment by Google Cloud Build. You can do this by adding the script ‘gcp-build’ to your package.json. All of this gets deployed to an App Engine standard environment using automatic scaling. So far with 4 Shopify stores using this integration, the entire platform stays under GCP’s daily usage quotas and costs nothing to run!

Next Steps

Obviously, if anyone else shows interest in this solution, the most immediate need is to make the UX better by rolling out a nice Shopify Admin UI, but there are a few other next steps that I’m currently working on.

  • There is still too much heavy lifting in the webhook handler. I’m working on just pulling enough out of the payload to know what ultimately needs to be done and then publishing this as a Shopify-agnostic event to Google Pub/Sub where it will ultimately be processed by a Google Cloud Function. This will allow me to let Shopify know that the webhook has been processed much more quickly and sets up the necessary infrastructure to start removing my other AWeber integrations in favor of directly publishing the necessary information as an event into this platform.
  • The event driven architecture opens up many additional possibilities… it allows me to do more analysis on the data before it gets into my email service provider. This allows me to better tag and identify the sources of this data. The AWeber Etsy integration for instance doesn’t provide any capability for tagging or otherwise identifying these subscribers. Eventually, this will be the place where I can plugin my ML project that I have been working on that correlates behavior and surfaces insights across sales channels
  • Turn this into a full blown marketing channel app for Shopify. I’ve always dreamt about a day where I can do my email marketing from Shopify the same way that I run Google Shopping or Facebook Ads campaigns. This platform provides the foundation for doing that. I’m really excited about the possibilities!
  • If enough interest exists, turn this into an actual product. Reach out if you’re a Shopify and AWeber customer that is already experiencing the original problem I solved, or have ideas for how this can become perfect for something that you’re trying to do.

Annoying Time Sink of the Month…

Every day there is an annoying thing that I MUST do.  It annoys me that I need to, but as part of running a professional business it must be done.  I discovered the issue about a month ago and have been doing this job manually every day since… all the while thinking about a solution to automating this task out of my life for good.

The Background

Threddies, being both an E-Commerce and Brick and Mortar business, has two distinct buyer’s journeys that start out in very different ways.  In one case, a prospective lead finds us online and is ultimately driven to our website (this journey becomes much more complicated when you consider that a user’s journey could start from Google, Amazon, Etsy, eBay, etc.).  In the other, someone stops by our shop in person… talks to us about the local happenings… and if we’re lucky, makes a purchase. Our end goal is to turn visitors into customers, and even better REPEAT customers. Most of the time, people do not become a customer on the first visit, so our goal is to make them a potential customer, by getting them to sign up for our email list, or at the very least, joining us on their preferred social media platform.  A lot has been written about this Omnichannel retail problem and it was something that we felt we had a fairly good handle on.

We recently have been consolidating our technology stack and introducing processes to make things consistent across all of the sales/leads ‘channels’ that we support.  One area of integration that occured (and ultimately became the root of my current problem) was our choice to move the brick and mortar store to use Shopify POS instead of Square’s offering.  This had many benefits: reduction in payment processing fees, combined CRM and inventory systems, a common user interface for our employees to use across all channels, etc.

The Vexing Problem

The problem that I didn’t see coming involves the collection of emails in our email marketing platform.  We use AWeber and AWeber has both a Square and Shopify integration. We were actively using the Shopify integration to collect emails from our website and the Square integration to collect emails from B&M purchases (We also use the Etsy integration, but the issues with that is a subject for another blog post).  In both cases, we use separate web based sign up forms or the AWeber Atom mobile app (appropriately tagged) to collect email addresses from those that don’t ultimately make a purchase. If you are a Shopify and AWeber user, you should be aware that AWeber’s integration does not support collecting emails that are entered using Shopify’s Newsletter functionality, this was something that I discovered early last year and have built an acceptable workaround for that (I can document this for anyone interested).

After removing Square from the store, I quickly noticed that no emails were being collected from B&M purchases.  Initially, I thought it was a sync issue and that they would eventually show up, but they never did. I did some digging and testing and it seems that the AWeber/Shopify integration was only collecting emails from customers who made a purchase from the website, no emails would ever be added from Shopify POS.  My daily chore of adding emails had begun… even worse, since I was constantly watching what was going on with email adds, I started to notice other issues.

We don’t require users to provide an email address in order to make a purchase, providing a valid phone number will allow you to make a purchase as well.  We do require an email address, however, in order to create a Threddies account which gives the customer access to some additional features that they wouldn’t get otherwise.  I noticed over time that there were many customers that made a purchase without an email and then ended up creating an account with an email at a later date. NONE OF THESE EMAILS WERE BEING CAPTURED!  This was a fairly large problem that required some explaining to our customers when I finally added them all manually to our email list. I did some additional testing and discovered that this same use case occurs when a customer updates their email address from their account.  AWeber never gets the updated email address. This was a particular issue since many of our customers originate from channels that obfuscate their real email addresses (Amazon, eBay, etc.), but then they ultimately warm to us and provide their real email address after making repeat purchases.

Solutions

I immediately started looking for solutions since manually doing this  every day was a nightmare. The AWeber provided integration is free with an AWeber account, but there are several paid solutions in the Shopify app store.  All of the third party integrations suffered from various issues… either they used polling on a regular interval to collect email addresses rather than being reactive to events occurring in the Shopify ecosystem, or they came with heavy handed tag syncing.  None of them specifically guaranteed the Shopify POS or Newsletter functionality that I desired. Since this problem was/is on my mind every day, I started thinking about the ideal state that I would like to have… tailored email marketing automation that is triggered by the channel that the user originated from.  This is important because the Amazon/eBay/Etsy’s of the world tend to be very restrictive regarding the content of the email that you can send to their users. Due to this, many of the emails sent to customers originating from these channels tends to be manually sent (often through systems that don’t have guaranteed high deliverability like AWeber), rather than automated which takes a crazy amount of time.  I also want intelligence behind tag syncing between Shopify and AWeber. It was clear that none of the existing integrations could meet these needs. This is the problem without an automated solution… for now… Stay tuned!

If you made it this far, and are interested in the thrilling conclusion… I wrote about my solution here.