Tag: Google Cloud Platform

Shopify Webhooks driving AWeber

This post is a solution to a problem I had with the AWeber Shopify integration. To get the most out of this post, check out the original problem here.

…the continuation…

Being that I was spending 30+ minutes every day manually solving this problem, it was important that I had a MVP solution quickly. I took a step back to think about my immediate needs and the future direction that I would like to take this solution and came up with the following constraints:

  • Need to get something basic up and running quickly that can be easily iterated upon
  • Everything needs to be deployable to the Google Cloud Platform (and not cost a fortune to run)
  • The solution should be something that I can eventually monetize. This means a clean, UI based integration in the Shopify ecosystem (i.e. support for Node, React, Next) and the need to be able to handle many Shopify stores and scale appropriately.
  • Anything built must be easy to fit into the multichannel lead generation vision of Threddies. Eventually this would need to become the way that all leads get added to my email service provider without using any direct integrations.

Shopify Webhooks

I did a little digging and realized that I could solve just about every variant of the core problem if I were notified any time a customer was created or updated in Shopify. Conveniently enough, Shopify provides webhooks for both of these cases (in addition to many more). Webhooks are great for creating quick integrations and very easy to handle using Google Cloud Functions.

I prototyped the ‘create customer’ webhook and had something up and running in no time for my test store. I also started to think more about how I can quickly iterate on webhook based integrations in the future. The most simplistic integration using webhooks doesn’t require authentication, but it does require verifying the data sent is actually from the expected Shopify store and not just anyone on the internet. This is done using the X-Shopify-Hmac-Sha256 header. When you receive the webhook data, you need to verify the data in the body by generating this value (using a private key) and comparing it with what Shopify sends. There are two different ways to do this and it depends on how you integrate with Shopify. The preferred approach is to develop a Shopify app which has it’s own key that you can use to verify every authorized store that is using your app. The drawback of this approach is that you need a full blown Shopify app that implements the store authorization flow and requires some UI work. Since I’m not a React expert, I opted to take the second approach and avoid the UI by having each Shopify store owner that was going to be using this integration provide their store’s key to me. You can get this key by going to your store’s settings page and registering a webhook in the ‘Notifications’ area. This key is used to verify the integrity of all webhook data sent. Things I learned from this step:

  • Cloud Functions would likely not be the final way of deploying this since it did not provide a way to surface a UI in a customer’s Shopify store.
  • The Cloud Function for each webhook is going to have a lot of repeated boilerplate for verifying the integrity of the data sent and handling responses/errors. I would also need a more centralized storage location for keeping Shopify Store specific data so that it wouldn’t need to be duplicated in every Cloud Endpoint
  • Shopify webhooks require a timely response, so putting any heavy lifting in the cloud function is not going to happen. Take too long to respond, and Shopify will deregister your interest in the webhook data. This started to get me thinking about how to recover from this scenario.

Hookup to AWeber

I was getting data from Shopify and verifying the integrity of the data, but at this point nothing was happening with it. In order to get the data into AWeber, I had to again obtain some information from each admin of the Shopify stores that I was integrating with. At the bare minimum, I needed an AWeber account id and a list id to add all subscribers to. I also needed my customer to authorize my app to interact with their AWeber account. More requirements for UI, but I still wanted to put that off and focus on solving the original problem. I also didn’t want to force my users to go into AWeber in order to add my integration. I found this great NodeJs wrapper library around AWeber’s API that did everything I wanted it to do. Using this, you can enter your AWeber integration information and use it to generate an auth URL that you can send to your customer. They can then use that URL to provide your integration with the necessary permissions to their account. They send back a verifier code after authorization that you can use to get all of the necessary tokens needed for your integration to access that account. This information doesn’t change unless the user removes your integration, so it works perfectly until I actually setup the full blown authorization path in my Shopify App.

There were two AWeber issues that I discovered at this point: there is no way to add the subscriber’s location without using an originating ip address for geolocation. This means the ability to send using the subscribers local time window is not something that will work with subscribers added this way. Also, AWeber integrations are confirmed opt-in by default; AWeber would not turn this off for everyone using my integration, so I had to turn it off for each account/list I wanted to use it with and tell my customers to do the same. This was necessary since my integration already leverages Shopify’s confirmed opt in and I didn’t want to confuse my subscribers by making them do it again. The Shopify webhook payload already includes a flag for ‘accepts_marketing’ and I verify that this flag is true before attempting to add any information to AWeber. Things learned at this point:

  • I really need that UI!
  • The create customer/update customer flows look very similar from a Cloud Function perspective, so there needs to be some consolidation.
  • The AWeber deauth path needs to be handled. This can’t be done in a cloud function since it’s so far removed from the user capable of fixing this problem. For now, just alert on the error when it occurs and add it to the list of issues to handle later. This is another item that indicates the need for a systems health check in the customer’s Shopify store (and a way to recover all subscribers added between the time when a failure occurs and the Shopify store owner resolves the problem)

Ready for Production

At this point, things were working well enough that I felt confident allowing this integration to start doing my job for me. Before moving everything to production, I refactored everything to eliminate the obvious problems that I saw at this point.

Instead of Cloud Functions being the primary entry point, I created a NodeJs app to do this instead. This allowed me to setup all of the webhook routing inside this app and put all of the webhook verification and ‘health check’ code into this app. If there was a problem, I could fail fast without fear of Shopify deregistering the webhook.

This also provided a place where I could add all of the UI code for the integration and the intelligence for recovering from failures. This app can also morph into a frontend that is capable of handling and routing any future webhook integration that I want to create.

I then deployed the NodeJS app to Google App Engine. At first it wasn’t working and the errors indicated that the Next.js build step wasn’t occurring on deployment. I solved this by adding a custom build step that is automatically run on deployment by Google Cloud Build. You can do this by adding the script ‘gcp-build’ to your package.json. All of this gets deployed to an App Engine standard environment using automatic scaling. So far with 4 Shopify stores using this integration, the entire platform stays under GCP’s daily usage quotas and costs nothing to run!

Next Steps

Obviously, if anyone else shows interest in this solution, the most immediate need is to make the UX better by rolling out a nice Shopify Admin UI, but there are a few other next steps that I’m currently working on.

  • There is still too much heavy lifting in the webhook handler. I’m working on just pulling enough out of the payload to know what ultimately needs to be done and then publishing this as a Shopify-agnostic event to Google Pub/Sub where it will ultimately be processed by a Google Cloud Function. This will allow me to let Shopify know that the webhook has been processed much more quickly and sets up the necessary infrastructure to start removing my other AWeber integrations in favor of directly publishing the necessary information as an event into this platform.
  • The event driven architecture opens up many additional possibilities… it allows me to do more analysis on the data before it gets into my email service provider. This allows me to better tag and identify the sources of this data. The AWeber Etsy integration for instance doesn’t provide any capability for tagging or otherwise identifying these subscribers. Eventually, this will be the place where I can plugin my ML project that I have been working on that correlates behavior and surfaces insights across sales channels
  • Turn this into a full blown marketing channel app for Shopify. I’ve always dreamt about a day where I can do my email marketing from Shopify the same way that I run Google Shopping or Facebook Ads campaigns. This platform provides the foundation for doing that. I’m really excited about the possibilities!
  • If enough interest exists, turn this into an actual product. Reach out if you’re a Shopify and AWeber customer that is already experiencing the original problem I solved, or have ideas for how this can become perfect for something that you’re trying to do.

Ghost in the (Google Cloud) Shell

One of the perks of being a technologist that is not tied to a traditional 9 to 5 is that you have immense freedom in terms of where you can complete your ‘work’ from. I’ve always toyed with the digital nomad lifestyle… but it’s kind of ridiculous when you need to lug around an insane amount of equipment in order to effectively complete your tasks. I have several computers; optimized for specific types of work, or tied to specific clients. This always required me to think ahead before traveling about which project I was going to work on while away (there was a time where I would take everything with me, but traveling with kids has definitely made me want to pack as minimally as possible).

Development and image processing requires horsepower and even the best laptops for doing this are big and heavy…. and expensive… so much so that it’s something to think about if you travel to a country like India, Russia or China where in today’s political climate the likelihood of your hardware getting confiscated is higher than ever. Having this happen while traveling was what ultimately drove me to become a Chromebook advocate. Losing the hardware is one thing, but losing the data contained on the device is even worse. Chromebooks solved the data problem… You could powerwash the device and then restore it back to its former state at any point in time from the cloud. Worst case scenario, you lose a reasonably inexpensive piece of hardware, but your data is intact. Unfortunately, Chrome OS hardware hasn’t historically been the best option for development… especially if you want to maintain the security offered by the powerwash technique that I mentioned.

The desire to be able to travel anywhere, any time at a moments notice and feel confident that I can deal with anything that comes up while I’m gone using just my Chromebook ultimately drove me to experiment with setting up containerized development environments. I wanted something divorced from the hardware that I could easily get up and running and know that everything is setup the way I need it to be. This was great, but I still needed someplace where I could access these containers from anywhere. I eventually became more and more a fan of the Google Cloud Platform (GCP)… the container centric approach to everything and the fact that the price was right ultimately led me to migrate all of my cloud infrastructure to GCP. It wasn’t long before my containerized development environments followed… and then I discovered Google Cloud Shell.

Google Cloud Shell takes this whole idea a step further. It gives me a 5GB persistent space accessible from any browser. I don’t even need the Chromebook any more. Everything that I store in my home directory stays there across sessions. Even better, it’s directly connected to all of my projects in GCP. I’ve been doing almost all of my recent development using Google Cloud Shell and the integrated Orion Editor exclusively… and I LOVE it! For web based development and microservices, it’s absolutely great. Especially if you’re ultimately deploying to GCP. The only time I’ve gone back to my ‘development’ laptop has been to do Android development as I haven’t really found a good solution for running things like Android Studio or emulators using this approach.

But I want to develop for ‘free’

Ok, I can hear a bunch of you thinking that you don’t want to be forced to develop on GCP (and potentially incur costs) before you’re ready to deploy to production. Guess what? ngrok works great in Google Cloud Shell… you can expose your local dev environment securely anywhere on the web without deploying your project to GCP. What about localhost? ngrok exposes debug information on 127.0.0.1, so there’s no way to access that from Google Cloud Shell, right? Wrong… with GCS, you can expose a ‘ ‘web preview’ from any port just by clicking on the icon within GCS, you can map this to expose ngrok’s debug interface.

Onward to Production

Google Cloud Shell obviously has all of the Google Cloud SDK integrated by default, so when you’re ready to go to production, it’s a piece of cake. GCS even knows which Cloud Project you’re working on (and reminds you of that fact in the terminal). Turn off ngrok, push to your cloud environment and update your systems to point to the production version!

Conclusion

Is Google Cloud Shell the absolutely flawless solution to every development need that a digital nomad has? Definitely not, but it’s pretty damn good. I haven’t found a good way to do Android development this using it. It’s absolutely fantastic for doing Node development though…. especially if you’re deploying to GCP ultimately. Google Cloud Shell does have a usage limit of 60 hours per week, so if you’re burning the candle from both ends, you’ll want to remember to shut it down when you do take a break so that you don’t hit that limit. Give it a shot for yourself and let me know what you think.