#002: Using Zapier for Real-Time Data (Without Code)

Imagine... you have a client with a lead-generating website.

They tell you they want all form submissions logged in a database.

More importantly, because time is money, they also want to be notified immediately whenever one happens.

They don't have the budget for a complex architecture nor the tech skills to maintain it long-term.

They bring you in and tell you to work your magic to help them. What do you do?

You could brush them off and tell them real-time data is beyond of their capabilities. 

You advise them to stick to batch scheduling and be happy with hourly updates.

Or....

You could propose what's known as a "no-code" automation tool.

Not only will it allow you to capture event (near real-time) data, but will keep the complex integration components out of view.

One in particular I really like is called Zapier. But there are a handful of others that do the same thing.

Tools like Zapier connect easily with many commonly used business applications and allow you to create custom, event-driven workflows through a nice UI.

Sounds pretty close to real-time data to me.

(note: this is not sponsored in any way - it's just a tool I think is helpful and personally use)

To appreciate how this works without code, we need to understand how it can access all these tools in the first place:

1. Most software products are cloud based.
2. Cloud based products can be accessed through APIs.
3. Each API has a standard format.
4. APIs can share event-level information
5. Events from one API can be used to "trigger" other API events

Zapier takes the API part out of view so you can focus on the triggers & actions.

This may be oversimplified - but you get the idea.

Back to your client - how does it get implemented?

In this example they use Webflow (website) as the initial trigger, Airtable (database) to store the records and Slack (for notifications).

Once connections to each application are established, a workflow might look like this:

Zapier handles all of the behind the scenes API connectivity and all you need to do is select the different fields to use.

The moment a new form is submitted on Webflow, Zapier will capture the info from the API feed and then send it off to the Airtable API feed (instantly) to log a new record.

Once that completes, Zapier then posts a message to the Slack Channel of your choice.

With this in place, you've provided a solution to both log records and provide notifications in real time.

Furthermore, your client won't need to worry about managing the complex API logic or updating it something changes with Slack (or any other tool). 

Now of course this won't be able to completely replace an enterprise ETL/ELT process.

But it does have the potential to be taken much further than the simple example provided.

It can be a nice option to help fill gaps where more real-time data is requested but creating a full data-streaming platform is out of the question.

Think of it as another tool in your data engineer toolbox.

Worst case,  it's a fun way to automate some of the data flows in your personal life or on a side project.

Cheers,

Mike