In Data Engineer’s Lunch #37: Pipedream: Serverless Integration and Compute Platform, we will discuss Pipedream, a serverless integration and compute platform that is free for individual developers to use. The live recording of the Data Engineer’s Lunch, which includes a more in-depth discussion, is also embedded below in case you were not able to attend live. If you would like to attend a Data Engineer’s Lunch live, it is hosted every Monday at noon EST. Register here now!
What is Pipedream?
Pipedream Website: https://pipedream.com/
Pipedream is a “production-scale serverless platform to connect APIs, remarkably fast”. It gives users the ability to perform the following tasks:
- Connect OAuth and key-based API accounts easily
- Use those connected accounts in no-code or Node.js building blocks
- Build and run workflows which can be triggered on various events occuring
- I.E. HTTP Requests, on a schedule, application events, and more.
Pipedream makes it easy to connect APIs with code-level control when a user needs it, and also no-code when a user does not through the hundreds of prebuilt code blocks. Currently, developers with Node.JS or Javascript will get the most out of Pipedream. However, support for more languages including Python, TypeScript, and GitHub integration is coming in the near future. Additionally, getting started is as easy as making an account, and attaching a payment method from the get-go is not necessary.
Use Cases of Pipedream
As stated on Pipedream’s website, “pipedream supports use cases from prototype to production and is trusted by 150k+ developers from startups to Fortune 500 companies”. Some of these companies include the following:
Pipedream’s platform is capable of processing billions of events and is “priced for use at scale”. A sample list of some of the wide variety of use cases for Pipedream includes:
- Connecting SaaS (Software as a Service) applications
- General API orchestration and automation
- Prototyping and demos
- Mobile and JAMstack backends
- And many more…
Additionally, Pipedream’s website contains a community forum where users can post questions and ask about / contribute to upcoming features. The community forum can be found here: https://pipedream.com/community/
Workflows and Steps
In Pipedream, the primary entities that a user builds are called workflows. A workflow is:
- Composed of a set of linear blocks of code, each of which is known as a step
- Step: Building blocks that create a workflow
- Steps include triggers, code blocks, and pre-built action blocks
- Workflows start with a trigger:
- A trigger is an initial step that regulates how/when a workflow gets triggered to run
- Triggers can be any of the following:
- Scheduled (minutes or hours) – the workflow runs on a schedule
- HTTP requests – I.E. Pipedream can generate a URL that makes a workflow start whenever that URL gets an HTTP request
- And more…
- Steps are executed in the order that they appear in the workflow.
- See the section at end of the blog for a couple of examples.
- Values can be exported in each step and can then later be accessed via the steps object in future steps
- Additionally, the $checkpoint object can be used to store values from a particular workflow to future runs of that same workflow.
- Workflow code, by default, is private
- However, the code and layout of any particular workflow can be made public. Execution details and data from your particular workflow will still remain private when this is done.
Some example trigger steps are shown in the image below:
Invocations, Compute Time and Pricing
In Pipedream, two primary metrics are kept track of to determine how much a particular user gets charged. The first of these two metrics is known as the number of Invocations. Invocations are:
- “Pipedream counts an invocation each time a workflow or event source is triggered by an incoming event.”
- “Pipedream increments the count of invocations by one for each incoming event, regardless of the number of steps in your workflow.”
- I.E. An HTTP request triggered workflow with five steps will only count as one invocation per HTTP request trigger
The second metric that is kept track of by Pipedream for the purpose of charging a user is Compute Time. Compute time is calculated as the total time a particular user’s workflow/event source runs user code and various prebuilt steps. There is a minimum of 100ms block per execution of a workflow, therefore if a process takes <100ms then it will count as a 100ms compute time block.
Depending on the type of user (free vs paid), these two metrics may limit a user in how much they can use the service unless they choose to pay more. The primary two plans for a single user, along with their compute time and invocation limits, are as follows:
- Developer (Free) plan:
- 333 daily invocations
- 1,800,000 ms daily compute time
- Pro (Paid) plan ($19/month):
- 20,000 invocations per month, with $0.0002/extra invocation
- Infinite compute time
So for example: even if you only use very few invocations daily, but each invocation takes up a very large amount of compute time, it is possible that you will be limited by compute time on a free plan.
More information about various plans Pipedream has for both individual users and organizations can be found here: https://pipedream.com/pricing
Open Source component of Pipedream
As stated on Pipedream’s website, “Pipedream maintains an open-source component registry on GitHub”. This GitHub repository can be found here: https://github.com/pipedreamhq/pipedream/
Using these prebuilt, open-source components allows users to avoid writing boilerplate code for common API integrations. Note that components can be used as no-code blocks in workflows, or they can be customized with your own code and then used. Here is a brief list of some existing popular open-source components/integrations in Pipedream:
- Discord Bot
- Slack
- AWS
- Google Drive/Sheets/Calendar/Cloud/Docs
- HTTP
- Spotify
- And more….
For contributing to this repository, Pipedream’s documentation has a page which describes how to both develop for Pipedream and how to properly contribute to the open-source component repository by submitting a pull request: https://pipedream.com/docs/components/guidelines/#overview
A Workflow: Example
For this example, we will go through a two-step workflow and then bring up an example from the Quickstart guide showcasing a significantly more complex workflow. For our first workflow, we created a workflow that looks like this:
From the above image, we can see that the workflow is composed of two steps:
- The first is a Twitter API trigger which is scheduled to run every x minutes and perform certain Twitter API actions.
- The second step is a pre-built email_me step in Pipedream which sends an email to the user email associated with the Pipedream account
A Twitter account was attached in Pipedream and now Twitter API blocks can be used with that particular account. This particular trigger step is set to run on a schedule (every few minutes) and if any new likes on tweets are done by the user, then the workflow will trigger. The second step of the workflow takes these newly liked tweets and sends the text of those tweets to the appropriate email.
For a more complicated example, you can follow Pipedream’s quick start guide, part of their documentation, linked here: https://pipedream.com/docs/quickstart/
The image below shows a sample workflow we built following the Quickstart guide, with some modifications:
For a complete rundown on this 8 step workflow, check out the Youtube video at the bottom of this blog.
And that’s a wrap on today’s blog on Pipedream: Serverless Integration and Compute Platform. For more content similar to this week’s DEL on Pipedream: Serverless Integration and Compute Platform, check out https://blog.anant.us.
References
- https://pipedream.com/
- https://github.com/PipedreamHQ/pipedream
- https://pipedream.com/docs
- https://pipedream.com/workflows
- https://pipedream.com/docs/quickstart/
Cassandra.Link
Cassandra.Link is a knowledge base that we created for all things Apache Cassandra. Our goal with Cassandra.Link was to not only fill the gap of Planet Cassandra but to bring the Cassandra community together. Feel free to reach out if you wish to collaborate with us on this project in any capacity.
We are a technology company that specializes in building business platforms. If you have any questions about the tools discussed in this post or about any of our services, feel free to send us an email!
