[Series] How to Build Data Pipelines in No Code? A 6-step guide with examples
By Brian Laleye · June 21, 2022 · 9 min read
Given the amount of data produced daily at around 2.5 quintillion bytes and an estimation of 80% of unstructured and semi-structured data collected by companies, data pipelines are becoming the backbone of any data-driven organization.
They’re responsible for collecting and storing data, transforming it into useful formats, and making it available to end-users.
Data pipelines are so important because they help organizations make sense of their data.
They provide a way to store and process large amounts of information in a structured way.
This enables companies to gain insights that can lead to better decision-making.
The goal of any good data pipeline is to automate repetitive tasks while still allowing you to change your (no) code 🙂 at any time if needed instead of asking developers/engineers to write code and manually interfacing with source databases.
What is a data pipeline?
A data pipeline is a system that ingests and moves data through a series of steps (like cleaning, transforming, filtering, aggregating and enriching data) to a given output.
Either for processing, storing, or analyzing the data in batch mode or in stream mode, data pipelines can be used for many purposes like adopting an operational analytics approach through reverse ETL or for ETL/ELT purposes for instance.
Let’s apprehend with this guide, the different steps needed to implement a data pipeline in no-code!
Summary
Step #1 - Data sources
Data sources are the places where data is stored.
This could be a database, a file, or even an API. Each data source will have its own schema and specific requirements for accessing it. For example, if you are using a SQL database as your source then you will need to know how to write SQL queries but fortunately you no longer need to write SQL with RestApp!
Let’s take an example of a pipeline we want to build here:
- Input: MongoDB and GoogleDrive
- Output: Snowflake
Once logged in RestApp, click on +Add Connectors:
Then choose and fill out the credentials of the selected connector:
Same procedure, for GoogleDrive and Snowflake connectors:
- GoogleDrive connector:
- Snowflake connector:
We can now view all the connected data within RestApp’s Data Viewer, let’s go to Step #2.
Step #2 - Collection
Before processing data, browse your connected data for 3 main reasons:
- Identify the format of columns (text, number, date..)
- Identify the relationships between tables or at least the unique columns (primary key and foreign key)
- Identify cleansing to do (null values, columns to be dropped, type of calculations required…)
Go to Open my data to browse your connected data:
Thanks to RestApp, you’re able to access your connected data from any data source (database, data warehouse, API, files…):
- Snapshot with GoogleDrive:
- Snapshot with MongoDB:
- Snapshot with Snowflake:
We can now process all the connected data with RestApp’s editor with built-in SQL, NoSQL and Python functions.
Step #3 - Processing (Transformation)
Data processing is the process of extracting value from raw data, so you can transform it with methods such as filters, aggregation, normalization, cleansing and deduplication.
Use the no-code SQL, NoSQL and Python functions to model your data.
First of all, drag-and-drop the Input operation to retrieve the data from your connected sources:
Then, give a name to your data pipeline:
Now the fun part, we can start our end-to-end model, from Input to Output:
Now, you can model your data coming from GoogleDrive, MongoDB and Snowflake with all the drag and drop SQL, NoSQL and Python built-in functions.
For instance, you can standardize the Date columns or clean all you dataset with the main SQL functions.
You can preview the ongoing results at each step of transformation before sending them to any of your output:
Step #4 - Destinations
The purpose of a data pipeline is to move data from one place to another — specifically, from source systems like databases and servers to destinations where analysis can be performed.
A destination is any system that accepts data from a source.
It could be an analytical database, a search engine or even a log file analyzer.
In our example, we took the following sources:
- GoogleDrive to retrieve the revenue per customer
- MongoDB to retrieve purchases analytics
- Snowflake to retrieve all the orders (purchased, returned, canceled..)
Example
The purpose in our case is to clean and enrich data to identify the best customers based on business rules to offer to a selected set of customers specific promotions.
It requires to send those insights in Hubspot.
Just drag and drop the Output function and select the Hubspot connector, you can then decide on the syncing mode (Add data, Add & Update data or Erase & Replace data).
Another example of building a data pipeline could be to replicate data from one database to another one, see this guide to connect MongoDB to PostgreSQL to know more.
Now, we have built our end-to-end model called data pipeline, what if we want to automate it?
Step #5 - Automation
As we know, automation is a process of integrating the steps that are repetitive, error-prone, and time consuming into a single step.
You can schedule your data pipeline by following these steps:
Go to Automation and click on +Automate:
Then, select the pipeline to automate:
Now, we’ve automated the pipeline, we want to monitor it within the platform to be sure everything runs smoothly.
Step #6 - Monitoring
Logs are a record of events that occur in a system, in data pipeline for our case, they are used for debugging, troubleshooting and reporting purposes.
Go to Automation app and click on Logs button:
Once there, you can see all the jobs/syncings done for this data pipeline:
Bonus: Governance
The main benefit of data pipelines is the ability to scale easily.
You can start with a small pipeline and then add more processes as your needs grow.
Thus, thanks to a low/no-code approach, you’re able to test, iterate and bring more and more value to your data pipeline with easy drag and drop built-in functions.
As data grows, pipelines grow also, so you need to have a clear overview on all your data pipelines by defining rights and permissions to specific users, scope of work and purposes.
That’s why the Domain app in RestApp comes handy to share and work securely with stakeholders (teammates, partners, clients, providers..):
Go to Domains and click on +Create domain:
Then, give a name to your domain and add some comments or description to share its purpose:
Now, just add people to share securely pipelines and connectors with:
Now, add pipelines to this specific domain:
Now, add connectors to this specific domain:
Thanks to a no-code SaaS Data pipelines, organizations no longer require to write code and build data pipelines from scratch, your Data & Ops teams don’t need to rely on the Tech team to get, process and analyze data.
If you’re interested in starting with connecting all your favorite tools, check out the RestApp website or book a demo.
Subscribe to our newsletter
Additional resources:
- Top 10 SQL functions to clean your data in No Code. A Complete Guide with Examples
- [Series] Business Ops – 5 SQL Date Functions in No Code to get started
- How to Automate Lead Scoring from PostgreSQL to Hubspot?
- [Series] Revenue Ops – How to calculate Net MRR from MongoDB to GoogleSheet? A full step-by-step guide with template
