Image Data pipeline

[Series] How to Build Data Pipelines in No Code? A 6-step guide with examples

By Brian Laleye · June 21, 2022 · 9 min read

Given the amount of data produced daily at around 2.5 quintillion bytes and an estimation of 80% of unstructured and semi-structured data collected by companies, data pipelines are becoming the backbone of any data-driven organization.

They’re responsible for collecting and storing data, transforming it into useful formats, and making it available to end-users.

Data pipelines are so important because they help organizations make sense of their data. 

They provide a way to store and process large amounts of information in a structured way. 

This enables companies to gain insights that can lead to better decision-making.

The goal of any good data pipeline is to automate repetitive tasks while still allowing you to change your (no) code 🙂 at any time if needed instead of asking developers/engineers to write code and manually interfacing with source databases.

What is a data pipeline?

A data pipeline is a system that ingests and moves data through a series of steps (like cleaning, transforming, filtering, aggregating and enriching data) to a given output.

Either for processing, storing, or analyzing the data in batch mode or in stream mode, data pipelines can be used for many purposes like adopting an operational analytics approach through reverse ETL or for ETL/ELT purposes for instance.

Let’s apprehend with this guide, the different steps needed to implement a data pipeline in no-code!

Step #1 - Data sources

Data sources are the places where data is stored.

This could be a database, a file, or even an API. Each data source will have its own schema and specific requirements for accessing it. For example, if you are using a SQL database as your source then you will need to know how to write SQL queries but fortunately you no longer need to write SQL with RestApp!

Let’s take an example of a pipeline we want to build here:

 

Once logged in RestApp, click on +Add Connectors

Step 1 Add connector

Then choose and fill out the credentials of the selected connector:

Step 1 Credentials

Then, you can decide on either choose this connector (MongoDB in our example) as an Input (to use this connector as a Source) and / or as an Output (to use this connector as a Destination):

Step 1 Choose as input or output

Same procedure, for GoogleDrive and Snowflake connectors:

Step 1 GoogleDrive
Step 1 Snowflake

We can now view all the connected data within RestApp’s Data Viewer, let’s go to Step #2.

Step #2 - Collection

Before processing data, browse your connected data for 3 main reasons:

  1. Identify the format of columns (text, number, date..)
  2. Identify the relationships between tables or at least the unique columns (primary key and foreign key)
  3. Identify cleansing to do (null values, columns to be dropped, type of calculations required…)

Go to Open my data to browse your connected data:

Step 2 Open my data

Thanks to RestApp, you’re able to access your connected data from any data source (database, data warehouse, API, files…):

Step 2 Snapshot with GoogleDrive
Step 2 Snapshot with MongoDB
Step 2 Snapshot with snowflake

We can now process all the connected data with RestApp’s editor with built-in SQL, NoSQL and Python functions.

Step #3 - Processing (Transformation)

Data processing is the process of extracting value from raw data, so you can transform it with methods such as filters, aggregation, normalization, cleansing and deduplication.

Use the no-code SQL, NoSQL and Python functions to model your data.

 

First of all, drag-and-drop the Input operation to retrieve the data from your connected sources: 

Step 3 Retrieve the data

Then, give a name to your data pipeline: 

Step 3 Functions

Now the fun part, we can start our end-to-end model, from Input to Output: 

Step 3 Start a model

Now, you can model your data coming from GoogleDrive, MongoDB and Snowflake with all the drag and drop SQL, NoSQL and Python built-in functions.

For instance, you can standardize the Date columns or clean all you dataset with the main SQL functions.

Step 3 Functions

You can preview the ongoing results at each step of transformation before sending them to any of your output:

Step 3 Preview

Step #4 - Destinations

The purpose of a data pipeline is to move data from one place to another — specifically, from source systems like databases and servers to destinations where analysis can be performed.

A destination is any system that accepts data from a source.

It could be an analytical database, a search engine or even a log file analyzer.

In our example, we took the following sources:

  • GoogleDrive to retrieve the revenue per customer
  • MongoDB to retrieve purchases analytics
  • Snowflake to retrieve all the orders (purchased, returned, canceled..)
 

Example

The purpose in our case is to clean and enrich data to identify the best customers based on business rules to offer to a selected set of customers specific promotions.

It requires to send those insights in Hubspot.

Just drag and drop the Output function and select the Hubspot connector, you can then decide on the syncing mode (Add data, Add & Update data or Erase & Replace data).

Step 4 Destinations

Another example of building a data pipeline could be to replicate data from one database to another one, see this guide to connect MongoDB to PostgreSQL to know more.

Now, we have built our end-to-end model called data pipeline, what if we want to automate it? 

Step #5 - Automation

As we know, automation is a process of integrating the steps that are repetitive, error-prone, and time consuming into a single step.

You can schedule your data pipeline by following these steps:

 

Go to Automation and click on +Automate:

Step 5 Automation

Then, select the pipeline to automate: 

Step 5 Select the pipeline

Now, we’ve automated the pipeline, we want to monitor it within the platform to be sure everything runs smoothly.

Step #6 - Monitoring

Logs are a record of events that occur in a system, in data pipeline for our case, they are used for debugging, troubleshooting and reporting purposes. 

Go to Automation app and click on Logs button:

Step 6 Logs

Once there, you can see all the jobs/syncings done for this data pipeline:

Step 6 Jobs

Bonus: Governance

The main benefit of data pipelines is the ability to scale easily.

You can start with a small pipeline and then add more processes as your needs grow.

Thus, thanks to a low/no-code approach, you’re able to test, iterate and bring more and more value to your data pipeline with easy drag and drop built-in functions.

As data grows, pipelines grow also, so you need to have a clear overview on all your data pipelines by defining rights and permissions to specific users, scope of work and purposes.

That’s why the Domain app in RestApp comes handy to share and work securely with stakeholders (teammates, partners, clients, providers..):

Go to Domains and click on +Create domain:

Create domains

Then, give a name to your domain and add some comments or description to share its purpose:

Domain name

Now, just add people to share securely pipelines and connectors with:

Add people

Now, add pipelines to this specific domain:

Add pipelines to a domain

Now, add connectors to this specific domain:

Add connectors to a domain

Thanks to a no-code SaaS Data pipelines, organizations no longer require to write code and build data pipelines from scratch, your Data & Ops teams don’t need to rely on the Tech team to get, process and analyze data.

If you’re interested in starting with connecting all your favorite tools, check out the RestApp website or book a demo.

Share
Share on linkedin
Share on twitter
Share on facebook

Subscribe to our newsletter

Brian Laleye
Brian Laleye
Brian is the co-founder of RestApp. He’s a technology evangelist and passionate about innovation. He has an extensive experience focusing on modern data stack.
Share this article
Share on linkedin
Share on twitter
Share on facebook
Subscribe to our newsletter
Ready to experience data activation
without code?

Product

Activate and combine any data sources without code

Transform your data with our No Code SQL editor

Automate your operational analytics with your business apps

Share your pipelines and collaborate smarter with your teammates

Discover how Data Transformation really means

Find out the new data architecture concept of Data Mesh

Learn how Operational Analytics actives your data

Learn how to deliver great customer experience with real-time data

Solutions

Empower your teammates

Prevent churn

Generate more revenue

Run better campaigns

Drive your metrics

Resources

Stay always up to date on data activation

Get access to tips and tricks to model your data

Discover our always evolving and regularly updated documentation

Find out how we keep your data safe