User Interface
Home Page Overview
After logging in successfully, you are directed to the Data Nimbus Designer (DND) home page. This page provides easy access to all the features and tools of the DND, offering a centralized location for creating and managing designs.
Blocks Panel: The Blocks Panel provides an organized view of the key components in the ETL pipeline, allowing users to visualize, configure, and manage different pipeline blocks effectively.
Design Library: The Design Library allows you to access, manage, and track ETL pipelines that have already been designed. It serves as a centralized repository for all user-created pipelines, enabling easy retrieval, modification, and management of their data workflows.
Search Bar: The search feature allows you to find ETL pipelines by name or assigned tags, helping quickly locate relevant pipelines based on user input.
Add New Designs: The Add New Design button allows you to create new ETL pipelines or designs using an intuitive drag-and-drop interface. This feature simplifies the process of building pipelines by enabling users to easily organize and configure components without writing code.
Design
In DataNimbus Designer, a design refers to the arrangement of ETL blocks or workflows in a specific sequence to achieve the desired output. To learn more about how designs are created and visualize the process, please refer to this video Link.
Connectors
A Connector in DataNimbus Designer refers to the essential connection details used to link data sources and destinations. It includes information like connection strings, authentication credentials, and configuration settings required to connect with various systems, including databases, cloud storage, and APIs. This metadata enables seamless integration between the source and destination.
Steps to add Connectors
Click on the Connectors tab in the Block Panel. Click on Add New Connector.
You will be directed to the New Connector Configuration Details page. Select the Connector Name and Connector Type.
Click Save to finalize. Your connector is now ready to use.
Configurations
The configuration process plays a crucial role in setting up the workspace environment. It involves configuring the Databricks catalog, defining the workspace path, and establishing schemas. These configurations ensure that the workspace is properly structured and optimized for data management and analysis, allowing seamless integration and accessibility across various components.
To configure a workspace environment:
Click on the Configurations tab.
Enter the configuration details and click Save.
Blocks
In the context of ETL (Extract, Transform, Load), "blocks" refer to individual, modular components within the data processing pipeline that perform specific tasks like connecting with the data source and sink, data extraction from a source, data cleansing, transformation operations, and loading data into a target system.
Extractor The Extractor is responsible for extracting data from the source systems. It connects to data sources (e.g., databases, file systems, APIs) and retrieves the raw data needed for further processing. The extractor pulls data in various formats such as JSON, CSV, Parquet, or SQL, preparing it for transformation.
Delta Extractor: Loads data from Delta Lake tables into the pipeline
Cloud File Extractor: CSV, Parquet, JSON
API Extractor: Fetches data from external APIs using provided connection details and API endpoints
Database Extractor: Retrieves data from databases like MySQL, PostgreSQL, MongoDB, etc., using the stored connection information
Transformer The Transformer blocks processes and transforms the extracted data to meet business requirements. This stage includes tasks like data cleaning, filtering, aggregation, and applying business logic. Transformations could involve changing data types, joining multiple datasets, or performing calculations to prepare the data for loading into the destination.
Loader The Loader block loads the transformed data into the target destination. This could be a database, data warehouse, data lake, or cloud storage. The loader ensures that the data is inserted, updated, or overwritten based on the requirements of the target system, completing the ETL cycle.
API Loader: Sends transformed data to external APIs.
Database Loader: Inserts or updates data in databases like MySQL, PostgreSQL, MongoDB, etc.
Delta Loader: Writes data back into Delta Lake tables.
Custom Blocks Custom blocks in DataNimbus Designer allow users to implement their logic by incorporating source files. These blocks offer flexibility, enabling users to define custom transformations or processing steps according to specific requirements. Custom blocks can be easily integrated into designs, allowing users to extend the default functionality of the platform and create tailored ETL workflows. Two major things are required to write and publish any custom code:
The Code itself - Currently we only support the Python language for custom code.
Config.json file - This file is required to specify the metadata about the custom code, like its name, input required, and block type.
Steps to Create Custom Blocks
Click on the Import Custom Block button to open the import dialog.
A new pop-up window will appear. Paste the path of the custom block in the provided text box. Click on the Import Block button to complete the import process.
The Custom Block will appear in your workspace if the information is correct.
Metadata The Metadata block manages the setup, validation, and cleanup of data and pipeline configurations, tracking essential information about the pipeline's flow and state. Start Block: Configures metadata for tracking data sources, tags, transformations, and destinations. The Start Block will always be added when creating new designs. Steps to add tags
Click on the Start Block.
A new pop-up window will appear. Click on Add Tags.
Enter the Key and Value fields. Click Save.
Workflow Design
The demo workflow design below showcases how easily workflows can be created using DataNimbus Designer.
Last updated