Welcome
Welcome to Agnostic Data Labs!
The Agnostic Data Labs app provides a modern, agnostic, agile metadata management development tool for data solution automation.
It provides an interface and a workflow for working with metadata and templates.
These are then combined to generate output, such as code, documentation, or other artifacts.
The app provides:
- Metadata management and modeling
- Template management, design, and development
- Template expansion engine for generating output, based on the metadata and the templates
The end goal
Section titled “The end goal”The app allows you to define metadata and templates. These are stored locally in a repository folder and opened through the browser. Changes made to the metadata objects are saved to these local files.
Once changes are completed, the files are committed to the repository, following a normal development workflow, with code review and merging as needed.
The App can then generate the whole output set, or show individual outputs.
Once the outputs are generated, also to the local file system, they are ready to be deployed to the target environments, or the next step in the process, through a normal CI/CD deployment process.
The metadata
Section titled “The metadata”All metadata is open and available for editing, in the App, or in any other editor. The JSON format used is the open source Generation Metadata Schema for Data Warehouse Automation. As it is open and readily available, there is no risk of lock-in.
The templates
Section titled “The templates”Templates define what output should be created. They are written in the open-source Handlebars templating language. This is a simple and easy to learn language. The templates are stored in the repository, and can be edited in the App, or in any other editor. As the metadata format includes all the metadata all the time, the templates can operate on a single metadata set, meaning the templates are simple and easy to understand.
A template can generate any output, such as code, documentation, or other artifacts. The output can be any format, such as SQL, text, JSON, Markdown, or any other plain text format. Each template generates a single output file for each metadata object. So, a template that generates SQL for a table, will generate a single SQL file for each mapped metadata object.
The output
Section titled “The output”All output is generated to an output folder in the repository. This output can then be used in the next step of the process, such as deployment to a database, a cloud service, or to a documentation system. If a special deployment script is needed, use a template to generate it.
To automate a full output process, the metadata is mapped to the templates. So the right template is used for the right metadata object. This is done through a template mapping, stored in the metadata object.