In Data Engineer’s Lunch #49: Meltano for Data Engineering, we discussed how to get started with Meltano and ELT The live recording of the Data Engineer’s Lunch, which includes a more in-depth discussion, is also embedded below in case you were not able to attend live. If you would like to attend a Data Engineer’s Lunch live, it is hosted every Monday at noon EST. Register here now!
Meltano for Data Engineering
Meltano is an open-source ELT platform that integrates multiple other open-source tools to leverage for each step of the data lifecycle. Meltano is self-hosted and was initially created by GitLab’s data team. ELT refers to the practice of Extracting, Loading, and Transforming data. It is Meltano’s mission to “Enable everyone to realize the full potential of their data.” Thus the motivation in order to have it open-sourced and free to use.
Getting started with Meltano for Data Engineering
Meltano offers a quick start guide in order to get up and running with the technology. This includes initializing a Meltano project. Integrating with your SaaS tool or database using Singer taps and targets. Transforming the data with a dbt project. Orchestration with Airflow built-in. It can be containerized via docker and includes a web UI to check the logs of your pipelines with options to configure extractors, loaders, and pipelines.
For a more in-depth explanation of Meltano and to see it run, check out the embedded YouTube video below. In addition to this, we show what Meltano’s dashboard looks like. Additionally, all of our live events can be rewatched on our YouTube channel, so be sure to subscribe and turn on your notifications!
Resources:
Cassandra.Link
Cassandra.Link is a knowledge base that we created for all things Apache Cassandra. Our goal with Cassandra.Link was to not only fill the gap of Planet Cassandra but to bring the Cassandra community together. Feel free to reach out if you wish to collaborate with us on this project in any capacity.
We are a technology company that specializes in building business platforms. If you have any questions about the tools discussed in this post or about any of our services, feel free to send us an email!
