In Data Engineer’s Lunch #23, Rahul Singh covers the topics of Thanos, Cortex, and a recap of last week’s Prometheus. The live recording of the Data Engineer’s Lunch, which includes a more in-depth discussion, is also embedded below in case you were not able to attend live. If you would like to attend a Data Engineer’s Lunch live, it is hosted every Monday at noon EST. Register here now!
Cortex
Cortex is a CNCF incubation project used in several production systems including Weave Cloud and Grafana Cloud. The primary use for Cortex is as a remote write destination for Prometheus, exposing a Prometheus-compatible query API. It provides horizontally scalable, highly available, multi-tenant, long-term storage for Prometheus. The way Cortex solves the problem with Global View is with centralized data. For high availability, Cortex can replicate data between machines. Finally, for long-term storage, it supports Amazon DynamoDB, Google Bigtable, Cassandra, S3, and GCS. So this allows to durably store data for longer than the lifetime of any single machine. In the presentation, Rahul explains the difference between the two sets of data Index/Chunk.
Cortex architecture
Thanos
Thanos provides a global query view, high availability, data backup with historical, cheap data access as its core features in a single binary. The way Thanos approaches the global view is with federating data, so it allows you to query and aggregate data from multiple Prometheus servers. When Thanos reads the data it dedupes it at the query time and saves it into long-term storage.
Thanos Architecture
As mentioned above, if you are looking for a more in-depth discussion, as well as a demo, be sure to check out the live recording of Data Engineer’s Lunch #23: Thanos/Cortex.
Cassandra.Link
Cassandra.Link is a knowledge base that we created for all things Apache Cassandra. Our goal with Cassandra.Link was to not only fill the gap of Planet Cassandra but to bring the Cassandra community together. Feel free to reach out if you wish to collaborate with us on this project in any capacity.
We are a technology company that specializes in building business platforms. If you have any questions about the tools discussed in this post or about any of our services, feel free to send us an email!
