Concepts on Amoro

Catalogs

Mon, 01 Jan 0001 00:00:00 +0000

Catalogs

Introduce multi-catalog

A catalog is a metadata namespace that stores information about databases, tables, views, indexes, users, and UDFs. It provides a higher-level namespace for table and database. Typically, a catalog is associated with a specific type of data source or cluster. In Flink, Spark and Trino, the multi-catalog feature can be used to support SQL across data sources, such as:

Self-Optimizing

Mon, 01 Jan 0001 00:00:00 +0000

Self-optimizing

Introduction

Lakehouse is characterized by its openness and loose coupling, with data and files maintained by users through various engines. While this architecture appears to be well-suited for T+1 scenarios, as more attention is paid to applying Lakehouse to streaming data warehouses and real-time analysis scenarios, challenges arise. For example:

Table Watermark

Mon, 01 Jan 0001 00:00:00 +0000

Table Watermark

Table freshness

Data freshness represents timeliness, and in many discussions, freshness is considered one of the important indicators of data quality. In traditional offline data warehouses, higher cost typically means better performance, creating a typical binary paradox in terms of cost-performance trade-off. However, in high-freshness streaming data warehouses, massive small files and frequent updates can lead to performance degradation. The higher the freshness, the greater the impact on performance. To achieve the required performance, users must incur higher costs. Thus, for streaming data warehouses, data freshness, query performance, and cost form a tripartite paradox.