https://amoro.apache.org/docs/latest/engines/ Recent content in Engines on Amoro Hugo en-us https://amoro.apache.org/docs/latest/flink-cdc-ingestion/ Mon, 01 Jan 0001 00:00:00 +0000 https://amoro.apache.org/docs/latest/flink-cdc-ingestion/ <!-- - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. --> <h1 id="apache-cdc-ingestion">Apache CDC Ingestion</h1> <p>CDC stands for Change Data Capture, which is a broad concept, as long as it can capture the change data, it can be called CDC. <a href="https://github.com/apache/flink-cdc">Flink CDC</a> is a Log message-based data capture tool, all the inventory and incremental data can be captured. Taking MySQL as an example, it can easily capture Binlog data through Debezium and process the calculations in real time to send them to the data lake. The data lake can then be queried by other engines.</p> https://amoro.apache.org/docs/latest/flink-datastream/ Mon, 01 Jan 0001 00:00:00 +0000 https://amoro.apache.org/docs/latest/flink-datastream/ <!-- - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. --> <h1 id="flink-datastream">Flink DataStream</h1> <h2 id="add-maven-dependency">Add maven dependency</h2> <p>To add a dependency on Mixed-format flink connector in Maven, add the following to your pom.xml:</p> https://amoro.apache.org/docs/latest/flink-ddl/ Mon, 01 Jan 0001 00:00:00 +0000 https://amoro.apache.org/docs/latest/flink-ddl/ <!-- - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. --> <h1 id="flink-ddl">Flink DDL</h1> <h2 id="create-catalogs">Create catalogs</h2> <h3 id="flink-sql">Flink SQL</h3> <p>The following statement can be executed to create a Flink catalog:</p> https://amoro.apache.org/docs/latest/flink-dml/ Mon, 01 Jan 0001 00:00:00 +0000 https://amoro.apache.org/docs/latest/flink-dml/ <!-- - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. --> <h1 id="flink-dml">Flink DML</h1> <h2 id="querying-with-sql">Querying with SQL</h2> <p>Amoro tables support reading data in stream or batch mode through Flink SQL. You can switch modes using the following methods:</p> https://amoro.apache.org/docs/latest/flink-getting-started/ Mon, 01 Jan 0001 00:00:00 +0000 https://amoro.apache.org/docs/latest/flink-getting-started/ <!-- - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. --> <h1 id="flink-getting-started">Flink Getting Started</h1> <h2 id="iceberg-format">Iceberg format</h2> <p>The Iceberg Format can be accessed using the Connector provided by Iceberg. Refer to the documentation at <a href="https://iceberg.apache.org/docs/latest/flink-connector/">Iceberg Flink user manual</a> for more information.</p> https://amoro.apache.org/docs/latest/spark-configuration/ Mon, 01 Jan 0001 00:00:00 +0000 https://amoro.apache.org/docs/latest/spark-configuration/ <!-- - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. --> <h1 id="spark-configuration">Spark Configuration</h1> <h2 id="catalogs-configuration">Catalogs configuration</h2> <h3 id="using-mixed-format-in-a-standalone-catalog">Using Mixed-Format in a standalone catalog</h3> <p>Starting from version 3.x, Spark supports configuring an independent Catalog. If you want to use a Mixed-Format table in a standalone Catalog, you can create a mixed_catalog and load catalog metadata from AMS with following properties:</p> https://amoro.apache.org/docs/latest/spark-ddl/ Mon, 01 Jan 0001 00:00:00 +0000 https://amoro.apache.org/docs/latest/spark-ddl/ <!-- - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. --> <h1 id="spark-ddl">Spark DDL</h1> <h2 id="create-table">CREATE TABLE</h2> <p>To create an MixedFormat table under an Amoro Catalog, you can use <code>using mixed_iceberg</code> or <code>using mixed_hive</code> to specify the provider in the <code>CREATE TABLE</code> statement. If the Catalog type is Hive, the created table will be a Hive-compatible table.</p> https://amoro.apache.org/docs/latest/spark-getting-started/ Mon, 01 Jan 0001 00:00:00 +0000 https://amoro.apache.org/docs/latest/spark-getting-started/ <!-- - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. --> <h1 id="spark-getting-started">Spark Getting Started</h1> <h1 id="iceberg-format">Iceberg Format</h1> <p>The Iceberg Format can be accessed using the Connector provided by Iceberg. Refer to the documentation at <a href="https://iceberg.apache.org/docs/latest/getting-started/">Iceberg Spark Connector</a> for more information.</p> https://amoro.apache.org/docs/latest/spark-queries/ Mon, 01 Jan 0001 00:00:00 +0000 https://amoro.apache.org/docs/latest/spark-queries/ <!-- - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. --> <h1 id="spark-queries">Spark Queries</h1> <h2 id="querying-with-sql">Querying with SQL</h2> <h3 id="querying-mixed-format-table-by-merge-on-read">Querying Mixed-Format table by merge on read</h3> <p>Using <code>Select</code> statement to query on Mixed-Format tables.</p> https://amoro.apache.org/docs/latest/spark-writes/ Mon, 01 Jan 0001 00:00:00 +0000 https://amoro.apache.org/docs/latest/spark-writes/ <!-- - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. --> <h1 id="spark-writes">Spark Writes</h1> <h2 id="writing-with-sql">Writing with SQL</h2> <h3 id="insert-overwrite">INSERT OVERWRITE</h3> <p><code>INSERT OVERWRITE</code> can replace the partition in a table with the results of a query.</p> https://amoro.apache.org/docs/latest/trino/ Mon, 01 Jan 0001 00:00:00 +0000 https://amoro.apache.org/docs/latest/trino/ <!-- - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. --> <h1 id="trino">Trino</h1> <h2 id="iceberg-format">Iceberg format</h2> <p>Iceberg format can be accessed using the Iceberg Connector provided by Trino. please refer to the documentation at <a href="https://trino.io/docs/current/connector/iceberg.html#">Iceberg Trino user manual</a> for more information.</p> https://amoro.apache.org/docs/latest/flink-using-logstore/ Mon, 01 Jan 0001 00:00:00 +0000 https://amoro.apache.org/docs/latest/flink-using-logstore/ <!-- - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. --> <h1 id="using-logstore">Using Logstore</h1> <p>Due to the limitations of traditional offline data warehouse architectures in supporting real-time business needs, real-time data warehousing has experienced rapid evolution in recent years. In the architecture of real-time data warehousing, Apache Kafka is often used as the storage system for real-time data. However, this also brings about the issue of data disconnection between offline data warehouses.</p>