ContextQMD
Back to Hudi

README

hudi-platform-service/hudi-metaserver/README.md

0.5.33.0 KB
Original Source
<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -->

How to compile

There are two ways, if there is a thrift binary in the local, please see the first one. If docker is more convenient for you, please see the second way.

Compile with local Thrift binary

Firstly, make sure /usr/local/bin/thrift exists, and its version is the same with the one declared in the pom.xml.

Then compile the module with maven options -Pthrift-gen-source. For example, mvn install -pl :hudi-metaserver -DskipTests=true -Phudi-platform-service

Compile with Docker

Firstly, make sure there is a docker in the local.

Then just compile the module as a normal one. For example, mvn install -pl :hudi-metaserver -DskipTests=true -Phudi-platform-service

Attention: Apple m1 cannot install thrift by docker successfully. The script will use homebrew to do it. So make sure homebrew exits.

Source code generated by Thrift

After packaging, the generated source code are placed in target/generated-sources/gen-java. It looks like,

text
├── gen-java
│       └── org
│           └── apache
│               └── hudi
│                   └── metaserver
│                       └── thrift
│                           ├── AlreadyExistException.java
│                           ├── FieldSchema.java
│                           ├── ...

How to run a job with Hudi Metaserver

Start Hudi Metaserver

  1. modify the hikariPool.properties and config the mysql address. For example,
properties
jdbcUrl=jdbc:mysql://localhost:3306
dataSource.user=root
dataSource.password=password

  1. start the server

    make sure hudi-metaserver-${project.version}.jar is under the directory,

shell
sh start_hudi_metaserver.sh

Write client configurations

properties
hoodie.database.name=default
hoodie.table.name=test
hoodie.base.path=${path}
hoodie.metaserver.enabled=true
hoodie.metadata.enable=false
hoodie.metaserver.uris=thrift://${serverIP}:9090

How to test

Run a unit test with Hudi Metaserver

Add the configurations mentioned in the Write client configurations part, and then set hoodie.metaserver.uris="".

The metaserver runs as an embedded one with h2 database that performs like mysql running locally.