[DRAFT] Restructure Repo for Table Format specific modules

[the-other-tim-brown]

Important Read

• Please ensure the GitHub issue is mentioned at the beginning of the PR

What is the purpose of the pull request

• Shows the implementation of the proposed design in [611] RFC: Module Restructuring #612

Brief change log

• Creates a module per table format
• Creates a module for the integration tests that rely on these table formats

Verify this pull request

TODO

[yihua]

Will this be a problem for running tests around HFile since the Hudi Spark bundle shades dependencies, particularly HBase ones, in a different way?

[the-other-tim-brown]

It should not if the dependencies are properly configured within the bundle, that is the main reason behind the shading. We can have different versions on the same path because they will be completely self contained.

[yihua]

That's true except for org.apache.hudi.io.storage.HoodieHBaseKVComparator, which is not relocated. In the Hudi Spark bundle, org.apache.hudi.io.storage.HoodieHBaseKVComparator extends from org.apache.hudi.org.apache.hadoop.hbase.CellComparatorImpl after Hudi bundle relocation. I'm wondering if that causes a clash here.

[the-other-tim-brown]

I'm adding a new framework for testing where I essentially execute java -cp ... style command with the bundles on the classpath to validate that the jars are not relying on any dependencies brought onto the test path by our test setup to do the conversion. This test also asserts that the results can be read back properly.

[yihua]

One way to validate the conjecture above is to remove this exclusion to see if the test passes or a different error is thrown (by removing the exclusion, there should be no clash any more, and the org.apache.xtable.shade.org.apache.hudi.io.storage.HoodieHBaseKVComparator is written as the comparator in the HFile if the test table is prepared using the XTable bundle). Still, we need to figure out how to properly handle org.apache.hudi.io.storage.HoodieHBaseKVComparator as the comparator class since the fully-qualified class name needs to be preserved for reading HFile format in Hudi.