[HUDI-8889] Trim unnecessary columns during MoR snapshot read

[TheR1sing3un]

Consider following case when we perform snapshot-read on MoR table:

1. full schema: (A, B, C, D)
2. primary key field: A
3. pre-combine key field: B
4. query with select D from table

for each file group

data file	log files	operation	payload allow projection pushdown	actual read schema
exist	empty	...	...	(_hoodie_record_key, B, D)
empty	exist	...	...	(_hoodie_record_key, B, D)
exist	exist	skip_merge	...	(_hoodie_record_key, B, D)
exist	exist	payload_combine	Y	(_hoodie_record_key, B, D)
exist	exist	payload_combine	N	(_hoodie_record_key, A, B, D)

However, except for the last two case, we only need to read column D on the file in other cases.

Change Logs

1. Trim unnecessary columns during MoR snapshot read

Impact

Improves performance when MoR snapshot-read with not actually merged

Risk level (write none, low medium or high below)

low

Documentation Update

none

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[TheR1sing3un]

@hudi-bot run azure

[hudi-bot]

CI report:

• 39f927a UNKNOWN
• 4842151 UNKNOWN
• 2ad28a1 Azure: FAILURE Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build