[flink] fix namespace for lineage from table path to warehouse path#7576
[flink] fix namespace for lineage from table path to warehouse path#7576jsingh-yelp wants to merge 1 commit intoapache:masterfrom
Conversation
jsingh-yelp
force-pushed
the
fix-namespace-name-for-flink-lineage
branch
from
a1eb7da to
63fd05e
Compare
|
@JingsongLi / @XiaoHongbo-Hope Sorry for the late addition, but is it possible to get a review on this and include it with the latest 1.4 release? |
jsingh-yelp
changed the title
|
cc: @yunfengzhou-hub can I please get a review on this, it is a follow up to: #7311 |
| */ | ||
| public static String getNamespace(Table table) { | ||
| return PAIMON_DATASET_PREFIX + CoreOptions.path(table.options()); | ||
| return CatalogUtils.warehouse(CoreOptions.path(table.options()).toString()); |
There was a problem hiding this comment.
CatalogUtils.warehouse() is fragile — it assumes the standard <warehouse>/<db>.db/<table> layout, maybe we should pass meaningful message from Catalog.
There was a problem hiding this comment.
@JingsongLi Thanks this is a valid concern.
I was thinking of using the warehouse property from Catalog.options() (Code Ref) directly as the namespace, with a fallback to a constant "paimon" string when it's not available.
Is this something in line what you had in mind or you were thinking something else?
2 participants
Purpose
Fix the namespace for Paimon table lineage datasets. Previously the namespace was set to the full table path which cannot be used to group datasets meaningfully, changed it to use the warehouse root path via
CatalogUtils.warehouse(), to better follow namespace conventions.Tests