Repository navigation
Add bidirectional Apache Gravitino converter #256
Description
Hi all 👋
I'd like to gauge interest in adding a bidirectional converter between Apache Ossie semantic models and Apache Gravitino under converters/gravitino/.
Background
Apache Gravitino is a high-performance, geo-distributed, federated metadata lake (Apache Incubating). It provides a unified REST API to manage metadata across many data sources — Hive, JDBC (MySQL/PostgreSQL/Doris/StarRocks), Iceberg, Paimon, Hudi, Kafka, and file-based sources — through a single "catalog of catalogs" abstraction.
Gravitino already knows about physical metadata (catalogs, schemas, tables, columns, types, partition keys, tags, properties) for the entire big-data ecosystem. What it does not carry is Ossie's semantic layer — metrics, cross-dataset relationships, time-dimension markers, and AI context. A converter between the two would let any Ossie-compatible BI/AI tool access the full inventory of a big-data platform through Gravitino's unified API, and conversely let a semantic model authored in Ossie be materialized as table metadata in a big-data catalog.
This is also a two-Apache-podlings integration — both Gravitino and Ossie are Apache Incubating projects, and a converter between them benefits both ecosystems.
I'd like to contribute this converter myself. I'm a big-data platform engineer with deep experience in the Hive/Spark/Iceberg ecosystem, and I've already studied the existing converters/polaris/ codebase and prototyped the Gravitino REST API integration. I'm happy to follow the "Contributing a New Converter" steps (vendor entry, both directions, tests, README) and would open a PR once the approach is agreed.
What's included
converters/gravitino/— a Java/Maven module following the pattern ofconverters/polaris/.GravitinoClient— a thin REST client calling Gravitino's existing API (Java 11HttpClient+ Jackson), shared by both directions:GET /api/metalakes/{m}/catalogs→ list catalogsGET /api/metalakes/{m}/catalogs/{c}/schemas→ list schemasGET /api/metalakes/{m}/catalogs/{c}/schemas/{s}/tables→ list tablesGET /api/metalakes/{m}/catalogs/{c}/schemas/{s}/tables/{t}→ load tablePOST /api/metalakes/{m}/catalogs/{c}/schemas/{s}/tables→ create table
Import (Gravitino → Ossie)
Reads table/schema metadata from a Gravitino metalake and produces an Ossie semantic model YAML.
| Gravitino concept | Ossie concept |
|---|---|
| Metalake + Catalog | SemanticModel |
| Schema | dataset grouping within the model |
| Table | Dataset |
table.name |
dataset.name |
catalog.schema.table |
dataset.source |
table.columns[] |
dataset.fields[] |
column.name + column.type |
field.name + expression (ANSI_SQL) |
column.type = timestamp/date |
field.dimension.is_time = true |
table.partitionKeys[] |
flagged in field.description or custom_extensions |
table.properties |
dataset.custom_extensions[vendor_name: GRAVITINO] |
table.tags |
candidate source for dataset.ai_context |
CLI: ossie-gravitino-converter import --url URL --metalake M [--catalog C] [--auth ...] -o output.yaml
Export (Ossie → Gravitino)
Takes an Ossie semantic model and creates/updates the corresponding tables in Gravitino via its REST API.
| Ossie concept | Gravitino concept |
|---|---|
SemanticModel |
Metalake + Catalog (1:1, or configurable mapping) |
Dataset |
Table |
dataset.name |
table.name |
dataset.source |
resolved to catalog.schema.table |
dataset.fields[] |
table.columns[] |
field.expression (ANSI_SQL) |
column.name (extracted from expression) |
field.dimension.is_time |
column.type set to timestamp / date |
dataset.primary_key |
table.primaryKey or table.constraints |
dataset.unique_keys |
table.constraints (unique) |
dataset.custom_extensions[vendor_name: GRAVITINO] |
table.properties (round-trip) |
dataset.description / field.description |
table.comment / column.comment |
CLI: ossie-gravitino-converter export --url URL --metalake M --catalog C ossie_model.yaml [--auth ...] [--dry-run]
Design considerations
- Round-trip fidelity:
Gravitino → Ossie → Gravitinoshould be lossless. Gravitino's relational model does not have a native "relationship" or "metric" concept, so Ossie relationships and metrics are serialized intocustom_extensions[vendor_name: OSSIE]on the corresponding table, following the round-trip-fidelity pattern used by the Databricks converter ([OSSIE][DATABRICKS] Add Databricks Unity Catalog Metric View converter #224). - No changes on the Gravitino side: the converter is a pure client of Gravitino's existing REST API. Gravitino does not need to know about Ossie.
- Idempotency: the exporter checks if a table already exists before creating (GET then POST/PUT), so re-running is safe.
- Type inference: Ossie fields are expression-based and don't carry explicit physical types. The exporter uses heuristics (field name patterns +
is_timeflag) to infer Gravitino column types, similar toPolarisExporter.inferIcebergType(). Gravitino'sExternal Typefallback is used when inference fails. - Big-data ecosystem gap: Ossie currently has converters for BI/SaaS tools (dbt, Snowflake, Tableau, GoodData, Omni, …) but none for big-data catalog/metadata services. Gravitino is the natural bridge — one converter unlocks Hive, Iceberg, Paimon, Hudi, JDBC, and more.
Approach
I'd follow the existing converter pattern (closest analogue: converters/polaris/):
GravitinoClient— REST client authenticating via Gravitino's OAuth or basic auth.GravitinoImporter— walks metalake → catalog → schema → table, maps each table to an OssieDatasetwith fields derived from columns. Time types (timestamp,date,timestamptz) setdimension.is_time.GravitinoExporter— reads an Ossie model, infers column types, and calls Gravitino's create-table API.- Reuse
OsiYamlGenerator/OsiModelParser(or the sharedossie-commonmodule if refactor(converters): group Java modules and share generated Ossie models #235 lands first). - Gravitino tags/properties that carry semantic hints (e.g.
description,owner) flow intoai_contextordescription; non-semantic properties go intocustom_extensions[vendor_name: GRAVITINO]. - CLI + tests (mocked Gravitino API responses with Hive + JDBC-MySQL fixtures) + README, following CONTRIBUTING.md.
Before I open the PR: is there anything you'd want me to keep in mind on approach or scope? And should I wait for #235 (Java shared module) to land, or build against the current converters/polaris/ structure for now?
I'll go ahead and start the implementation either way — just want to make sure I'm aligned with the project's direction.
Thanks!
Activity
wForget commentedon Jul 24, 2026
Sorry that PR #235 did not lead to much discussion. Also, the
commonmodule it introduced only contains the OSI models generated fromcore-spec/osi-schema.json. I think you can feel free to move forward with your idea.