ATLAS-4988: BusinessMetadata with attribute of data type Array takes time to Delete.

[UmeshPatil-1]

ATLAS-4988
What changes were proposed in this pull request?
Background: In Apache Atlas, deleting a Business Metadata (BM) definition requires a pre-check to ensure no existing entities are assigned to those attributes. For attributes with a data type of Array (multi-valued), the deletion process was taking an exceptionally long time (e.g., over 55 minutes for -1,18,300 entities), even when the BM was not assigned to any entity.

Root Cause:

1. Array attributes are not Solr-indexed
   ARRAY BM attributes are not indexed in Solr.
   Solr therefore performs a full collection scan when queried with NOT_EMPTY, causing severe latency.
2. Single-path Solr validation
   The previous logic forced all BM attributes (STRING + ARRAY) through Solr.
   This was efficient for STRING attributes but pathological for ARRAY attributes.
3. Historical entities ignored
   The earlier implementation excluded deleted entities.
   This allowed historical BM references to be missed, leading to potential integrity violations.
4. Unnecessary data retrieval
   The Solr query requested attribute payloads, increasing I/O cost even though only existence was required.

Changes Proposed:

1. Hybrid Validation Strategy: Introduced a conditional check to determine if an attribute is indexable before deciding the search path.
2. Optimized Solr Validation for STRING Attributes: Uses NOT_EMPTY existence check only , Includes historical (deleted) entities.
3. Direct Graph Fallback: Introduced isBusinessAttributePresentInGraph() for non-indexable attributes. This method queries the Graph database (JanusGraph) directly.
4. Subtype & Inheritance Enforcement: Applicable entity types are expanded to include all subtypes.
5. Integrity Maintenance: Ensured that even unindexed attributes are checked before deletion to prevent "dangling" metadata, while maintaining sub-second performance and Prevents accidental deletion under partial failure conditions also maintain strict metadata integrity.

Impact:

Performance Gain: Deletion time for Business Metadata with Array attributes reduced from -55 minutes to -25 second's.
Reliability: Maintains strict data integrity by ensuring no "in-use" Business Metadata can be deleted.
Scalability: The fix ensures that as the number of entities grows, the deletion of metadata remains performant.
Correctness:Prevents deletion when BM is referenced by: Active entities, Deleted (historical) entities, Subtypes via inheritance and Custom entity types.

How was this patch tested?
Maven Build:
Build Successful.

Manual Testing:

1. Creation & Deletion (No Assignment): Created Business Metadata with String and Array types. Verified deletion is nearly instantaneous (-20s - 25s).
2. Deletion Blocked (With Assignment): * Assigned a Business Metadata attribute to a hive_table entity and custom entities.
3. Attempted to delete the BM definition.
4. Historical Validation: BM assigned - Entity Deleted - BM delete attempted - Deletion Correctly Blocked.
5. Verified the system correctly throws ATLAS-409-00-002: Given type has references.
6. Large Dataset Validation: Tested against a repository containing -1,18,000 hive_table entities and custom entities to confirm the Solr timeout and latency issue is resolved.

The Patch was tested on cluster:- Result's as per below
1. Time taken to delete unassigned BM of String type :- Existing Code :- 2 sec - 4 sec, Updated Code:- 2 sec - 3 Sec.
2. Time taken to delete unassigned BM of Array type :- Existing Code :- 1 Hr 40 min, Updated Code:- 1.5 min - 3 min.
Please refer the attached sheet on JIRA-4988 ticket for more insight's

[mneethiraj]

Is it necessary to add ENTITY_TYPE_PROPERTY_KEY filter here? If yes, note that all subTypes of types inapplicableTypes should be checked as well; consider a business attribute associated with a type like DataSet.

[mneethiraj]

Is it necessary to check only for ACTIVE entities? The business metadata can't be deleted even when a deleted entity has reference to it, right?

[sheetalshah1007]

The hasIndexableAttribute check could be expensive if there are many attributes.
Consider caching this in the BusinessMetadataDef or TypeRegistry.

[sheetalshah1007]

The hasIndexableAttribute check could be expensive if there are many attributes.
Consider caching this in the BusinessMetadataDef or TypeRegistry.

[sheetalshah1007]

Suggestions:

1. Add JavaDoc explaining WHY sub-types are needed as this is an important business logic - without JavaDoc, developers won't understand the inheritance scenario.

2. Consider caching the result in TypeRegistry to avoid repeated calculations