Metadata Sync in Fabric Lakehouse

The post discusses improvements to metadata refresh processes in Microsoft Fabric Lakehouse. I previously built a Spark Notebook using an undocumented endpoint for forcing delta lake refreshes but have since discovered an officially supported alternative.

Previous Approach

The older method relied on an unsupported endpoint with a command structure:

{"commands": [{"$type": "MetadataRefreshExternalCommand"}]}

Current Solution

Microsoft now provides an official REST API endpoint:

POST https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/sqlEndpoints/{sqlEndpointId}/refreshMetadata

Improvements

Our team has enhanced the process by replacing Spark entirely with a pure Python Notebook implementation, offering faster startup times and improved efficiency. A complete Python implementation is available via our public GitHub repository, including functions for obtaining SQL endpoint IDs and monitoring refresh completion with configurable timeout parameters.