Reitse Eskens has the need for speed:
The issue was quite straightforward. The client has a mapping data flow in Synapse that processes a few hundred to a few thousand rows but takes 15 minutes to complete. The low number of rows compared to the time necessary is a cause for concern.
The data extraction needs a staging storage account where the data is written into TXT files. The second step of the mapping data flow reads the TXT files and writes them out in delta format, which is Parquet files.
The source is an S4Hana CDC table, the target of which is a regular Azure storage account.
Read on for Reitse’s summarization of the troubleshooting and testing process, as well as what ended up working for this customer.