logic diary: data consistency test via Power Query
Logic to verify data coherence between Views from Data Lake that are 'In progress' and data extracted from SAP QAS
task, task, task!
After the 'View Creation' task was marked as 'Closed', it was time for the 'View Validation' task to be 'In Progress'.
As a big fan of Power Query, I decided to use it to verify if all the rows in the .xlsx file, exported from the SAP QAS environment, were present in the View.
solution!
My solution was as follows:
I performed the necessary transformations to ensure a successful JOIN.
I used the "Merge Queries" option with the "Full Outer Join" type to include all rows from both tables, ensuring that all data was compared.
After merging, I expanded the resulting table to include columns from both original tables.
I used the "View" tab to analyze the results. I checked for "null" values, which would indicate missing data in one of the tables.
be nice!
This testing logic can be applied to any data integration between different systems. Adapting this technique to your specific needs can help improve data quality and consistency.
I would love to hear your thoughts on this logic! Other strategies to ensure data coherence are always welcome, or feel free to join the discussion around this topic!
post recommendation!
so I was listening to this song throughout the entire task process. It really kept me motivated and focused. Maybe you'll enjoy it too! (Thanks, Sabrina Carpenter!)