Hello,
Checking to see if anyone else has successfully setup a data feed to Visier with the Extend using the Snowflake API? We have successfully made a connection to our Snowflake instance and are able to execute data extraction jobs however it seems no matter how we configure the extraction or settings the job does not extract all of the records from the table or view. There are records we can clearly see on the Snowflake side that Visier does not include in the extraction for no obvious reason.
We have tried using the 3 different options available in the Snowflake API connection settings (Temporal, Transactional Profile, and Full Historical Restatement) but they all result in the same thing and loss of records during the extraction. I am a bit stumped after many attempts to extract data that it does not extract everything I am asking it to from the snowflake table.
Looking for some assistance, thank you!
1 Like
Hi Austin,
I’ve reached out to a member from the engineering team and this is what they’ve said:
We’re still working on updating our external docs for our data warehouse connectors, but in general, to pull data as expected from Snowflake. For each of the connector settings:
Temporal, Transactional Profile: They should set a Key and Last Updated Date column which gives a distinct ordering for the data (ie. there should not be multiple records with the same Key and Date)
Full Historical Restatement: The selected Key column should have a unique record for each individual key (ie. there should not be multiple records with the same Key)
Hopefully this helps - let me know if you have any other questions!
Christina
1 Like
Hi Christina,
Thank you for the response, this was helpful information.
One additional question I have is what if I leave the key empty? I want to always pull a rolling restatement window back in the data of 2 weeks from the snapshot date.
For context, the data set we are pulling is work hours tied to an employee per day and per pay code. An employee can have multiple pay codes per day (i.e. regular pay, overtime, breaks, etc.) and these can be associated with more then one department per day so there can be many rows per employee per day.
1 Like
Hi Austin,
Sorry for the late reply! Here is an update I gathered from our engineers.
Looking at the data being pulled in, it appears that, unfortunately, there isn’t a straightforward way to retrieve it as-is. The challenge lies in the lack of a distinct combination of event date and key column.
We have created a story to implement a solution for this, as it’s an issue we’ve encountered before and are likely to face again until it’s resolved. We’re aiming to have this available by the end of the year.
In the meantime, the best approach would be to create a view within Snowflake that concatenates the three columns (employee ID, department, pay code) into a new column. This new column can then be set as the key column for pulling in the table.
Christina
1 Like