Setting Parameters for Historical Data Extraction
When you need to avoid extracting data from before a certain start date, consider the following:
- Specify a “Last Extract Date” upon the initial load to exclude employee records not modified since that date.
- Apply a row filter within your Data Load Definition (DLD) to exclude records before the cut-off date. However, be cautious as this could unintentionally exclude relevant historical transactions.
- Ideally, set the correct application start date to ensure cleaner data extraction moving forward, which is the recommended method for dealing with this situation.
Note that simply overriding the start date in a job may not provide the desired outcome, as certain historical data might still be pulled. Therefore, a combination of initial load restrictions and application-specific configurations should be used.