Incremental load based on several columns

[edwin-aguilar-iwar]

Hi, I have a table that should be incrementally loaded based on several timestamp columns (timecreated, timemodified, lastaccess, lastlogin). I'm currently loading this table via only timemodified as I (incorrectly) thought that it got updated whenever any of the other 3 got updated. However, this is not the case. Hence, I have some records that are not up to date.

Is there a way to specify multiple incremental keys? Or would the correct strategy be to perform different loads per incremental key (in this case, I would have to execute the ingestr command 4 times). If this is the case, what impact would it have on the state that is managed behind doors?

Thanks!

[turtleDev]

Hi @edwin-aguilar-iwar

ingestr currently only supports incremental loads using a single column. This isn't strictly a technical limitation, but a sensible convention we adhere to.

With regards to your specific scenario, I can suggest the following:

• can you create a view over your table that can create a column called last_updated as greatest(timecreated,timemodified,lastaccess,lastlogin) and use that as your incremental key?
• if creating a view is not possible, you can use custom queries to project a last_updated column and filter it manually. Here's an example: query: select *, greatest(timecreated, timemodified, lastaccess, lastlogin) as last_updated from table

Would any of these solutions be feasible for you?

P.S. it would be helpful to know the ingestr version and source that you're using.