speed up chimerge

[zsyf102900]

i have 10million intervals to merge

is there need to add loop for topk_smallest_chiinterval to speed up

[lisette-espin]

Do you mean the merging part from lines 159 to 165?
If so, one can re-write that block of code using masks to change the desired row/cols of frequency_matrix. Then, you won't need the for-loops.

[zsyf102900]

later,i found it's not correct to add FOR loop from line 159 to 165
the truth is that each time we merge one pair intervals,actually the interval frequency_matrix has changed , consequently the chi2_val also changed ,we need to recalcute chi2_val each time

[zsyf102900]

i have another question from line 159 -165 which involves intervals-merging
we should not use (lower,upper) index ,each time we merge intervals ，(upper) index may be deleted.

1. next time , you add frequency_matrix , upper index may not exists
2. the (lower,upper) index may not correspond to interval_values

solutions:
we can reset_index interval _values as dataframe_df index

looking forwards to your reply

[zsyf102900]

sorry, i couldnot upload my codes for the info_security regulation in huawei