I downgraded till 8.1.1 to get rid of the lag.
EDIT: I test 8.3.0 also (even no raid bug) so it helps to find the change that cause this. 8.3 is also lag free, so it start with 8.4.x for me
EDIT2: I let it run for some time on 8.5.5 somehow it recovered now. I have a only attack at night mod. And it just got day, so i will check how it reacts at night again.