Right now, it seems the memory unit is updated every second or so. Very often, this is not fast enough and the memory unit is the bottleneck. At other times, it's a waste of cpu to check when there is no activity.
How about a more dynamic algorithm, say, within 1 to 256 ticks (possibly configurable), double the update frequency when there was somehting to do, halve the update frequency when there was nothing to do, or something similar?