I've noticed the same thing. The inactivity condition fixes the failure mode where the train leaves because it's order has been fulfilled and some inserters still have items they intended to deliver, but now have no wagon to place them in.
There's another failure mode where the order completely fills the wagon, and the inserters have nowhere to put their items because the wagon is full, not because it left.
This can happen any time there are multiple inserters working on the same wagon (which if you care about speed is always), or when an inserter has a stack size greater than 1 (which again, if you're not at the very beginning of the tech tree, is always).
The easiest way to deal with this is to ensure that each inserter only transfers one item type. This is not optimal for speed, but it's easy and reliable.
The only other method I've found is to use combinators to keep track of the number of required items, then on each subsequent inserter, run the signal through an arithmetic combinator and subtract 12 (the max stack inserter stack size) from the total. This way, when a lot is required, all inserters will be working, but as the count nears the storage limit inserters shut off until only one is working. That last one also needs it's stack size to be set by a signal so that it only inserts exactly the required amount on the last swing.