The performance impact is same as any other chest with large inventory. When inserter is trying to take item out, it has to iterate more inventory slots. Inserters iterate from end of the inventory so having chests which are mostly empty have more impact than full chests.
As far as I know there isn't any impact from just having any size of inventory (except of its impact on save file size). Only inserters (and maybe other entities) interacting with it impacts the performance.
The performance impact scales linearily with number of interacting entities and number of inventory slots the chest has. Chest size itself isn't a factor, just its inventory size.
This mod is all about "setting the limits as low as you can". You can always increase the limits later if you need to, but if you make the inventory smaller later, you will lose items in those slots which were removed.