I'm not sure if this is a bug, but a filter inserter picks up any fish, even though I set it to only pick up the fully grown fish. If it is not a bug, how can I set up an automated fishery?

The filter insert cannot filter fish. You can set any filter (for example a box) and it will work. I’m not sure that the factorio developer, who added the option of catching fish in this way, was thinking about the possibility of several species of fish.
forums.factorio.com/viewtopic.php?t=64898

Yea, i figured the developers did not intend for it. The fish catcher mod in the link you provided works perfectly though.