This might not be a bug -- upgrading a belt-weaving underground is disallowed as it could break underground belt weaving. Allowing this makes ensuring that undergrounds are in the correct orientation really hard. As in, depending on the order you place blueprints, the underground belt's direction can be different...
You can perhaps work around this this by deleting and underground and re-placing it.
Let me know if this doesn't work for you, or if you think there is still a bug.
An possible change is to make the underground handling an option instead of always-on, but then ensuring underground consistency becomes the player's responsibility instead.