If I'm reading the reference manual right, the only time you are going to be able to use the output enable function on the I/O pin component is when the pin is in GPIO mode.
That would be if you connected the pin to nothing, a control register, UDB logic, or the like.
Once you connect that pin to the UART output, the UART is going to take over the OUT_EN function, presumably setting it always true.
The tx_en_out pin on the UART component is intended for routing to a separate pin to control the driver enable of an external transceiver chip.
But this begs the question of why you need an output enable at all.
You are running an open-collector bus. UARTs idle with the output high. In your case, that means open. If you delete the tx_en_out pin from the UART and uncheck the output enable function on the I/O pin, I believe that you get what you need. The pin stays open except when the UART is transmitting start bits or zero bits.
Many thanks, @WiCo_1641241! I still don't entirely understand table 21-3, but I take you point that I don't need to. If my bus is open-collector (==open-drain), I just need an "open drain, drives low" transmit pin and it should all work.
If anybody is reading this thread in the future, note that the default UART transmit pin is "strong drive", so you do have to use the Show UART Terminals feature to get an "open drain, drives low" pin.