osmo-e1-xcvr should put pin 44 to GND
As Dieter figured out, pin 44 of the IDT82V2081 should be on GND, while we have it floating:
See Table-1 on Page 14 of the data sheet:
IC - 44 IC: Internal Connection, Internal Use. This pin should be connected to ground when in normal operation.