I have no idea what could be causing it from what you have said. I will say that the through hub exhaust keeps outboards from having much power in reverse. How fast are you trying to back up? If slow and you are getting a "vibration" does it seem you may be hitting something even though you know you are in deep water? The shift dog could have rounded off edges and be jumping in and out of gear. This usually only happens in forward, but the dog could be reversed in the older engines. No good reason for doing so other than the cost of new dog and gear.
Loose prop, nut backing off, worn gears, worn clutch dog
shift rod out of adjustment, exhaust related ventilation
Tilt lock not engaging when in reverse,
prop pulls engine away from the transom
then slams back when it ventilates.
Worn out rubber mounts
how does she feel when its in forward? i would pull the prop and check the prop shaft for runout, check the prop for bent blades or any other visual damage, inspect the mounts and make sure shes secured tight to the transom