In at least 2018.2 and possibly other versions there appears to be an undocumented bug that readback requests mess with the command buffer:

• Readback requests inserted in the middle will split the command buffer¹⁾, however the second part will have LoadAction=Clear which destroys everything rendered before the request
• Readback requests inserted at the end will make Unity skip presentDrawable which results in the screen not being updated, and the behavior differs between devices:
  • Those configured for debugging or currently debugging: nothing renders but input is accepted and the game continues to run.
  • Others: crash caused by an OS timeout

The workaround is to move the readback request to the beginning of the next frame.

In some cases^{[research needed]}, the callback will be called but when attempting to call GetData, it will throw an exception.²⁾

The cause is unknown, it does not appear to be detectable beforehand. All AsyncGPUReadbackRequest properties indicate that it's a valid completed request. It seems to be safe to catch and ignore this exception.