In at least 2018.2 and possibly other versions there appears to be an undocumented bug that readback requests mess with the command buffer:

• readback requests inserted in the middle will split the command buffer (which is fine) however the second part will have LoadAction=Clear which effectively destroys everything rendered before the request
• readback requests inserted at the end will make Unity skip presentDrawable which results in the screen not being updated, and the behavior may differ between devices:
  • those configured for debugging or at least currently debugging - can't see anything happening in the game but input is accepted and things continue to work
  • others - crash caused by an OS timeout

The workaround is to move the readback request to the beginning of the next frame.

In some cases^{[research needed]} (observed at least on 2018.2.21 when exiting play mode and sometimes while paused in editor), the callback will be called but when attempting to call GetData, it will throw an exception.

The cause is unknown, it does not appear to be detectable beforehand (all AsyncGPUReadbackRequest properties indicate that it's a valid completed request) but it seems to be safe to catch and ignore this exception.