When you say 'the customiser breaks' what do you mean?
I have an open issue that I initially thought was the customiser - image path issues. Turns out that the issue is also with the media library.
Is this similar to your problem or different?
see: https://wordpress.org/support/topic/image-path-in-media-library-uses-original-site-path