I just delete the Xcode.app from /Applications and don’t mess around with any other files! Caches should be OK, as they would (presumably) be re-created if necessary.

However, it’s certainly possible that some supporting files are shared between different versions so you might break something by deleting them. For example files in these directories look like they are shared:

/Library/Developer/CoreSimulator/Profiles/Runtimes/

~/Library/Developer/Xcode/

That said, there could still be things within those directories that are only needed by older versions (e.g. old simulators). Unless you are desperate for space, I would suggest caution and only delete the Xcode.app itself.

I use a tool called “DevCleaner for Xcode” from the Mac App Store to delete some old files. For example, when connecting a real device, the iOS runtime is copied over (this is what is happening when Xcode copies files from the device when it’s first connected). As you install newer iOS versions, the older ones aren’t needed (e.g. after updating to iOS 14.3, I don’t need the iOS 14.2.x ones any more). These are safe to delete and if you accidentally delete a new one, it will be re-created the next time you connect your device to Xcode.

If you Google around, you’ll find various articles explaining how to clean up old files and obsolete simulators. I’m always worried that will remove a simulator from an older Xcode which I still want. But at the time of writing, we should all be using Xcode 12.x and I think we won’t even be able to submit apps built with Xcode 11 from April 2021. So the older Xcode versions become obsolete quite quickly.

Hope this helps.