I found information about how to speed it up after googling for a while. In the end I think it is worth it to summarize the steps here:
- In the current folder where you are going to run matlab from, create a file name java.opts with the content
- Use compression with ssh, by connecting to the server with
The speed up obtained is particularly noticeable when changing focus between different windows in the Matlab GUI.