Xamarin Studio 5.7 and Xamarin for Visual Studio 3.9 introduced a change in the "MaxPermSize" option that the Android designer uses when starting Java. The first sign that you're hitting this problem is a message in the Android designer:
Connection to the layout renderer failed. This may be be caused by a misconfiguration of Java.
Please close and open the file again.
Note that this message itself can have several underlying causes. To check if the workarounds will help in your particular case, you'll need to check in the log files  for the following error:
Error occurred during initialization of VM
Could not reserve enough space for object heap
Error: Could not create the Java Virtual Machine
(This error message might be broken up by other error messages.)
: Log file locations
Xamarin Studio on Windows:
Xamarin Studio on Mac:
Install Java JDK 1.8 (either 64-bit or 32-bit will work).
Create or update the "String Value"
HKEY_CURRENT_USER\Software\Novell\Mono for Android\JavaSdkDirectory in the registry. Set it to the location of Java JDK 1.8. For example, for the default install path of the 32-bit JDK 1.8 on 64-bit Windows:
C:\Program Files (x86)\Java\jdk1.8.0_31
(If you have Xamarin Studio 5.7 or higher installed, you can also modify this setting under "Tools -> Options -> SDK Locations -> Android -> Java SDK (JDK) -> Location". Visual Studio does not yet have a UI for adjusting this setting, but one will be added in a future release.)
Restart the IDE. The Android designer should now use Java 8, which does not use the "MaxPermSize" setting.
Note that this workaround will affect all Java processes on the system, so it might cause problems for other programs that use Java.
Create or update the
_JAVA_OPTIONS environment variable so that it includes the string
If 256 megabytes ("256m") does not stop the problem, you can experiment with adjusting the number down or up.
Restart the IDE.
If you see a different error message in the log files or no promising error messages at all, then a good next step would be to attach back your log files on an email to one of the support email addresses listed on https://store.xamarin.com/account/.
The candidate improvement for this issue will decrease the default "MaxPermSize" setting to a compromise between the new value and the old value that was used in Xamarin Studio 5.5.4 and Xamarin for Visual Studio 3.8. It will also provide a way for users to adjust the "MaxPermSize" setting in cases where the default setting does not work on their systems.
For reference, the corresponding bug tracking these upcoming changes is https://bugzilla.xamarin.com/show_bug.cgi?id=26026.