To enable a VNC Server on Opensolaris you just have to follow some easy steps:

  1. Edit the /etc/X11/gdm/custom.conf (in newer snv releases the gdm directory lies directly in /etc). Add the following two properties in the respective sections: 
    [security]
    DisallowTCP=false

    [xdmcp]
    Enable=true
  2. restart gdm
    # svcadm restart gdm
    remember, that this will disconnect everyone logged in through gdm, i.e. your X Session!
  3. Start the vnc service
    # svcadm enable xvnc-inetd

Now you can connect to you new OpenSolaris VNC Server with the VNC client of your choice. I recommend JollysFastVNC since it is indeed fast, and allows you to automatically tunnel your VNC connection through ssh. (Remember, VNC is itself not encrypted)

Have fun!