The permissions to the /tmp/.X11-* directories have been restricted for security reasons.
In order to still run unprivileged programs such as Xvnc/Xnest/Xvfb, you need to start those programs with the "-pn" option.
You can no longer use displays of the form ":<num>"; only displays with the full hostname will still work: "<hostname>:<num>".
The Solaris 9 versions of Xnest/Xvfb are able to create these sockets as they also run set-gid.