By default, the Solaris kernel maps the system stack RWX; this behaviour is mandated by the SPARC V8 ABI. Since an non-executable stack gets in the way of certain classes of security bug exploits, a feature was added to Solaris 2.6 that allows system administrators to remove the "X" protection from the stack.
To enable this feature, add the following to /etc/system:
* Foil certain classes of bug exploits set noexec_user_stack = 1 * Log attempted exploits set noexec_user_stack_log = 1
This is no general "cure-all" protection for buffer overflow exploits. It may also break certain SPARC V8 ABI conforming programs.
This feature also requires hardware support; it is only available on UltraSPARC (sun4u), sun4d and sun4m systems.
The SPARC V9 ABI no longer maps the stack executable, so 64 bit applications have less to worry about. 32-bit applications running on a 64-bit kernel are not so lucky.
This feature also works on x86 family CPUs which support the "NX" (no execute) page table bit. In Solaris 10, we've marked the datasegment and stacksegment of Solaris executables no-execute.