I found that experimental Intrusion Detection module as explained in my previous blog doesn't work as expected if an external plugin's AuthTrans SAF is added in obj.conf request processing and if that SAF returns REQ_PROCEED. This may be a rare case.

My id.conf :

SecRuleEngine on 
SecRequestBodyAccess on
SecRule REQUEST_BODY "junk"

case 1: I created a dummy plugin having AuthTrans function myauth1; which just returns REQ_NOACTION it works fine. (look at <ws7-install-dir>/samples/nsapi/ for examples of how to create a plugin)

    #ifdef XP_WIN32
    #define NSAPI_PUBLIC __declspec(dllexport)
    #else /* !XP_WIN32 */
    #define NSAPI_PUBLIC
    #endif /* !XP_WIN32 */

    #include "nsapi.h"

    extern "C"
    NSAPI_PUBLIC int myauth1(pblock *pb, Session *sn, Request *rq)
    {
        return REQ_NOACTION;
    }

Added in Magnus.conf

Init fn="load-modules" shlib="myauth.so" funcs="myauth1"

Error logs in that case show :

    ...
    ... func_exec reports: executing fn="match-browser" browser="*MSIE*" ssl-unclean-shutdown="true" Directive="AuthTrans" magnus-internal="1"
    ... func_exec reports: fn="match-browser" browser="*MSIE*" ssl-unclean-shutdown="true" Directive="AuthTrans" magnus-internal="1" returned -2 (REQ_NOACTION)

    ... func_exec reports: executing fn="myauth1" Directive="AuthTrans"
    ... func_exec reports: fn="myauth1" Directive="AuthTrans" returned -2 (REQ_NOACTION)

    ... func_exec reports: executing fn="magnus-internal/secrule-filters-insert"
    ... func_exec reports: fn="magnus-internal/secrule-filters-insert" returned -2 (REQ_NOACTION)

    ... func_exec reports: executing fn="ntrans-j2ee" name="j2ee" Directive="NameTrans"
    ...
       

case 2: When I change this AuthTrans SAF to return REQ_PROCEED, it doesn't work as expected:

    #ifdef XP_WIN32
    #define NSAPI_PUBLIC __declspec(dllexport)
    #else /* !XP_WIN32 */
    #define NSAPI_PUBLIC
    #endif /* !XP_WIN32 */
    #include "nsapi.h"

    extern "C"
    NSAPI_PUBLIC int myauth2(pblock *pb, Session *sn, Request *rq)
    {
        return REQ_PROCEED;
    }

Added in Magnus.conf

Init fn="load-modules" shlib="myauth.so" funcs="myauth2"

Error logs in that case shows :

    ... func_exec reports: executing fn="match-browser" browser="*MSIE*" ssl-unclean-shutdown="true" Directive="AuthTrans" magnus-internal="1"
    ... func_exec reports: fn="match-browser" browser="*MSIE*" ssl-unclean-shutdown="true" Directive="AuthTrans" magnus-internal="1" returned -2 (REQ_NOACTION)

    ... func_exec reports: executing fn="myauth2" Directive="AuthTrans"
    ... func_exec reports: fn="myauth2" Directive="AuthTrans" returned 0 (REQ_PROCEED)

    ... func_exec reports: executing fn="ntrans-j2ee" name="j2ee" Directive="NameTrans
    ...

You can add a workaround add this secrule-filters-insert SAF above your ExternalPluginAuthTransSAF function:

<Object name="default">
AuthTrans fn="match-browser" browser="*MSIE*" ssl-unclean-shutdown="true"
AuthTrans fn="magnus-internal/secrule-filters-insert"
AuthTrans fn="ExternalPluginAuthTransSAF"
NameTrans fn="ntrans-j2ee" name="j2ee"
...
</Object>

<Object name="default">
AuthTrans fn="match-browser" browser="*MSIE*" ssl-unclean-shutdown="true"
AuthTrans fn="magnus-internal/secrule-filters-insert"
AuthTrans fn="ExternalPluginAuthTransSAF"
AuthTrans fn="myauth"
NameTrans fn="ntrans-j2ee" name="j2ee"
...
</Object>

Sun Web Server: The Essential Guide has been featured on Dr. Dobbs Multicore Reading list. Here's the link for more details:

http://www.ddj.com/hpc-high-performance-computing/219401196

First, A World Record in Web Workloads - SPECweb

Sun
has obtained a world record SPECweb2005 performance result of 100,209
SPECweb2005 on the Sun SPARC Enterprise T5440 server, using a Sun
Storage F5100 flash array, running Solaris 10, Sun Web Server 7.0
update 5, and Java HotSpot™ Server VM.

Sun SPARC
Enterprise T5440 server delivers 21% greater SPECweb2005 performance
than the HP DL370 G6 with 3.2GHz Xeon W5580 processors, 40% greater
SPECweb2005 performance than the HP DL 585 G5 with four 3.114 GHz
Opteron 8393 SE processors, and  2x the SPECweb2005 performance of the
HP DL 580 G5 with four 2.66GHz Xeon X7460 processors.  There are no IBM
POWER6 results on the SPECweb2005 benchmark. 

For more details, please see the disclosure here on SPEC.org.

Second, A World Record In Transaction Processing Workloads - TPC-C.

Oracle Corporation announced
last month a world record TPC-C benchmark result for Oracle® Database
11g running on Sun SPARC® servers with CMT technology and the Sun
Solaris Operating System. It's a little known fact that the benchmark
actually used Sun Web Server 7 for connection multiplexing as it runs
TUXEDO (transaction processing monitor) NSAPI plugin. The system under test
comprised of - Oracle Database 11g with Oracle Real Application
Clusters and Oracle Partitioning and Tuxedo CFS-R and Sun Web Server
7.0 with Tuxedo plugin running on Solaris 10 10/09 - and achieved a
record-breaking 7.7 million tpmC at $2.34/tpmC.

For more details, please see full disclosure here on TPC.org.

Well, noticed tonight on twitter that the recent SSL/TLS protocol vulnerability is being mentioned and information has been published here.

As far as our Sun Web Server 7 is concerned, we have been looking at this for a number of weeks now but the timing of the public discussion tonight is a bit of a surprise. I will point you at further information from here once officially available.

About LDAP connections in Sun Web Server 7.0 

When I have in my server.xml confiured an Sun LDAP server as given below  :

  <auth-db>
    <name>ldap</name>
    <url>ldap://localhost:1369/o%3DTestCentral</url>
    <property>
        <name>binddn</name>
        <value>cn=Directory Manager</value>
    </property>
    <property>
        <name>bindpw</name>
        <value>...</value><encoded/>
    </property>
  </auth-db>

When I have ACL set

acl "uri=/";
deny (all) user="anyone"; 
allow (all) user="all";  

When I send a request via browser as user "alpha" and its correct password.
First it binds as the user specified in server.xml "binddn" property ( in this case "cn=Directory Manager")
Then sends a LDAP search query with filter "uid=alpha".
Third step it does is it binds as that user "alpha" with the password user specified. If bind is successful, access is allowed.

Here are the entries I get in LDAP server access logs:

[07/Oct/2009:13:16:16 +051800] conn=2 op=-1 msgId=-1 - fd=34 slot=34 LDAP connection from 127.0.0.1 to 127.0.0.1
[07/Oct/2009:13:16:16 +051800] conn=2 op=0 msgId=1 - BIND dn="cn=Directory Manager" method=128 version=3
[07/Oct/2009:13:16:16 +051800] conn=2 op=0 msgId=1 - RESULT err=0 tag=97 nentries=0 etime=0 dn="cn=directory manager"

[07/Oct/2009:13:16:16 +051800] conn=2 op=1 msgId=2 - SRCH base="o=testcentral" scope=2 filter="(uid=alpha)" attrs="c"
[07/Oct/2009:13:16:16 +051800] conn=2 op=1 msgId=2 - RESULT err=0 tag=101 nentries=1 etime=0

[07/Oct/2009:13:16:16 +051800] conn=2 op=2 msgId=3 - BIND dn="uid=alpha,o=TestCentral" method=128 version=3
[07/Oct/2009:13:16:16 +051800] conn=2 op=2 msgId=3 - RESULT err=0 tag=97 nentries=0 etime=0 dn="uid=alpha,o=testcentral"

To understand why we see these three entries, I started debugging Web Server from LASUserEval function and came at get_is_valid_password_ldap function. Note that line numbers in Open Web Server may differ but the concepts are the same :

t@16 (l@16) stopped in get_is_valid_password_ldap at line 455 in file "ldapacl.cpp"
  467       if ((rv = ld->find_userdn(raw_user, basedn, &userdn)) == LDAPU_SUCCESS) {
(dbx) p filter
filter = "uid=alpha"
... 
t@16 (l@16) stopped in LdapSession::find_userdn at line 1253 in file "LdapOps.cpp"
 1253           retval = find(base, LDAP_SCOPE_SUBTREE, filter, attrs, 1 /* no attrs */, res);
(dbx) s
...
t@16 (l@16) stopped in LdapSession::find at line 1174 in file "LdapOps.cpp"
 1174       res = search(base, scope, filter, (char **)attrs, attrsonly, 0);
(dbx) p base
base = 0x6ed6048 "o=TestCentral"
(dbx) p filter
filter = 0xfbfbc98c "uid=alpha"
...
(dbx) n
t@16 (l@16) stopped in LdapSession::search at line 289 in file "LdapSession.cpp"  
289           rv = bindAsDefault();
(dbx) s
t@16 (l@16) stopped in LdapSession::bindAsDefault at line 235 in file "LdapSession.cpp"
  235       int msg_id = ldap_simple_bind(session, ldapRealm->getBindName(), ldapRealm->getBindPwd());
(dbx) p msg_id
msg_id = 1
...
(dbx) n
t@16 (l@16) stopped in LdapSession::bindAsDefault at line 240 in file "LdapSession.cpp"
  240       int ret = ldap_result( session, msg_id, 0, &time_out, &res);
(dbx) p ret
ret = 97
...

As we can see Web Server first tries to bindAsDefault ie. bind as cn=Directory Manager. by looking at LDAP Server logs, we confirm this.

[07/Oct/2009:14:52:03 +051800] conn=4 op=-1 msgId=-1 - fd=34 slot=34 LDAP connection from 127.0.0.1 to 127.0.0.1
[07/Oct/2009:14:52:03 +051800] conn=4 op=0 msgId=1 - BIND dn="cn=Directory Manager" method=128 version=3
[07/Oct/2009:14:52:03 +051800] conn=4 op=0 msgId=1 - RESULT err=0 tag=97 nentries=0 etime=0 dn="cn=directory manager"

When bind is successful, now it tries LDAP search on basedn="o=TestCentra" with filter "uid=alpha"

...
(dbx) n
t@16 (l@16) stopped in LdapSession::search at line 296 in file "LdapSession.cpp"  
296           rv = lastoprv = ldap_search_ext_s(session, base, scope, filter,
297                                      attrs, attrsonly,
298                                      NULL, NULL, &time_out, LDAP_NO_LIMIT, &search_results);
(dbx) p base
base = 0x6ed6048 "o=TestCentral"
(dbx) p scope
scope = 2
(dbx) p filter
filter = 0xfbfbc98c "uid=alpha"
...
(dbx) n
t@16 (l@16) stopped in LdapSession::find at line 1178 in file "LdapOps.cpp"
 1178           break;
(dbx) n
t@16 (l@16) stopped in LdapSession::find at line 1187 in file "LdapOps.cpp"
 1187       numEntries = res->entries();
(dbx) n
t@16 (l@16) stopped in LdapSession::find at line 1189 in file "LdapOps.cpp"
 1189       if (numEntries == 1) {
(dbx) p numEntries
numEntries = 1
(dbx) n
...
(dbx) n
t@16 (l@16) stopped in LdapSession::find_userdn at line 1278 in file "LdapOps.cpp"
 1278       *dn = entry->DN();
(dbx) p *dn
*dn = 0x2d87b0 "uid=alpha,o=TestCentral"
(dbx)
...

Looking at LDAP server access logs it has created this "SRCH" entry :

[07/Oct/2009:14:55:51 +051800] conn=4 op=1 msgId=2 - SRCH base="o=testcentral" scope=2 filter="(uid=alpha)" attrs="c"
[07/Oct/2009:14:55:51 +051800] conn=4 op=1 msgId=2 - RESULT err=0 tag=101 nentries=1 etime=0

So at the end of find_userdn function, we get our userdn "uid=alpha,o=TestCentral". Coming back to debugger

(dbx) p userdn
userdn = 0x2d87b0 "uid=alpha,o=TestCentral"
(dbx)
t@16 (l@16) stopped in get_is_valid_password_ldap at line 531 in file "ldapacl.cpp"
  531                   rv = ld->userdn_password(userdn, raw_pw);
(dbx) s
...
(dbx) n
t@16 (l@16) stopped in LdapSession::userdn_password at line 1052 in file "LdapOps.cpp"
 1052       int msgid = ldap_simple_bind(session, userdn, password);
(dbx) p userdn
userdn = 0x2d87b0 "uid=alpha,o=TestCentral"
...
(dbx) n
t@16 (l@16) stopped in LdapSession::userdn_password at line 1058 in file "LdapOps.cpp"
 1058           rc = ldap_result(session, msgid, 0, &zerotime, &res );
(dbx) n
(dbx) p rc
rc = 97
...

As we can see Web Server tries to bind to LDAP server with userdn "uid=alpha,o=TestCentral" and we can confirm that by looking at LDAP server logs :

[07/Oct/2009:15:00:37 +051800] conn=4 op=2 msgId=3 - BIND dn="uid=alpha,o=TestCentral" method=128 version=3
[07/Oct/2009:15:00:37 +051800] conn=4 op=2 msgId=3 - RESULT err=0 tag=97 nentries=0 etime=0 dn="uid=alpha,o=testcentral"

Also refer Jyri's blog for troubleshooting

Enabling Client Certificate Authentication in Sun Web Server 7.0 reverse proxy server and origin server

For this I have setup two Sun Java System Web Server 7.0 update 6 instances. One acting as a reverse proxy and the other origin server that serves the request. Enabled SSL and client authentication on both these instances. Sent SSL requests with client certificates from a test client.

Installed Server Certificates in both Web Server instances

I created self signed certificate in reverse proxy server.

$cd <ws-install-dir>/https-test/config

$rm *.db

$../../bin/certutil -N -d .

$../../bin/certutil -S -d . -n Server-Cert-Reverse-Proxy-Server -s "CN=test.sun.com" -x -t "CT,CT,CT"

$../../bin/certutil -L -d .

Certificate Nickname                                         Trust Attributes

                                                             SSL,S/MIME,JAR/XPI
Server-Cert-Reverse-Proxy-Server                             CTu,Cu,Cu

 You can use Admin Server CLI to create a self signed certificate

wadm>create-selfsigned-cert --config=test.sun.com --server-name=test.sun.com --nickname=Server-Cert-Reverse-Proxy-Server

For convenience I copied these *.db to origin server instance config directory, and hence used the same server certificate in origin server. In real world use a new server certificate for origin server.

Changes made in Reverse Proxy Web Server Instance

In obj.conf I have confiugured reverse proxy in such a way that all requests are redirected to origin server. In real world situation you can if you want redirect only certain requests depending on your requirements.

Run create-reverse-proxy CLI from Administration server

wadm>create-reverse-proxy --config test --vs test --uri-prefix=/ --server=https://test.sun.com:4444

*obj.conf gets modified as shown below :

<Object name="default">

AuthTrans fn="match-browser" browser="*MSIE*" ssl-unclean-shutdown="true"

NameTrans fn="map" from="/" name="reverse-proxy-/" to="/"

...

</Object>

<Object ppath="*">

Service fn="proxy-retrieve" method="*"

</Object>

<Object name="reverse-proxy-/">

Route fn="set-origin-server" server="https://test.sun.com:4444"

</Object>

...

Enable "ssl" in http-listener, added server certificate nickname (if its different from "Server-Cert") and set client authentication as "required" :

 You can use Admin Server CLI to do these steps 

1. Enable SSL for this listener and set the server certificate
   wadm> set-ssl-prop --config=test.sun.com --http-listener=http-listener-1 server-cert-nickname=Server-Cert-Reverse-Proxy-Server enabled=true client-auth=required
2. Deploy the changes
   wadm> deploy-config test.sun.com

  <http-listener>

    <name>http-listener-1</name>

    <port>3333</port>

    <server-name>test.sun.com</server-name>

    <default-virtual-server-name>test</default-virtual-server-name>

   <ssl>
       <server-cert-nickname>Server-Cert-Reverse-Proxy-Server</server-cert-nickname>

       <client-auth>required</client-auth>
   </ssl>

  </http-listener>

In server.xml I have also modified access log format so that we can see what is happening. This will slow down Web Server performance so do not  do this in production environment.

<access-log>

    <file>../logs/access</file>

    <format>%Ses->client.ip% "%Req->reqpb.clf-request%" %Req->srvhdrs.clf-status% %Req->srvhdrs.content-length% %Ses->client.cipher% %Req->vars.auth-cert% </format>

  </access-log>

Changes made in Origin Server Web Server Instance

Enable "ssl"  in http-listener, added server certificate nickname (if its different from "Server-Cert") and set client authentication as "required" :

  <http-listener>

    <name>http-listener-1</name>

    <port>4444</port>

    <server-name>test.sun.com</server-name>

    <ssl>

        <server-cert-nickname>Server-Cert-Reverse-Proxy-Server</server-cert-nickname>

        <client-auth>required</client-auth>

    </ssl>

    <default-virtual-server-name>test</default-virtual-server-name>

  </http-listener>

In server.xml I have also modified access log format so that we can see what is happening. This will slow down Web Server performance so do not  do this in production environment.

 <access-log>
    <file>../logs/access</file>
    <format>%Ses->client.ip% "%Req->reqpb.clf-request%" %Req->srvhdrs.clf-status% %Req->srvhdrs.content-length% %Ses->client.cipher% %Req->vars.auth-cert% %Req->headers.proxy-auth-cert%</format>
  </access-log>

Created a test.html file that should be served when the client requests for it :

$cat ../docs/test.html

This is test.html on origin server

Test Client I used

I used "tstclnt" which is bundled with NSS-NSPR binaries in bin directory. Note that I used the server certificate of reverse proxy server as client certificate because I am too lazy to create more certificates. In real world use a proper client certificate.

Created a test request file :

$cat > sslreq.dat 

GET /test.html HTTP/1.0^M

^M

Sent request to the reverse proxy server

$tstclnt -h test.sun.com -p 3333  -n Server-Cert-Reverse-Proxy-Server -d https-test/config -c n -v -o -f < sslreq.dat

tstclnt: connecting to test.sun.com:3333 (address=xxx.xxx.xxx.xxx)

...

tstclnt: stdin read 27 bytes

tstclnt: Writing 27 bytes to server

tstclnt: SSL version 3.1 using 128-bit RC4 with 160-bit SHA1 MAC

tstclnt: Server Auth: 1024-bit RSA, Key Exchange: 1024-bit RSA

subject DN: CN=test.sun.com

issuer  DN: CN=test.sun.com

...

tstclnt: Read from server 350 bytes

HTTP/1.1 200 OK

Server: Sun-Java-System-Web-Server/7.0

Date: Fri, 18 Sep 2009 10:59:13 GMT

Content-type: text/html

Last-modified: Thu, 17 Sep 2009 10:44:10 GMT

Content-length: 35

Etag: "23-4ab212fa"

Accept-ranges: bytes

Via: 1.1 https-test

Proxy-agent: Sun-Java-System-Web-Server/7.0

Connection: close

This is test.html on origin server

...

tstclnt: exiting with return code 0

You can see the request is being served from origin server.

I have used in this test client cipher "n" which is SSL3 RSA WITH RC4 128 SHA because I know this cipher that is enabled in Sun Web Server 7.0 update 6. I can see that at the time of server startup when run in <log-level>finest</log-level>. You can use other ciphers as well.

Access log of Reverse Proxy Web Server shows the certificate

$cat ../logs/access
format=%Ses->client.ip% "%Req->reqpb.clf-request%" 
%Req->srvhdrs.clf-status% %Req->srvhdrs.content-length%  
%Ses->client.cipher% 
%Req->vars.auth-cert%

xxx.xxx.xxx.xxx "GET /test.html HTTP/1.0" 
200 35  
RC4 
MIIBujCCASOgAwIBAgIFAI566gYwDQYJKo...fBqhD710VkFmOScYjWBxZe1vhnTbu/NexX4NqLsZG9So=

Note cipher is RC4.

Access log of origin server shows the certificate

$cat ../logs/access

format=%Ses->client.ip% "%Req->reqpb.clf-request%" 
%Req->srvhdrs.clf-status% %Req->srvhdrs.content-length%  
%Ses->client.cipher% 
%Req->vars.auth-cert%
%Req->headers.proxy-auth-cert%

xxx.xxx.xxx.xxx "GET /test.html HTTP/1.1" 
200 35 
RC4 
MIIBujCCASOgAwIBAgIFAI566gYwDQYJKo...fBqhD710VkFmOScYjWBxZe1vhnTbu/NexX4NqLsZG9So=
MIIBujCCASOgAwIBAgIFAI566gYwDQYJKo...fBqhD710VkFmOScYjWBxZe1vhnTbu/NexX4NqLsZG9So=

Note that as expected in origin-server, rq->headers pblock has the certificate in "proxy-auth-cert". Reverse proxy server sends the certificate to origin server in this header.

References

• Sun Java System Web Server 7.0 Update 6 Administrator's Configuration File Reference - "forward-auth-cert"
• Configuring Reverse Proxy in Sun Java System Web Server 7.0
• Configuring reverse proxy in Sun Java System Web Server 7.0 when origin server is SSL enabled
• About trust flags of certificates in NSS database that can be modified by certutil
• http://forums.sun.com/thread.jspa?threadID=5397719
• http://forums.sun.com/thread.jspa?threadID=5373182
• http://forums.sun.com/thread.jspa?threadID=5359313

I tried building Open Web Server on OpenSolaris SPARC.

Install package as given in http://wikis.sun.com/display/wsFOSS/Checkout+and+Build+Instructions

SUNWmozldap, SUNWxercesc, SUNWxalanc in http://src.opensolaris.org/source/xref/webstack/ were probably not built for 64 bit. I have only tested on OpenSolaris x86 32 bit. They do not work on SPARC yet. 

Building Mozilla LDAP C SDK for 64 bit

Run this script it builds and puts Mozilla LDAP C SDK and puts it in /usr/local/include/mozldap/, /usr/local/lib/mozldap, /usr/local/lib/mozldap/64

#!/bin/sh 
#building 32 bit 
cd
cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co -P -rLDAPCSDK_6_0_5_RTM DirectorySDKSourceC

cd mozilla/directory/c-sdk

./configure --with-sasl --with-nspr-inc=/usr/include/mps --with-nspr-lib=/usr/lib/mps --with-nspr --with-nss-inc=/usr/include/mps --with-nss-lib=/usr/lib/mps --with-nss

gmake

cd ../../dist/

sudo mkdir -p /usr/local/include/mozldap /usr/local/lib/mozldap

sudo cp public/ldap/* /usr/local/include/mozldap/

sudo cp lib/* /usr/local/lib/mozldap/

cd 

mv mozilla mozilla32
#building 64 bit

cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co -P -rLDAPCSDK_6_0_5_RTM DirectorySDKSourceC

cd mozilla/directory/c-sdk

./configure --with-sasl --with-nspr-inc=/usr/include/mps --with-nspr-lib=/usr/lib/mps/64 --with-nspr --with-nss-inc=/usr/include/mps --with-nss-lib=/usr/lib/mps/64 --with-nss --enable-64bit

gmake

cd ../../dist/

sudo mkdir -p /usr/local/lib/mozldap/64

sudo cp lib/* /usr/local/lib/mozldap/64

cd 

mv mozilla mozilla64

Building Xerces and Xalanc for 64 bit

Build Xerces C and Xalan C 32 bit and 64 bit and put it in /usr/local/ area.

Build Xerces C 2.6 from http://xerces.apache.org/xerces-c/build-2.html

Build Xalan C  1.9 from http://xml.apache.org/xalan-c/build_instruct.html

Check Out Open Web Server

hg clone ssh://anon@hg.opensolaris.org/hg/webstack/webserver

Patch these diffs

Building Open Web Server

gmake BUILD_VARIANT=OPTIMIZED
If you want 64 bit support also

gmake BUILD_VARIANT=OPTIMIZED BUILD64=1

Installing Open Web Server

gmake BUILD_VARIANT=OPTIMIZED install

If you want 64 bit also

gmake BUILD_VARIANT=OPTIMIZED install BUILD64=1

 Go to work/B1/*/https-test/config directory. To run Open Web Server in 64 bit you need to add <platform>64</platfrom> in server.xml "server" element.

If you get error that looks like

ld: fatal: file /usr/lib/64/libsqlite3.so: version `SQLITE_3' does not exist:
        required by file /usr/lib/mps/64/libsoftokn3.so
$mv /usr/lib/64/libsqlite3.so /usr/lib/64/libsqlite3.so.BACK

References

http://wikis.sun.com/display/wsFOSS/Checkout+and+Build+Instructions

http://forums.sun.com/thread.jspa?messageID=10810336#10810336

Sun Web Server
is the secure web serving platform of choice for large-scale
enterprises world wide. Published by Sun Microsystems and Pearson Education (Prentice Hall), here comes the authoritative, comprehensive guide to Sun
Web Server 7.0, by Bill Nelson, Arvi Srinivasan and myself. We are
indebted to Bill Nelson - chief instructor training Web Servers customers worldwide - who authored significant portion of
the book. Arvi - one of web tier architects - filled in several valuable
sections too. Of course, we are proud to have the foreword by Scott
G. McNealy!

• Get it via amazon.com: http://bit.ly/1K6dOH


• Preview at Safari Books Online: http://my.safaribooksonline.com/9780137153329


• Visit The Essentials Guide web site: http://sunwebserver.com/
  

Here's a quick snap of my complimentary copy that arrived the other day.

Is it that time of the year again? I guess so!

Back in February I posted my (sort-of) biannual review of SFW build times so it has been six months. The SFW build continue to chug along towards collapsing under its own weight as I predicted two years ago. I can't claim much in the way of visionary powers for the observation since it is a rather obvious outcome of consolidating all sources into one tree. Unfortunately not obvious enough though since the practice still continues!

To review, refer to my original article on unconsolidating which exposes the problem with the Solaris build concept of building all applications together in one single source tree. I updated the data in June 2008 and later in February 2009.

Aside from the build times nothing has really changed so no other news. If you haven't read the previous articles check them out since I won't repeat the data here.

As of this month, SFW build produces 416 packages, takes 9.7 hours to build and a built workspace takes 19.4GB!

The time/space predictions for 5000 packages are within the range previously seen (current one in bold in above table: 116 hours and 233GB) so no big surprises.

Well the big surprise is we're still building OpenSolaris applications this way!

For the Web Stack project and components we are now looking into dropping out of SFW since this is unsustainable and is consuming too much of our limited resources. Hopefully my next biannual SFW update will be that there isn't one! ;-) Time will tell...

Q: How many requests per second can the Web Server handle?

Short answer: It depends.

Long answer: It really depends on many factors.

Ok, ok.. sillyness aside, can we make any ballpark estimates?

The Web Server can be modeled as a queue. By necessity such modeling will be a simplification at best, but it may provide a useful mental model to visualize request processing inside the server.

Let's assume your web application has a fairly constant processing time^[1], so we'll model the Web Server as a M/D/c queue where c is the number of worker threads. In this scenario, the Web Server has a maximum sustainable throughput of c / (processing time).

To use some simple numbers, let's say your web app takes 1 second to process a request (that's a very slow web application!). If the Web Server has c=128 worker threads, that means it can indefinitely sustain a max request rate of:

128/1 = 128 requests per second

This makes a lot of sense if we think about it:

• At t = 0 seconds, 128 request come in and each one is taken by a worker thread, fully utilizing server capacity.
• At t = 1 second, all those requests complete and responses are sent back to the client and at the same time 128 new requests come in and the cycle repeats.

At this request rate we don't need a connection queue at all^[2] because all requests go straight to a worker thread. This also means that at this request rate the response time experienced by the end user is always 1 second.

To expand on that, the response time experienced by the end user is:

end user response time = (connection queue wait time) + (processing time)

Since we're not using the connection queue the end user response time is simply the same as the processing time^[3].

So far so good. Now, what happens if the incoming request rate exceeds the maximum sustainable throughput?

• At t = 10 seconds, 129 requests come in. 128 go straight to worker threads, 1 sits in wait in the connection queue.
• At t = 11 seconds, 128 requests come in. 128 (the one which was waiting + 127 of the new ones) go straight to worker threads, 1 sits in wait in the connection queue.

The connection queue absorbs the bumps in the incoming request rate, so connections are not dropped and worker threads can remain fully utilized at all times. Notice that now out of every 128 requests, one of them will have a response time of 2 seconds.

So what happens next?

If we go back to receiving a steady 128 requests per second, there will always be one requests in the connection queue.

If at some point we only receive 127 requests (or less), the server can "catch up" and the connection queue goes back to staying empty.

On the other hand, if the incoming request rate remains at 129 per second we're in trouble! Every second the connection queue waiting list will grow longer by one. When it reaches 129 entries, one end user will experience a response time of three seconds, and so on.

And of course, the connection queue is not infinite. If the max connection queue size is 4096 then 4096 seconds later it will fill up and from that point onwards, one incoming request will simply be dropped every second since it has no place to go. At this point the server has reached a steady state. It continues processing requests at the same rate as always (128 per second), it continues accepting 128 of the 129 new requests per second and dropping one. End users are certainly unhappy by now because they are experiencing response times of over 30 seconds (4096 / 128 = 32, so it takes 32 seconds for a new request to work its way through the queue. Almost like going to the DMV...

If the incoming request rate drops below the maximum sustainable rate (here, 128/sec) only then can the server start to catch up and eventually clear the queue.

In summary, while this is certainly a greatly simplified model of the request queue behavior, I hope it helps visualize what goes on as request rates go up and down.

Theory aside, what can you do to tune the web server?

• The single best thing to do, if possible, is to make the web app respond quicker!
• If you want to avoid dropped connections at all cost, you can increase the connection queue size. This will delay the point where the server reaches a steady state and starts dropping connections. Whether this is useful really depends on the distribution of the incoming requests. In the example above we've been assuming a very steady incoming rate just above the maximum throughput rate. In such a scenario increasing the connection queue isn't going to help in practice because no matter how large you make it, it will fill up at some point. On the other hand, if the incoming request rate is very bumpy, you can damp it by using a connection queue large enough to avoid dropping connections. However... consider the response times as well. In the example above your end user is already seeing 33 second response times. Increasing the connection queue length will prevent dropped connections but will only make the response times even longer. At some point the user is simply going to give up so increasing the connection queue any further won't help!
• Another option is to increase the number of worker threads. Whether this will help or hurt depends entirely on the application. If the request processing is CPU bound then it won't help (actually, if it were truly CPU bound, which is rare, then you'll probably benefit from reducing the number of worker threads unless your server has 128+ CPUs/cores...) If the web app spends most of its time just waiting for I/O then increasing the worker threads may help. No set answer here, you need to measure your application under load to see.

[1] In reality the response time can't be deterministic. At best it may be more or less constant up to the point where the server scales linearly but after that the response time is going to increase depending on load. On the flip side, cacheing might make some responses faster than expected. So M/D/c is certainly a simplification.

[2] Not true for several reasons, but it'll do for this simplified model and it helps to visualize it that way.

[3] Plus network transmission times but since we're modeling only the web server internals let's ignore that.

Coincidentally last week I heard a couple related queries about check-request-limits from different customers. I haven't covered that feature in a while so it's a good time to revisit it for a bit.

To review, Web Server 7 has a feature (function) called check-request-limits which can be used to monitor and limit the request rate and/or concurrency of request which match some criteria. It can be used to address denial of service attacks as well as just to limit request rates to some objects or from some clients for other reasons (for example to reduce bandwidth or cpu usage).

I usually refer to 'matching requests' when speaking of this capability. Matching what? Probably the most common use case is to match the client IP address. This is useful when you wish to limit request rates coming from a given client machine. Here's a basic example of that scenario:

PathCheck fn="check-request-limits" max-rps="10" monitor="$ip"

The common theme to both customer requests I heard last week was whether it is possible to limit requests based on something other than the client IP?

Yes, certainly!

The monitor parameter above is set to "$ip" which expands to the client IP address but you can set it to anything that you prefer. In my introduction to check-request-limits article I gave examples of both "$ip" and "$uri" (and even both combined). You're not restricted to only these though, you can use any of the server variables available in WS7 as the monitor value.

You can also construct more complicated scenarios using the If expressions of Web Server 7. I gave a few examples of that in this article on check-request-limits.

To give a couple more examples, let's say your web server is behind a proxy and this the client $ip is always the same (the proxy IP). Clearly monitoring the $ip value isn't terribly useful in that case. Depending on how your application works you may be able to find other useful entries to monitor. For example if the requests contain a custom header named "Usernum" which contains a unique user number, you could monitor that:

PathCheck fn="check-request-limits" max-rps="1" monitor="$headers{'usernum'}"

Or maybe there's a cookie named customer which can serve as the monitor key:

PathCheck fn="check-request-limits" max-rps="1" monitor="$cookie{'customer'}" 

These two are made-up examples, you'll need to pick a monitor value which is suitable for your application. But I hope these ideas will help you get started.

By the way check-request-limits can also be used to limit concurrency.

If you are one of those geeky folks using Sun Web Server (also known as Sun Java System Web Server , Sun ONE Web Server, iPlanet Web Server, Netscape Enterprise Server - Hmm... lot of incarnations) in your enterprise server with multiple cluster deployment or simply running a secure web server from your garage, you should be pleased to know that Amazon is now carrying this title  Sun Web Server: The Essential Guide . Ah, you know, what is more cool - Amazon Kindle carries this title as well. If you are one of those lucky few having a Kindle, you can get this instantly (ok, not that instant - probably in under 2 minutes) from here .

We love to hear what you think of this book. Please feel free to write to us at our Forum

Last week I wrote about the time spent dealing with email. While writing that entry I though it'd be nice to also visualize where all the time went, not just how much was spent on email. So tonight I went over the data to generate the following pie chart, showing relative allocation of working hours from early May until today, split into high level categories:

The 'Email' slice is self evident.. 'ARC' is the time I've spent in my role in Sun's Architecture Review Committee. 'Communications' includes conferences, presentation, blog entries, articles and other related work. 'Administrivia' is a catch-all category for all kinds of mindless unproductive overhead. Finally, 'Engineering' represents the time spent doing "real work".

About the only thing I can add is that this is about as concise a representation as we can get on why very large companies have trouble competing with agile startups. Part of my goal in this exercise is to find ways to grow that nice blue pie slice, but I realize there's a limit to what can be achieved in this environment. All those TPS^H^H^HPTL reports needs to be filed, after all.

About two months ago I posted on attempting to keep email in check so it's a good time to review some statistics and results...

The following graph shows the percentage of time I spent reading email each day:

The average over the past three months is about 45% Wow.. So over the last quarter I've spent just under half of all working hours reading (and answering) email. No wonder it is hard to get concrete work done!

This is somewhat higher than the 37.5% (three hours a day out of eight) that I had predicted in the previous article a couple months ago. This is largely explained due to the recent release of Web Stack 1.5. Due to the impending release I found myself having to check email more often than scheduled to keep on top of last minute pre-release activities.

A few points worth noting out of the experiment so far...

• It is not easy to limit email activity to the scheduled two or three hours a day. Ideally the graph above should be mostly flat. While part of this is inevitably due to the release activities, I'll try harder going forward to stick to the scheduled email times.
• While the total times may have fluctuated more than I wanted, I did (mostly) manage to contain my email activities to bounded windows of time within the day, instead of checking emails every three minutes all day long. This has helped a great deal. Even while spending nearly half my hours on email, I've managed to get many other non-email tasks done more productively than before. This part has been a success and I highly recommend it. Shut down that email client!
• I found myself doing three (or even four) email sessions per day. This is too many. I need to more strictly limit myself to reading email only twice a day, at the beginning and end of the day. If these sessions need to be longer it is better to make them longer but stick with only two. Whenever I started inserting email tasks in the middle of the day, it fragmented my concentration too much, making the day less productive.
• I'm convinced the ideal arrangement is to do one single email session per day, at the end of the day. That way all the concentration disruption occurs after the days work is done, so it does no harm. The end of the day is also a good time to be entering new tasks into the to-do list so they'll be there tomorrow. Given our distributed time zones it is difficult to do only one email session per day, but that would be ideal. Maybe I'll try that at some point.

As a longer term goal I need to think of ways of reducing the time spent on email. Not sure how to do that yet but spending 45% or even "only" 37% of all working hours on email is totally insane. I suppose email overload is inevitable at a large company with tens of thousands of employees (all of whom, it seems at times, are emailing me) but there has to be a better way. I suppose I could cap my email time to an hour a day and let whatever goes unread just go unread. I'm sure people will be unhappy but will that unhappiness be greater than my productivity gain at doing real work? It's all about tradeoffs, after all. Hard to say what's worse.

While Sun's Web Server has a very nice threading model, once a worker thread is processing a specific request it will continue working on that request even if it takes a while or blocks.

This is rarely an issue. Static content is served very quickly and code which generates dynamic application content needs to be written so it responds promptly. If the application code takes a long time to generate response data the site has more problems than one, so the web application developers have a motivation to keep it snappy.

But what if you do have a bad application which occasionally does take a long time? As requests come in and worker thread go off to process them, each long-running request ties up another worker thread. If requests are coming in faster than the application code can process them, eventually the Web Server will have all its worker threads busy on existing connections.

As you can infer from Basant's blog entry, the server will still continue accepting new connections because the acceptor thread(s) are separate from the worker threads, so it is still accepting new connections. But there won't be any spare worker threads to take that new connection from the connection queue.

If you're the client making the request, you'll experience the server accepting your request but it won't answer for a (possibly long) while. Specifically, until one of the previous long-running requests finally completes and a worker thread frees up to take on your request (and of course, there may be many other pending requests piled up).

If this is happening with your application one option is to check the perfdump output and see which requests are taking a while. But, as these things are bound to do, it'll probably happen sporadically and never when you're watching.

So how can we easily gather a bit more info? It's been said countless times but always worth repeating.. dtrace really is the greatest thing since sliced bread (and I like bread). I can't imagine attempting to maintain a system without dtrace in this day and age, it would be limiting beyond belief! One of the many key benefits is being able to gather arbitrary data right from the production machine without any prior preparation (such as producing debug builds) or downtime or even any access to the sources you're diagnosing.

So in that spirit, I tried to gather a bit more data about the requests which appear to be taking a while using dtrace and without attempting to look at what the code is actually doing (well, also because I only had fairly limited time to dedicate to this experiment so didn't want to go looking at the code ;-). Although, I should mention, since Sun's Web Server is open source you certainly could go review the source code if you wish to know more detail.

So what am I looking for? Basically I'd like to know when the worker thread starts on a request and when it is done with it. If the time between those two grows "too long", I'd like to see what's going on. Sounds simple enough. Searching around a bit I saw Basant's article on dtrace and Web Server so using his pid$1::flex_log:entry as an exit point seems like a suitable thing to try. I didn't find (on a superficial search, anyway) a mention of an adequate entry point so instead I took a number of pstack snapshots and looked for something useful there and wound up selecting "pid$1::__1cLHttpRequestNHandleRequest6MpnGnetbuf_I_i_:entry" (ugly mangled C++ function name). With that, ran the following dtrace script on the Web Server process:

% cat log.d
#!/usr/sbin/dtrace -qs

pid$1::__1cLHttpRequestNHandleRequest6MpnGnetbuf_I_i_:entry
{
  self->begin = timestamp;
  printf("ENTER %d, %d to n\n", tid, self->begin);
}

pid$1::flex_log:entry
/self->begin/
{
  self->end = timestamp;
  printf("DONE %d, %d to %d\n", tid, self->begin, self->end);
  self->begin = 0;
}

This gets me entry/exit tick marks as the threads work their way through requests. On a mostly unloaded server it's easy enough to just watch that output, but then you're probably not experiencing this problem on an unloaded server. So we need a little bit of helper code to track things for us. Twenty minutes of perl later, I have

#!/usr/bin/perl

$PATIENCE = 9;                  # seconds - how long until complains start

$pid = shift @ARGV;
$now = 0;
$npat = $PATIENCE * 1000000000;

open(DD, "./log.d $pid |");
while (<DD>)
{
    chomp;
    ($op, $tid, $t1, $t2) = /(\S*) (\d*), (\d*) to (.*)/;
    if ($t1 > $now) { $now = $t1; }

    # dtrace output can be out of order so include start time in hash key
    $key = "$tid:$t1";          

    if ($op eq "ENTER") {
        if ($pending{$key} != -1) {
            $pending{$key} = $t1 + $npat; # value is deadline time
        }

    } else {
        $took = (($t2 - $t1) / 1000000000);
        if (!$pending{$key}) {
            $pending{$key} = -1; # if DONE seen before ENTER, just ignore it
        } else {
            delete $pending{$key};
        }
    }

    # Once a second, review which threads have been working too long
    # and do a pstack on those.
    # 
    # Note: we only reach here after processing each line of log.d output
    # so if there isn't any more log.d output activity we'll never get here.
    # A more robust implementation is left as an exercise to the reader.
    #
    if ($now > $nextlook) {
        $c = 0;
        foreach $k (keys %pending)
        {
            if ($pending{$k} != -1 && $pending{$k} < $now) {
                ($tid, $started) = $k =~ /(\d*):(\d*)/;
                $pastdue = ($now - $started) / 1000000000;
                print "=================================================\n";
                system("date");
                print "Thread $tid has been at it $pastdue seconds\n";
                system("pstack $pid/$tid");
                $c++;
            }
        }
        if ($c) { print "\n"; }
        $nextlook = $now + 1000000000;
    }
    
}

The perl code keeps track of the ENTER/DONE ticks (which may occasionally be out of order) and if too long (more than $PATIENCE) goes by, gives you pstack output what's going on.

I don't actually have a suitably misbehaving application so I'll leave it at that. If I had a real application issue, it'd be useful to fine tune the dtrace script to key off of more specific entry and exit points and it'd also be useful to trigger more app-specific data gathering instead of (or in addition to) the pstack call (for instance, checking database availability if you suspect a database response problem, or whatever is suitable for your concrete application).

dtrace is like lego blocks, there's a thousand and one ways of coming up with something similar. Care to try an alternative or more efficient approach? Please share it in the Web Server forum!

Sun Web Stack installation, registration, and the Enterprise Manager features are localized to several locales including Simplified Chinese, Japanese, French, and German. Please refer to Web Stack 1.5 release notes in various locales and the following blog by Masaki Katakai shows Enterprise Manager in Japanese.

Sun Startup Essentials Program is organizing a webinar on web stack next

When: Tuesday August 11th 8AM Pacific Standard Time.
http://blogs.sun.com/startups/entry/free_startup_webinar_architecting_your

You are welcome, registestration for this event is free.

Jython Web Application on Sun Java System Web Server 7.0

Download Jython from http://softlayer.dl.sourceforge.net/project/jython/jython/2.5.0/jython_installer-2.5.0.jar

Run Jython installer and install it in lets say in /opt/jython/jython2.5.0

$java -jar jython_installer-2.5.0.jar

Lets try deploying a simple demo web application that comes along with this. This demo application has a simple demo_app.py script.

cd /opt/jython/jython2.5.0/Demo/modjy_webapp/WEB-INF
cp /opt/jython/jython2.5.0/jython.jar lib/

In web.xml change python.home i.e.
<init-param>
  <param-name>python.home</param-name>
  <param-value>C:/jython2.5</param-value>
</init-param>
to
<init-param>
   <param-name>python.home</param-name>
   <param-value>/opt/jython/jython2.5.0</param-value>
</init-param>

cd /ws7-install-dir/https-instance-name/config

Add in server.xml in virtual server element, the location of this web application

<web-app> 
    <uri>/modjy_webapp</uri> 
    <path>/opt/jython/jython2.5.0/modjy_webapp</path>
</web-app>

Start the web server instance, and access the web application via browser using URI http://host:port/modjy_webap

you should see a page starting with the contents:

Python CGIs in Sun Java System Web Server 7.0

Add this test script in <ws7-install-dir>https-<instance-name>/docs/cgi-bin directory and call it hello.cgi

#!/usr/bin/env python

print("Content-Type: text/plain;charset=utf-8")

print("\r\n")

print("Hello World!")

Change obj.conf default object 

<Object name="default">

AuthTrans fn="match-browser" browser="*MSIE*" ssl-unclean-shutdown="true"

NameTrans fn="ntrans-j2ee" name="j2ee"

NameTrans fn="pfx2dir" from="/cgi-bin" dir="/<ws-install-dir>/https-<instance-name>/docs/cgi-bin" name="cgi"

...

<Object name="cgi">
ObjectType fn="force-type" type="magnus-internal/cgi"
Service fn="send-cgi"
</Object>

Send a request through the browser http://host:post/cgi-bin/hello.cgi

You can see Hello World! displayed.

References

http://docs.python.org/3.1/howto/webservers.html

Some time back, I wrote this blog on how to leverage the PHP runtime integrated within OpenSolaris and use it to run PHP applications within Sun Java System Web Server 7. Well, if you are currently using Sun Web Server 7 on Solaris 10 or RedHat / CentOS Linux and wondering if there is a more simplified way to get PHP working within Sun Web Server 7, then my today's topic might be of more use to you.

Installing Sun Web Server

Now, if you are new to Sun Web Server 7 and wonder why you need this beast, please do check out some of these  performance benchmark blogs within our wiki. You can download and install the latest version of Sun Java System Web Server from here .

If you are adventurous, you can also build the open sourced version of this beast (available under BSD license) by following the instructions provided here. 

If you would like a little hand holding in installing our product, check out this nice demo . Hope, you are able to successfully install Sun Web Server 7 with the help of this demo. If not, you can always post your questions in our forums - where we developers will be happy to help you out !.

Installing Sun GlassFish Web Stack PHP

If you haven't yet heard of Sun Web Stack or more confusingly known as Sun GlassFish Web Stack - is an open source project within OpenSolaris - known as  Web Stack project - offers free download and production support (as part of Sun GlassFish Portfolio) for open source web tier components like Apache Web Server, PHP,  MySQL, Tomcat, Hudson etc on Solaris 10 Update 5 (or above) and RedHat Enterprise Linux 5.2 (or above).

Here is a quick way to get Web Stack PHP installed on your Solaris 10 or RHEL 5. If you prefer a little hand holding, please refer to my earlier blog on the different ways to install Sun Web Stack (along with some screenshots to guide you along the way).

Here is a quick walk through on how to get your PHP runtime installed

• Visit our Sun Web Stack 1.5 download web page and download 'click on IPS based Packaging' link for your respective OS.
  

• Now, you will need to unzip the downloaded tar ball and either fire up the update tool GUI for installing PHP runtime (as shown in this blog) or simply install it in the command line as shown below:
  

Note:  Unlike OS native packages, IPS based installation (that you just did here) does not verify or warn if you do not have installed any underlying dependent libraries that Web Stack 1.5 stack requires. So, we strongly recommend you to visit our System Package dependency link to ensure that you have these packages installed on your system.

How to get Web Stack PHP working within Sun Web Server 7 ?

Ok, Assuming you have successfully installed Web Server 7 and Sun Web Stack PHP on your system, here is a simple one liner configuration script that you can run to successfully configure Web Server 7 to execute PHP scripts .

Note: On Linux, because of a bug with the above script, please download this script and overwrite it over the default.

cp /tmp/configure-sun-webserver ~/webstack1.5/php/5.2/samples/scripts/

That is all there to it.

Now, you should be able to run php scripts under your web server's document root directory (which is typically <ws7-install-root>/https-<hostname>/docs/).

Note: In the above example, I have assumed that you will be installing both Web Server 7 and Web Stack PHP as the same user and you wanted to configure PHP runtime with the default instance (https-<hostname>) created by Web Server 7 installer. If in case, you would like to configure a separate instance for Web Server 7 then you could do the same either using Web Server 7 Administration CLI /GUI.  Here is a simple example of how to do so with CLI :

As you may have seen, Web Stack 1.5 is out. Go ahead and give it a try!

One of the slightly confusing parts of the Web Stack distribution is that it varies by platform so there are several ways of "giving it a try". So it might be worth summarizing how and where to get it:

If you are on OpenSolaris 2009.06

Web Stack 1.5 is integrated into OpenSolaris 2009.06 out of the box. There is nothing to download. Simply install your favorite Web Stack components via the pkg(5) command. One shortcut is to install the 'amp' metapackage which will bring in a number of AMP-related Web Stack components (check the amp manifest to see precisely which ones it includes).

% pfexec pkg install amp

Note the 'amp' metapackage doesn't install all Web Stack components and you can just as well install only the ones you need, individually.

Yes, I realize OpenSolaris 2009.06 shipped last month! That means Web Stack 1.5 has been available in OpenSolaris for over a month now. So what is this week's announcement all about?

If you are on Solaris 10 or RedHat Linux

Well, unfortunately our shipping dates are a bit out of sync on different platforms (this is something I want to get aligned for the next time around) so what we're announcing this week is the availability of Web Stack 1.5 components on Solaris 10 and RedHat Linux.

Download link for Web Stack 1.5 for Solaris 10 and RedHat Linux

There are two different packaging formats you may download for these platforms: native packages and update center images.

The native packages (svr4 for Solaris 10 and rpm for RedHat Linux) are what you'd expect, similar to the Web Stack 1.4 packages (and yes, you can upgrade your previous install if you wish).

The update center image is new in this release and it is quite interesting. Instead of downloading actual Web Stack components you only download an IPS user image which you may unzip anywhere you like, such as in your home directory. From within this user image you will then invoke the pkg(5) CLI to download and install those components you wish to use (there is also a GUI, updatetool, if you're into that).

Yes, this means you can install Web Stack components into any location you like while running as your regular nonprivileged (non-root) user. This is quite nice for experimenting and development work. Please refer to the README and documentation for details. Give it a try and let us know how you like it.

The other brand new cool thing in 1.5 is the Enterprise Manager GUI, which provides a really nice monitoring interface to component statistics.

Others from my team have written in more detail about various features so I won't repeat that here, just check these out:

• Brian has a release announcement.
• CVR also has a release announcement and some notes on the update center packaging.
• Sriram wrote about the release, how to install it and the Enterprise Manager.
• Irfan wrote about the Enterprise Manager and its navigation panel.
• Jeff wrote about Apache httpd updates in 1.5.

Finally, please note that while the full marketing name this quarter is Glassfish Web Stack, the Web Stack product is separate and completely unrelated to the Glassfish Application Server! I realize this has been the source of much confusion lately, particularly at OSCON last week.

Hopefully this helps clarify a bit how and where to obtain Web Stack 1.5! With that out of the way, now go give it a try!

One of the nice things about Web Stack 1.5 release we said was that it's fully reloctable - that it can be installed any number of locations on a given system, with our without superuser (root) privileges. Unlike our default native package distribution (SVR4 packages or the RPMs), this fully relocatable web stack distribution is built with a cross-platform version of the OpenSolaris image packaging sysem IPS/pkg(5). With this distribution, you can update the packages with ease, using the update tool, which is bundled with the distribution. With a sequence of screenshots in his blog, Jeff Trawick demonstrates how you could update Apache web server using this tool.

Please check it out and share your feedback on the Web Stack product forum.

Web Stack Enterprise Manager (screenshots below)
Fully relocatable Update Center ips(5) image distribution
Apache HTTP Server 2.2.11
lighttpd 1.4.21
memcached 1.2.5
MySQL 5.0.67 and 5.1
PHP 5.2.9
Python 2.6
Ruby 1.8.7
Squid 2.7.STABLE5
Tomcat 6.0.18
GlassFish v2.1
Sun Continuous Integration Server 1.312

Web Stack web site has more information including product data sheet and links to download and documentation:
http://sun.com/webstack
http://www.sun.com/software/webstack/get.jsp
http://wikis.sun.com/display/WebStack/Sun+GlassFish+Web+Stack+Documentation and the Web Stack product forum.

Cheers.
cvr
[1] Web Stack Enterprise Manager dashboard screenshots:

It's been so hectic here lately I actually forgot to blog about this before it happened but thanks to everyone who attended our ( CVR and myself) Web Stack session at OSCON 2009 this week.

As we mentioned, we should be announcing the availability of Web Stack 1.5 'real soon now', watch this space for the details...

Running Open Web Server on FreeBSD

Downloaded Free BSD from ftp://ftp.freebsd.org/pub/FreeBSD/releases/i386/ISO-IMAGES/7.2/7.2-RELEASE-i386-dvd1.iso.gz

and installed it.

For reference, see http://wikis.sun.com/display/wsFOSS/Checkout+and+Build+Instructions

To search for a package, I used cd /usr/ports; make search name="mercurial" and so on.

First make sure that you already have these components

Install/Build these Components

*Note  There is one /usr/ports/net/ldapsdk. It downloads and builds ldapsdk_12311998.tar.gz.

But I can see it in ancient now from mozilla site http://ftp.mozilla.org/pub/mozilla.org/directory/c-sdk/ancient/

So I built Mozilla C SDK myself.

Hack IN NSPR

One manual hack you need to do in "/usr/local/include/prinet.h" 

#if defined(FREEBSD) || defined(BSDI) || defined(QNX)
#include <rpc/types.h> /* the only place that defines INADDR_LOOPBACK */
#endif

replace it by

#ifndef INADDR_LOOPBACK
#define INADDR_LOOPBACK         (u_long)0x7F000001
#endif

Building Open Web Server

• /usr/local/bin/hg clone ssh://anon@hg.opensolaris.org/hg/webstack/webserver
• cd webserver
• DOWNLOAD AND PATCH THESE DIFFS
• gmake BUILD_VARIANT=OPTIMIZED
  
• gmake BUILD_VARIANT=OPTIMIZED install

Go to work/FreeBSD7.2-RELEASE_OPT.OBJ/https-test/bin/ directory and start the server instance using startserv script.

For the last two gmake commands, you can add WS_INSTALL_ROOT=/opt/ws if you want the server instance to be installed in /opt/ws/https-test

References:

• http://wikis.sun.com/display/wsFOSS/Checkout+and+Build+Instructions
• http://www.freebsd.org/cgi/cvsweb.cgi/ports/textproc/xerces-c2-devel/files/patch-src-xercesc-Makefile.incl?rev=1.2;content-type=text%2Fplain
• https://bugzilla.mozilla.org/show_bug.cgi?id=505418
• https://bugzilla.mozilla.org/show_bug.cgi?id=504248
• http://www.freebsd.org/cgi/query-pr.cgi?pr=136984
  

Running Open Web Server on MacOS 

I tried to run Open Web Server on iMac with Mac OS X 10.5.7 on it.

Install http://svn.macports.org/repository/macports/downloads/MacPorts-1.7.1/MacPorts-1.7.1-10.5-Leopard.dmg as per http://www.macports.org/install.php

Already existing Components

Install Mercurial, NSS, NSPR, Xerces C++, Xalan C++, PCRE, ICU

sudo port install mercurial nspr nss pcre icu xercesc xalanc

Building and Installing Mozilla LDAP C SDK

Building Open Web Server

Note that

I am using "_xpatomic_locked.h" for now. Their performance needs to be optimized using functions in /usr/include/libkern/OSAtomic.h.
NSPR's PR_GetLibraryFilePathname dumps if called with NULL as first argument on MAC so added a hack of PR_GetLibraryFilePathname("ns-httpd40"..).
Some filenames are ugly like defines___.mk will fix it later.

If you see entries in error log like :

failure : HTTP3360: connection limit (1) exceeded.
PollManager::RequestReservation() keep-alive subsystem full

fine: operating system file descriptor limit is 256
fine: allocating 1 file descriptors to the connection queue, 1 file descriptors to keep-alive connections, and 1 file descriptors to the file cache
fine: 130 connection maximum (1 queued, 128 active, 1 keep-alive)
fine: HTTP3066: HTTP keep-alive subsystem will accomodate up to 1 connections

$ulimit -a
open files                      (-n) 256
...

$ulimit -n 1024

$./https-test/bin/startserv
fine: operating system file descriptor limit is 1024
fine: allocating 128 file descriptors to the connection queue, 128 file descriptors to keep-alive connections, and 8 file descriptors to the file cache
fine: 384 connection maximum (128 queued, 128 active, 128 keep-alive)
fine: HTTP3066: HTTP keep-alive subsystem will accomodate up to 128 connections

References

1. Build instructions for Open Web Server
2. https://bugzilla.mozilla.org/show_bug.cgi?id=504893
   
3. http://developer.apple.com/documentation/Darwin/Reference/ManPages/man3/barrier.3.html
   
4. http://devworld.apple.com/technotes/tn2002/tn2071.html
5. http://lists.apple.com/archives/unix-porting/2002/Sep/msg00021.html
6. http://dev.eclipse.org/newslists/news.eclipse.tools.cdt/msg16863.html
7. http://blogs.sun.com/Janice/entry/http3360_connection_limit_1_exceeded
   

Installing Sun Java System Web Server 7.0 on CentOS 5.3 or Fedora 10/11 or Ubuntu

In my free time, I just tried out Sun Java System Web Server 7.0 update 5 on CentOS 5.3 and Fedora10 and 11 and Ubuntu. Joe has also written a similar blog on this

Download Sun Java System Web Server 7.0 for Linux from http://www.sun.com/download/index.jsp?cat=Web%20%26%20Proxy%20Servers&tab=3&subcat=Web%20Servers and extract the contents of tar.gz file

$gunzip  sjsws-7_0u5-linux-i586.tar.gz; tar -xvf sjsws-7_0u5-linux-i586.tar

Run installer, it will fail

$./setup

error while loading shared libraries: libstdc++.so.5: cannot open shared object file: No such file or directory

CentOS

$sudo yum -y install compat-libstdc++-33

Now run setup, it will work fine.

Fedora 10/11

$sudo yum -y install compat-libstdc++-33

There is one more problem in Fedora 11 but is fixed in 7.0 update 6.

If you get error message like

lib/libfreebl3.so: version `NSSRAWHASH_3.12.3' not found (required by /lib64/libcrypt.so.1)

You need workaround as given in http://forums.sun.com/thread.jspa?messageID=10769043#10769043

Ubuntu

You need to install libstdc++5 as shown below

$sudo apt-get install libstdc++5

If you get an error which has something that looks like /bin/domainname not found, you need to install

$sudo apt-get install nis

I have tested on Ubuntu 9.04.

*Note that Sun Java System Web Server 7.0 update 5 or 6 is not officially certified or supported on CentOS or Fedora or Ubuntu, but here's how you can make it work.

References

http://wikis.sun.com/display/WebServer/Installing+on+Ubuntu

http://jmccabe.org/blog/CentOS_WebServer_Install

http://blogs.sun.com/kkranz/entry/installing_sun_java_system_web

http://ubuntuforums.org/showthread.php?t=855603

Managing Certificate Trust flags in NSS Database

We can modify certificate trust flags using certutil. But before we do so we must know more about these trust flags. Here are my notes about trust flags from  Nelson Bolyard 's Brown bag:


There are three available trust categories for each certificate, expressed in this order: "SSL, email, object signing". In each category position use zero or more of the following attribute codes c,C,p,PT,u,w. The attribute codes for the categories are separated by commas.

 These trust flags are allow overrides, they tell NSS that it is ok to use certificates for this purpose even though they may not seem like it is. There are two categories : Validity override and trust overrides.

Flags that apply to CA Certificates
c - its for validity override - It tells even though this certificate doesn't look like a CA certificate - In version 1 certificates like old root CA certificates (that predated X509 v3) its necessary to set this.
C trusted CA flag implies c - This certificate should be treated like a CA certificate but also should be treated as root certificate we trust. By default roots are not trusted. We can take any certificate and mark it with a  "C" so it will be treated as a root CA certificate. When NSS clients/server validates certificate chain we can stop right there. NSS will build chain to send out as far as it reaches a root CA i.e. when it sees "C" flag. So intermediate CA certificates should not have this trust flag "C".

Flags that apply to end entity certificates like Server certificates or Client certificates. certificates that are not CA certificates :
p - valid peer flag - even though the certificate doesn't look like a peer cert, treat it like a peer cert.
P - Trusted Peer flag - implies p - This is a peer certificate & I want it take it at face value. I want u to trust this cert. Don't even bother to look & see if is issued by the issuer that you know and we are going to trust this certificate just by itself and so in the world of self signed server certificates its sometimes necessary to set this trusted peer flag in the client so the client will trust the certificate when it comes from the server.

T - special trust flag that is used in SSL Column only, it is used only on the server. It is not used on the client. It tells the server that this certificate is one whose name it should send out when it requests client authentication. When a server requests a client to authenticate itself with a cert, the server sends out a list of names of certificates that are issuers from whom it is willing to accept certificates. It figures out the names it should sent out because it looks for certificates with this "T" flag, those are the names it sends out to the remote client.

Flag that can not be set by certutil
u - User flag - This is  not a trust flag and this is not a flag that you can set with certutil, this is a dynamic flag it says that NSS has discovered that NSS has the private key associated with this cert. That's essential for server certificate. If you are a server and you are trying to send out your server certificate and its chain you have to have the private key associated with this server cert. This is something to look for.

Lately there's been some noise about slowloris, a perl script which sends HTTP requests slowly. While there's nothing new about this technique, I've been asked about it a few times so I wanted to show how easy it is to protect against it if you're lucky enough to be using Sun's Web Server 7.

In a nutshell, the script opens a connection to the target web server and sends valid request headers and then continues to send more headers, slowly. Specifically, it first sends:

GET / HTTP/1.1
Host: $hostname
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.503l3; .NET CLR .0.4506.2152; .NET CLR 3.5.30729; MSOffice 12)  
Content-Length: 42
X-a: b

Then it continues to send:

X-a: b

after every $timeout delay. It has a default $timeout of 100 seconds but you can change this with -timeout switch.

Let's look at the more general cases here instead of just slowloris specifically.

The most rudimentary form of this attack is to open a connection to the web server and either don't send anything or send a partial request and nothing else after that (as described above, this is not what slowloris does).

You'll want your web server to eventually time out and close the connection if this happens. In Web Server 7 this is controlled by the io-timeout element in server.xml. The default value is 30 (seconds). Let's try it:

% date;telnet localhost 80;date
Mon Jun 29 19:05:43 PDT 2009
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connection to localhost closed by foreign host.
Mon Jun 29 19:06:14 PDT 2009

As you can see, 31 seconds went by before the connection was closed. You can change io-timeout to be shorter if you wish:

  <http>
    <io-timeout>15</io-timeout> 
  </http>


% date;telnet localhost 8080;date
Mon Jun 29 19:15:12 PDT 2009
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connection to localhost closed by foreign host.
Mon Jun 29 19:15:27 PDT 2009

Above I changed the io-timeout to 15 and indeed it took 15 seconds before closing the mute connection. Let's try the same thing but send a partial request:

% date;telnet localhost 8080;date
Mon Jun 29 19:14:38 PDT 2009
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET / HTTP/1.1
Host: localhost
HTTP/1.1 408 Request Timeout
Server: Sun-Java-System-Web-Server/7.0
Date: Tue, 30 Jun 2009 02:14:54 GMT
Content-length: 148
Content-type: text/html
Connection: close

<HTML><HEAD><TITLE>Request Timeout</TITLE></HEAD>
<BODY><H1>Request Timeout</H1>
The server timed out waiting for the client request.
</BODY></HTML>Connection to localhost closed by foreign host.
Mon Jun 29 19:14:54 PDT 2009

Ok, let's try to make the attack more interesting. Instead of just going silent, the client can continue sending more request data, just slowly. This is what slowloris does. As long as the client sends a little bit of valid request data often enough to not get disconnected by the timeout it can hold on to the connection.

Fortunately Web Server 7 also monitors the time it takes to receive all the request headers. This can be configured using the request-header-timeout element in server.xml. This can be used to defeat a slowloris-type attack. Even thought the slowloris request never actually completes (since it just keeps sending more headers forever), Web Server 7 will stop waiting and close it off after request-header-timeout seconds go by.

  <http>
    <request-header-timeout>5</request-header-timeout> 
  </http>

Of course, if you set request-header-timeout to 5s you could then run slowloris with a -timeout of less than 5 seconds. However, this quickly starts to defeat the premise of this style of attack. The idea behind a slowloris-style attack is to attempt to tie up the web server quietly without the client having to generate hundreds or more connections per second. For fun, I set my request-header-timeout to 1s and ran slowloris with a -timeout of 1s. The result is the client machine uses up all its CPU generating new connections while Web Server 7 continues to be happily responsive.

A variant of this attack is to send a POST request, send all the request headers and then start to send the body data, slowly. Note that slowloris does not implement this (the -httpready flag sends a POST instead of a GET, but it continues to send X-a:b request headers, not request body data). However it is easy enough to write a tool to do this instead.

If you encounter that scenario you're in luck because Web Server 7 also monitors the time for the request body to arrive and you can set a timeout on that as well, using the request-body-timeout element:

  <http>
    <request-body-timeout>5</request-body-timeout> 
  </http>

That's all there is to it to protect against slowlaris and similar slow-client attacks if you're using Sun Web Server 7! Enjoy!

It's no secret that email overload is a problem these days, here's just a few of many articles on the topic:

• http://www.nytimes.com/2008/06/14/technology/14email.html?_r=1
• http://www.thepersonalfinancier.com/2008/11/email-becoming-distraction-heres-how-to.html

A quote from the second article above is particularly interesting (or scary):

   In this study Dr Jackson found that it takes an average of 
   64 seconds to recover your train of thought after interruption 
   by email. So people who check their email every five minutes 
   waste 8 and 1/2 hours a week figuring out what they were 
   doing moments before.

In nearly 20 years (wow!) of reading lots of email daily this has never been much of a problem for me though. I always managed to keep my inbox almost empty from day to day (I long considered 100 emails to be the maximum threshhold to ever have pending in the inbox).

Thinking back, I'd say historically the bulk of my incoming email has been either

• Administrivia (meeting announcements and such): quickly dispatched without thought or mental interruptions
• Engineering content, directly related to whatever I'm working on that day: these take time to read and process but since the emails are relevant to the current project they don't cause a mental context switch and may even help further the project at hand so there is a net win

As resources get tighter and I find myself filling more and more roles simultaneously the dynamic has changed in the last 6-9 months or so. From a perpetually clean inbox I've now gone to a significant backlog. Even more annoying is that I find there are many days where all I get to do is read email!

After some months it is clear this is not a temporary crunch, so I need to change strategies from what has worked in the past. I spent some time monitoring my email activity to figure out what is different. It's not really quantity, I've always received lots of email but it hasn't been a problem. The key difference appears to be that now I'm involved in many projects each one with many unrelated trains of discussion.

As emails arrive, each one is more often than not unrelated to the previous ones and also unrelated to what I'm actually trying to get done at the moment. And thus, I find myself facing the case made in the Dr. Jackson study quoted above.

As each email arrives I read it and start thinking of that particular project/problem for a few moments (a few seconds to a minute or two). It is not enough time to solve or address the issue, just enough to get distracted. Hoping to get back to the real work I was doing instead of spending more time on this new train of though, I don't actually process the email, so it remains in the queue.

By then, several other emails have arrived so I repeat the cycle with each one. By the time I finally get back to what I was actually working on, that project is so many mental context switches behind I no longer have any idea what I was doing and need to spend several minutes getting back into it. By which time, of course, ten more emails have arrived... and the cycle repeats all day.

So I need to address the interruption and context switch problem. A few weeks ago I started to allocate limited time to email. Specifically:

• Only read email in specific blocks of time preallocated for email on that day.
• If I can answer or resolve the issue in less than 5 minutes, do so right then, within the time allocated for email handling.
• If it's going to take any longer than that, add a task to the bottom of my to-do list and move on.
• The rest of the time, quit mutt and resist all urges to go look at email.

I started by allocating two hours a day to email, one in the morning and one in the afternoon. Quickly it became apparent this is not enough to keep up, so I increased it to three hours. I'll gather more data before settling on the final timing but looks like it'll have to be a bit over three hours a day for email processing.

Here's a graph, showing only a few days from last month. I'll post another one with much more data once some more time goes by so I have more numbers. The yellow area is my current email backlog and the blue line is the number of minutes a day spent processing email.

It's particularly a great endorsement when your customers say - I have fallen in love with your products, is it not? Sun.com, javafax.com, opensolaris.com and a number of online properties are managed by our Sun.com engineering (whom I like to call it as the Enterprise 2.0 team!) I thought it was pretty neat when I read (only recently, my bad), Jed Michnowicz praises simplicity of using Web Server 7.0.

http://blogs.sun.com/twothirds/entry/webserver_7_tricks.

Speaking of Sun Web Server 7.0 - check out http://www.sunwebserver.com/  - an ancillary web site for an upcoming book on Sun Web Server - to be published fairly soon by Pearson Media (Prentice Hall.) Bill Nelson - of GCA - our chief technology trainer (trainer of the trainers!) - is the lead author. Arvind Srinivasan (one of three co-authors including Bill, Arvind and myself) made significant contributions too. I am proud to contribute a couple of introductory chapters myself. I believe the book is already available for pre-ordering on Amazon and elsewhere, for example, http://www.informit.com/store/product.aspx?isbn=013715142X.
Cheers.
cvr

In addition to the unconference, Web Stack will also have an afternoon-long deep dive session at CommunityOne. It will be on Wednesday (June 3rd). Here is a link to the deep dive sessions page.

As of today, the agenda shown on the above page is fairly preliminary and we'll be tailoring it depending on the schedule of the speakers as well as feedback we get in the next few days on topic interest (so feel free to let us know if there is anything in particular you want explored in depth - webstack-discuss@opensolaris.org).

It'll be informative and a good chance to meet several of the Web Stack component developers and maintainers, so I hope to see you all there!

At the Web Stack webinar this week I mentioned the free unconference on the Sunday before CommunityOne (Sunday May 31st). Here is the signup sheet:
http://wikis.sun.com/display/GlassFishConferences/GlassFish+2009+unconference+planning

As noted in the above page, you don't need to be a participant in CommunityOne nor JavaOne in order to attend the unconference. Just sign up on the above page (add your name to the table) and show up! If you have an interest on any particular Web Stack topic areas please add them to the bottom of the page.

See you there!

Free CommunityOne Registration: http://developers.sun.com/events/communityone/2009/west/index.jsp. For More information on the deep dives, visit:
http://developers.sun.com/events/communityone/2009/west/deep_dives.jsp

Installing Sun Java System Web Server 7.0 on Mandriva

If you are getting an error "The Runtime User ID does not exist. Specify a valid UNIX user." while installing Sun Java System Web Server on Mandriva, all you have to do is create a softlink as shown below.  Its simple !

#ln -s /bin/id /usr/bin/id

You can also check if /usr/bin/cat exists or not. If it doesn't, then you have to create a softlink as given below

#ln -s /bin/cat /usr/bin/cat
  

*Note that Sun Java System Web Server is not officially certified or supported on Mandriva

 Open Web Server code has been updated

This is to announce that Open Web Server code has been updated. It  has all relevant bug fixes that went into Sun Java System Web Server 7.0 update 4 and 5 releases.

Free GlassFish Web Stack Webinar
Reserve Your Seat Today

Interested in finding out how you can to get production support for all the components of the LAMP/SAMP stack from a single vendor? Look no further!

Attend a FREE informative webinar to learn how you can easily develop an enterprise-wide strategy to meet your LAMP/SAMP needs and greatly simplify deployments across your organization.  Additionally, uncover how Sun GlassFish Web Stack, a component of the Sun GlassFish Portfolio provides out of box optimized performance.

Webinar: Sun GlassFish Web Stack: Highly Performant, Easy to Use Solution for LAMP/SAMP Deployments
Date:       Wednesday, May 13, 2009
Times:    10:00 am PDT / 1:00 pm EDT / 17:00 GMT
               7:30 pm PDT / 8:00 am May 14 India

Reserve your seat today: https://dct.sun.com/dct/forms/reg_us_0204_642_0.jsp

- Bring your questions to the live Q&A.
- Even if you can't make the live event, sign up anyway so we can send you the replay information.

As always, Sun representatives are available to talk to you about how you can leverage GlassFish Portfolio to speed development and improve your bottom line. Just contact us! - https://dct.sun.com/dct/forms/reg_us_0202_106_0.jsp

Thank you.

Murthy Chintalapati (cvr)

Sun Blogs (blogs.sun.com)  turns 5 today! Terrific job by .Sun engineering team!

We are proud to note that Sun Web Server 7 and MySQL are integral part of the Sun Blogs infrastructure - and you will find more details here in this article - http://www.sun.com/bigadmin/features/articles/sunblogs_webserver.jsp

Happy Birthday, Sun Blogs!

Cheers.
cvr

Kind of a last minute announcement, but for those at the MySQL Conference today, CVR and myself will be hosting a Web Stack BoF tonight. We'll cover Web Stack in general but of course with an eye on MySQL on OpenSolaris in particular. We should have a few demos from Sriram and some slides, but mostly just open to any Web Stack discussion. Join us at 7:30pm!

The MySQL Conference & Expo 2009 in Santa Clara is now in full swing, and we have BoF session tonight, titled:

OpenSolaris Web Stack: Where MySQL Meets the Next Generation OS

OpenSolaris Web Stack shines on Nehalem based servers

We are proud to be part of Nehalem (Intel Xeon 5500) based Sun's latest x64 servers launch.

As Shanti Subramanyam and her colleagues in our Systems organization show, OpenSolaris Web Stack on a pair of x2250 servers can scale to 10,000 users with 1 Million MySQL entries! Web Stack (officially Sun GlassFish Web Stack), as you might know, includes Apache web server, PHP, MySQL Community, lighttpd etc.

http://www.sun.com/offers/details/820-7729.html

Also, our colleagues in ISV Engineering demonstrated SugarCRM Performance on Sun Fire X4270 (with SSD Drives and Intel X5570 Processors) - using OpenSolaris Web Stack
http://blogs.sun.com/vanga/entry/sugarcrm_performance_on_sun_x4270.

Sun Web Server 7.0u5 Released

We are delighted to announce that Web Server 7.0 Update 5 has been released.

Download here and the Web Server 7.0 product Release Notes are here.
In addition to customer escalation & other bug fixes (>50) this release also contains important bug-fixes related to performance enhancement. For more details on specific issues resolved in this update release, please refer to our wiki. All users of Web Server 7.0 through Web Server 7.0 Update 4 are encouraged to upgrade.

What do you do when you get a 64 threads machine? I mean other than trying to find the hidden messages in Pi?
Our group recently acquired a T5120 behemoth for builds, and I wanted to see what it was capable of.

|uname -a
SunOS hypernova 5.10 Generic_127127-11 sun4v sparc SUNW,SPARC-Enterprise-T5120
|psrinfo | wc -l
      64

In my case I settled a slightly less ambitious endeavor. I recently had to implement Gaussian elimination as part of a university course work, I converted it to use the OpenMP and compiled with SunStudio.

|cat Makefile
gauss: gauss.omp.c
               /opt/SUNWspro/bin/cc -xopenmp=parallel gauss.omp.c -o gauss

|diff -u gauss.single.c gauss.omp.c
--- gauss.single.c      Tue Apr 14 14:32:57 2009
+++ gauss.omp.c Tue Apr 14 14:44:48 2009
@@ -7,6 +7,7 @@
 #include <sys/times.h>
 #include <sys/time.h>
 #include <limits.h>
+#include <omp.h>

 #define MAXN 10000  /* Max value of N */
 int N;  /* Matrix size */
@@ -35,7 +36,7 @@
     char uid[L_cuserid + 2]; /*User name */

     seed = time_seed();
-    procs = 1;
+    procs = omp_get_num_threads();

     /* Read command-line arguments */
     switch(argc) {
@@ -63,7 +64,7 @@
                 exit(0);
             }
     }
-
+    omp_set_num_threads(procs);
     srand(seed);  /* Randomize */
     /* Print parameters */
     printf("Matrix dimension N = %i.\n", N);
@@ -170,6 +171,7 @@

 }

+#define CHUNKSIZE 5
 void gauss() {
     int row, col;  /* Normalization row, and zeroing
                     * element row and col */
@@ -178,7 +180,9 @@

     /* Gaussian elimination */
     for (norm = 0; norm < N - 1; norm++) {
+        #pragma omp parallel shared(A,B) private(multiplier,col, row)
         {
+            #pragma omp for schedule(dynamic, CHUNKSIZE)
             for (row = norm + 1; row < N; row++) {
                 multiplier = A[row][norm] / A[norm][norm];
                 for (col = norm; col < N; col++) {

As you can see, the changes are very simple, and requires very little modification to the code. Below was my result running it in a single thread and next using all 64 threads.

 First the single threaded version.

|time ./gauss 10000 1 4
Random seed = 4
Matrix dimension N = 10000.
Number of processors = 1.
Initializing...
Starting clock.
Stopped clock.
Elapsed time = 1.11523e+07 ms.
(CPU times are accurate to the nearest 10 ms)
My total CPU time for parent = 1.11523e+07 ms.
My system CPU time for parent = 1080 ms.
My total CPU time for child processes = 0 ms.
--------------------------------------------
./gauss 10000 1 4  11163.06s user 1.64s system 99% cpu 3:06:04.96 total

And now using all threads.

|time ./gauss 10000 64 4
Random seed = 4
Matrix dimension N = 10000.
Number of processors = 64.
Initializing...
Starting clock.
Stopped clock.
Elapsed time = 254993 ms.
(CPU times are accurate to the nearest 10 ms)
My total CPU time for parent = 1.53976e+07 ms.
My system CPU time for parent = 37960 ms.
My total CPU time for child processes = 0 ms.
--------------------------------------------
./gauss 10000 64 4  15371.53s user 38.51s system 5757% cpu 4:27.65 total

Now I am all set to look for my name in Pi. :)

*the gaussian elimination source is here.

Having migrated originally from Civil Engineering, I have always been interested in parallel programming. Quite a few (almost all?) problems in that domain are what can be called embarrassingly parallel - be it Structural Mechanics, Fluid dynamics, or Virtual Modeling.

Recently I got interested in parallel programming again as part of my studies. While the university has a cluster setup, it is almost always in use, and is dead slow because of the number of users. So I tried setting up a simple OpenMP cluster locally for Ubuntu and Solaris,

Setting up OpenMP on Ubuntu is treated in quite a few places in the web, so I am not listing the steps for that. How ever I found that using the cluster tools from Sun was much more easy than messing with the MPICH distribution in Ubuntu.

Here are my notes on getting it to work.

As a prerequisite,

• You need some machines with the same OS and ARCH, NM
• A common NFS exported directory (mounted on the same path) on each machine. I used /home/myname as the NFS mount
  
• Ensure that you have password less login either using ssh or rsh.
  
• You also need to  install the cluster tools on each.

You can get the cluster tools from here. Ungzip it to directory and execute the ctinstall binary

|cat sun-hpc-ct-8.1-SunOS-sparc.tar.gz |gzip -dc | tar -xvpf -
|sun-hpc-ct-8.1-SunOS-sparc/Product/Install_Utilities/bin/ctinstall -l
|...

This will install the necessary packages. You might need to check the default parameters and verify that they are to your satisfaction.

|/opt/SUNWhpc/HPC8.1/sun/bin/ompi_info --param all all

In my setup, I wanted to use rsh while ssh is the default for clustertools

|/opt/SUNWhpc/HPC8.1/sun/bin/ompi_info --param all all | grep ssh
     MCA plm: parameter "plm_rsh_agent" (current value: "ssh : rsh", data source: default value, synonyms: pls_rsh_agent)
              The command used to launch executables on remote nodes (typically either "ssh" or "rsh")
     MCA filem: parameter "filem_rsh_rsh" (current value: "ssh", data source: default value)
|echo 'plm_rsh_agent = rsh' >> /opt/SUNWhpc/HPC8.1/sun/etc/openmpi-mca-params.conf

Once this is done, create your machines file (my machine names are host1 host2 host3 and host4)

|cat > machines.lst
host1
host2
host3
host4
^D

Now you are ready to verify that stuff works. Try

|/opt/SUNWhpc/HPC8.1/sun/bin/mpirun -np 4 -machinefile ./machines.lst hostname
host1
host2
host3
host4

This should also work

|/opt/SUNWhpc/HPC8.1/sun/bin/mpirun -np 4 -host host1,host2,host3,host4 hostname
host1
host2
host3
host4

If you get similar output, then you have successfully completed the initial configuration.
If you are unable to modify the /opt/SUNWhpc/HPC8.1/sun/etc/openmpi-mca-params.conf file, then you could try the below

|/opt/SUNWhpc/HPC8.1/sun/bin/mpirun -mca pls_rsh_agent rsh -np 4 -machinefile ./machines.lst hostname
host1
host2
host3
host4

Try an example,

|cat hello.c
#include <stdio.h>
#include <mpi.h>
int main(int argc, char **argv) {
     int my_rank;
     MPI_Init( &argc, &argv);
     MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
     printf("Hello world[%d]\n", my_rank);
     MPI_Finalize();
     return 0;
}

Try compiling and running

/opt/SUNWhpc/HPC8.1/sun/bin/mpicc -o hello hello.c
|/opt/SUNWhpc/HPC8.1/sun/bin/mpirun -np 4 -machinefile ./machines.lst ./hello
Hello world[2]
Hello world[3]
Hello world[0]
Hello world[1]

Now you are ready to try something larger. You can try with a simple scatter and gather of a matrix that is attached..

*Many thanks to the Sun HPC team for making this setup so easy.

For Sun WebStack, I have been working with multiple repositories (apache2, squid, mysql5, lighttpd etc). I often have to do the same or related changes in each of these repositories.The standard commands that I had to use to work with these repos are,

• Fetch the sources for all repos [ for i in $mylist; do hg clone ssh://webstack-root/$i ; done ]
• Fix some stuff and verify by doing a diff [ for i in *; do [ -d $i/.hg ] && (cd $i && hg diff ); done]
• Commit the fixes [for i in * ; do [ -d $i/.hg ] && (cd $i && hg commit -m "my message" ) ; done ]
• And push the same way. [for i in * ; do [ -d $i/.hg ] && (cd $i && hg push) ; done]

Quite recently I found the forest extension which allowed me to do
- on top dir in the server
hg init
- on the client systems, clone, push and pull
hg fclone ssh://webstack-root
hg fpull
hg up
hg fpush

These were very helpful, but I also wanted to review the diffs (and to request for review) and to commit related changes with the same aggregated commit message, Unfortunately the forest extension does not yet have commands for these.

I have modified the forest extension slightly to provide for new commands 'fdiff'  'fcommit' and 'fimport'. The source is here. In case you have the same need, drop this into your hgext directory (replace the forest.py if it exists) or set the extension path in your .hgrc

Beware that this is nothing more than a hack though.

Edit: added fimport