1. type "pfexec pkg set-publisher -O http://jucr.opensolaris.org/pending jucr-pending"
2. type "pfexec pkg refresh"

Powered by ScribeFire.

I needed to find a way to get the physical (MAC) address using C. From what I could gather from searching opensolaris.org, there are two methods for retrieving it: libdlpi and arp. libdlpi is the more elegant solution as it requires a simple call to dlpi_get_physaddr(). This is how ifconfig prints your network interface's MAC address. Unfortunately, libdlpi calls are only permitted as root.

As explained by James Carlson:

The reason it was like this was historical: getting the MAC address in
ifconfig meant opening up the DLPI node and talking to the driver. As
the drivers didn't have discrete privileges for each operation, and
you had to be almighty root to touch them, 'ifconfig' didn't show the
MAC address when not privileged.

The second solution is to use arp. In Solaris you can determine the physical address by looking at the arp tables directly (`arp -a | grep <INTERFACE>` or `netstat -p | grep <INTERFACE>`). With C, this can be done by using the if sockets and arp libraries.

I wrote up a solution called "getmac" using both methods. You can gather it here.

• Directions

  $ wget http://www.pauliesworld.org/project/getmac.c
  $ gcc getmac.c -o getmac -lsocket -ldlpi
  $ ./getmac <interface_name>
  arp:	ffffffffffff
  dlpi:	dlpi failure, are you root?
  $ pfexec ./getmac <interface_name>
  arp:	ffffffffffff
  dlpi:	ffffffffffff
  

When I build some source code and that I want to achieve the best performance I use the   Sun Studio compilers, especially on Solaris and SPARC.

Sun Studio offers a unique set of optimization features dedicated to processor instruction set that help me squeeze out the best perf out of C, C++ or Fortran code. Yet these options are so numerous that it can be a bit daunting to look into them.

If you are in a rush, you can use the  -fast option. What it really does is triggering a set of other options for maximum runtime performance. These options can be listed with:

$ CC -fast -dryrun
###     command line files and options (expanded):
### -xO5 -xarch=sparc -xcache=8/16/4:3072/64/12 -xchip=ultraT1  ...  -dryrun

Yet -fast has its own drawbacks. First, the options triggered might change from one compiler release to another. Also, the values for -xarch -xcache -xchip specify the processor for which to optimize, and -fast decides of these values based on the processor on which the compiler runs, which can deffer from the processor on which the code will eventually be executed. This is why I usually stay away from -fast.

Instead, here is a basic set of rules to easily decide on optimization options.

First, if some binary code already exists, I run a quick sanity check to see which options where used for this binary. On a non-strip executable, library, or object file, I run the following commands:

$ dump -C  Bar.o // for C++ code
... 
<122>  	.../tmanfe/; /opt/SUNWspro/bin/CC -G -xtarget=native -compat=4 -xO4  Bar.cpp

$ dwarfdump getpagesize //for C code
...
DW_AT_SUN_command_line   /opt/SUNWspro/bin/cc -c -xarch=sse2 -m32 -xO3 +w  getpagesize.c

Make sure the -g option is not present. This option tells the compiler to compile for debug, which in turns disables some optimization. Also verify that  -xOn (with n=[1|2|3|4|5]) is present: this turns on generic optimization. --xO1 and -xO2 are conservative. -xO5 is aggressive and may yield to perf degradation so I don't recommend to use it for a complete application. Limit its usage to some specific portions of code that are known to be heavily used and to benefit from optimization. I usual pick -xO3 as a basic level of optimization.

Use the -xarch and -xchip options specific to the targeted runtime processor. -xarch specifies the instruction set to be used while -xchip specifies the scheduling - or the ordering of the instructions.

The best value for -xarch can be found by running CC -xtarget=native -dryrun on the runtime platform. Here is a code snippet that does the work for you:

#!/bin/bash
for flag in `CC -xtarget=native -dryrun 2>&1 | grep xchip`
do
  if echo $flag | grep xchip >/dev/null ; then
      target=`echo $flag | grep xchip`
  fi
done
lenght=${#target}
echo ${target:7:$lenght}

The right value for -xchip is found the same way:

#!/bin/bash
for flag in `CC -xtarget=native -dryrun 2>&1 | grep xchip`
do
if echo $flag | grep xchip ; then
  target=`echo $flag | grep xchip`
fi
done
lenght=${#target}
echo ${target:7:$lenght}

The two code snippets above can be use to dynamically set up your compiler flags when generating Makefiles but again, make sure to run them on the processor targeted for runtime.

This type of generic optimization usually brings between 10 to 20% in terms of performance gain and it also sets the base-line for most sophisticated optimizations that will focus on the portions of code that are the more used during execution. These portions of code can be identified by running the Sun Studio Collector and Performance Analyzer on your code: no need to instrument your binary, no need to recompile for profiling. Just run the collect utility on the optimized binary you generated. Simple and easy!

vi upnp/src/api/upnpapi.c

#if defined(_sun)

#if defined(__sun)

$ ./configure CFLAGS="-DSPARC_SOLARIS" --disable-samples
$ gmake
# gmake install

Case in hand: Given a PeopleSoft Data Mover exported data file (db or dat file), how to extract the DDL statements [from that data file] which gets executed as part of the Data Mover's data import process?

Here is a quick way to do it:

1. Insert the SET EXTRACT statements in the Data Mover script (DMS) before the IMPORT .. statement.

   eg.,

   
   % cat /tmp/retrieveddl.dms
   
   ..
   SET EXTRACT OUTPUT /tmp/ddl_stmts.log;
   SET EXTRACT DDL;
   ..
   
   IMPORT *;
   
   

   It is mandatory that the SET EXTRACT OUPUT statement must appear before any SET EXTRACT statements.

2. Run the Data Mover utility with the modified DMS script as an argument.

   eg., OS: Solaris

   
   % psdmtx -CT ORACLE -CD NAP11 -CO NAP11 -CP NAP11 -CI people -CW peop1e -FP /tmp/retrieveddl.dms
   
   

   On successful completion, you will find the DDL statements in /tmp/retrieveddl.dms file.

Check chapter #2 "Using PeopleSoft Data Mover" in Enterprise PeopleTools x.xx PeopleBook: Data Management document for more ideas.

Novelties of Java EE 6 is in the improved ease of development, remodeled web services, and platform simplicity with introduction of profiles. GlassFish V3 is the first fully compliant Java EE 6 Enterprise Server.

Evolution of Java Platform Enterprise Edition since the year 1998 resulted in a more robust, scalable, highly performing, and extensible platform. GlassFish V3 Enterprise Server is built on modular OSGi Standard, with its MicroKernel HK2 reducing the startup time while providing the flexibility to extend the services at Runtime. GlassFish is now also embeddable.

Sun GlassFish V3 Enterprise Server,

My Presentation : A Technical Overview.

本博客关闭。

如果你还喜欢SUN的话，请抽空访问：http://thenetworkisthecomputer.com

Goodbye and have a nice day. 

Thank you for attending the Sun Startup Essentials webinar on Security for Web Applications, here is a list of useful links to learn more about and start implementing the different technologies that were covered during the presentation:

•  ZFS triple parity
• Securing Apache using SSL
• Securing MySQL in production
• Using virtualization to secure MySQL in a chrooted environment
• DTrace scripts to trace TCP connections

By now you certainly heard about the acquisition and about the 5-hour webcast given on the 27th about the Sun-Oracle joint strategy. So I am not going to discuss that. In fact, as an exception to this blog, I am not going to write about IT technologies. As something really unique, I am going to discuss mountain-biking.

This bike is certainly the lightest mountain-bike ever shipped fully equiped, ready to use. Weight being one of the top 3 - if not top 2 - key performance indicators in mtb-racing (with a 3% slope 60% of the biker energy is spent on fighting gravity), this makes this bike extremely competitive.

But what is really interesting is to understand how the engineers who designed the product could get such a result, and the answer is: integration. Nowadays the majority of bike manufacturers design their frame, get them built by a third party, and assemble them with of-the-shelf components. As a result, the way the frame and the components are put together is pretty much standard accross the industry.

Cannondale is one of the few player in the game that has its own set of components, such as the front suspension (yes, again, one of the lightest in the industry), the stem, or the bottom bracket, and they design them to get a straightforward integration that reduces weight. By the way it is still possible to set up their very special front suspension on a non-Cannondale frame.

So, it's because they design all the components that the complete system performs so well. Also, since Cannondale manufactures the frame themselves, they can offer a life-time guaranty on it  - wathever the weight of the pilot. Yes, controlling the technologies your product is made of comes with its own set of advantages.

That said, next time, I'll get back to IT technologies

Thank you for attending the Sun Startup Essentials webinar on Security for Web Applications, here is a list of useful links to learn more about and start implementing the different technologies that were covered during the presentation:

•  ZFS triple parity
• Securing Apache using SSL
• Securing MySQL in production
• Using virtualization to secure MySQL in a chrooted environment
• DTrace scripts to trace TCP connections
  

La semaine passée, j'ai eu le plaisir d'échanger avec deux startups du Web membres du programme Sun Startup Essentials.

Ce programme leur permet entre autres de bénéficier de conseils techniques gratuit au sujet de leur infrastructure, et nous avons donc réfléchi ensemble à des problématiques de montée en charge et de disponibilité.

Dans ce type de discussion, je commence toujours par demander si il existe un schéma de l'infrastructure informatique. Idéalement ce document couvre les aspects suivants (liste non exhaustive) :

• une liste des différents composants utilisés par le site Web (MySQL, Apache, etc.). Cette liste recense les dépendances externes de l'application, celles sur lesquelles la startup n'a pas la main - par opposition au code de l'application développée en interne. Afin d'améliorer l'indépendance de l'application vis-à-vis de la plate-forme il faut autant que faire ce peut réduire cette liste. Une dépendance exotique - rarement utilisée dans l'industrie - doit être justifiée par une valeur ajoutée spécifique. La liste des dépendances externes est normalement maintenue par les équipes de développement en association à un schéma fonctionnel de l'application,

• une vue d'ensemble de l'infrastructure. On y retrouve les différents composants logiciel (le serveur Web ou le serveur d'application, la base de donnée, l'application elle-même), mais aussi les composants matériel (serveurs, baie de stockage, réseau). Cette vue d'ensemble met en évidence les éléments redondés (comme les serveurs web frontaux ou une réplication de la base de donnée) et elle intègre des éléments comme le répartiteur de charge ou "load balancer". Cette vue d'ensemble doit traduire fidèlement la configuration d'exécution - encore appelée "runtime view" - de l'application. Par opposition, le schéma fonctionnel reflète l'architecture du code source, sans forcément prendre en compte l'environnement dans lequel l'application est déployée et exécutée. C'est en général à partir de cette configuration d'exécution qu'on va pouvoir réfléchir sur la montée en charge du site. Où faut-il rajouter de la capacité de calcul, où faut-il rajouter du stockage, et surtout, comment procéder à ces ajouts sans avoir à modifier de fond en comble l'application,
  

• un diagramme des flux peut être le bien venu, qui clarifie quels sont les composants mis en oeuvre pour traiter une requête client, et dans quel ordre. C'est utile quand il s'agit de prévoir ce qui va se passer quand le nombre de clients va augmenter. Ce diagramme peut préciser les différents protocoles utilisés par l'application à moins que ceux-ci n'apparaissent déjà dans la vue d'ensemble,

• un  schéma de sécurisation du site qui précise quelles sont les données critiques pour le business, les services exposés sur Internet, et comment celles-là et ceux-ci sont protégés contre les pannes matérielles, les attaques malveillantes, ou des des risques d'une autre nature. En complément d'une analyse des risques, on trouve aussi dans ce document les différents éléments de sécurité mis en place (cryptage, identification, virtualisation, sous-réseaux, réseaux virtuels et NAT, etc.) Encore une fois, ce schéma peut ne faire qu'un avec la vue d'ensemble,

• pour finir, le traditionnel schéma de base de données qui décrit comment ces dernières sont organisées et reliées entre elles et aide à comprendre si la base de données peut monter en charge ou non (limitez les jointures!)
  

Pourquoi se livrer à un tel travail ?

• parce que pour la grande majorité des société du Web, la qualité de service du site (temps de réponse, disponibilité) fait partie de la proposition de valeur du produit, et que cette qualité de service dépend de l'infrastructure du site : le schéma d'infrastructure permet d'identifier les éléments clés de l'infra qui la conditionnent,

• parce que documenter son infrastructure permet de mieux la maîtriser et que cette maîtrise dans le long terme est un facteur clé de succès, particulièrement si vous - l'entrepreneur - n'avez pas un profil technique,

• enfin, ces documents démontrent aux investisseurs votre maturité et votre professionnalisme technologique. A défaut de pouvoir rentrer dans les détails, il sauront apprécier une réflexion menée en amont sur la pérennité du site et donc de leurs investissements.
  

Indeed, for those companies who have strong IT expertise and sufficient resources to build their own data center, security may be a disadvantage of public clouds of cloud computing. However, vast majority of organizations are not capable to setup a sophisticated IT infrastructure on their own, because of lacking of either necessary conditions mentioned above. In most cases, organizations focus on functional requirements and are not able to pay adequate attention to security issues when building IT infrastructure. The consequence is many IT systems running without necessary security control procedures, and thus be in a dangerous environment. This is especially true for small and medium size enterprises (SME). For such organizations, cloud computing, even public clouds, actually becomes a more secure option. Cloud computing providers pervasively build network security and system security into the cloud infrastructure. They have well-equipped and professionalized staffs to protect the cloud system from network threats and virus. Security of cloud systems is one of the basic offerings of any mainstream cloud services and normally does not charge extra service fee. Thus, when facing government's requirement on system security, enterprises can effectively increase the security of their information system by leveraging cloud computing if they do not want to spend resources on this task. Here is an example of the security requirement from governments:

November 24, 2009, the state council of China required companies in the network media industry to take the responsibility of maintaining network security of their own information systems. One background of this requirement is, presently, most Chinese media companies are not specialized in network security. Most information systems have potential security problems, and the systems are vulnerable to network attacks. It is obligated for media companies to take action in response to this requirement from the government. In the traditional on-premise computing, the most common reaction is to allocate dedicated resources to take charge of network security. This manner normally implies more investment on computer hardware, software and human resources. Since this is not a one-off investment, so it has to be integrated into the cost structure of the company as a constant operating cost. In addition, network security does not belong to the core competence of a media organization. Hence, such investment may do harm to the profitability of organizations.

Mercredi 10 Février, 11h00 (Paris) : Sécurité pour les applications Web. Pour les startups du Web, la protection et la sécurisation de leurs applications, de leurs données, et de celles de leurs clients est un véritable facteur clé de succès.  Ce Webinar couvre les différents challenges liés à la sécurité ainsi que les solutions associées telles que l'encryption, l'authentification, les certificats, la sécurisation du stockage et le stockage à tolérance de panne, les environnements étanches. Les architectes de Sun Startup Essentials présenteront des implémentations économiques basées sur des composents standards et ouverts tel qu'Apache, MySQL et ZFS. Ce webinar fait partie de l'accompagnement du programme Sun Startup Essentials et est réservé à ses membres.

Votre société a moins de 6 ans et moins de  150 employés : Rejoignez Sun Startup Essentials >>

Some HPC ISV's may be interested in Graphing Solaris Performance Stats with gnuplot

Mercredi 10 Février : Securité pour les applications Web. Pour les startups du Web, la protection et la sécurisation de leurs applications, de leurs données, et de celles de leurs clients est un véritable facteur clé de succès.  Ce Webinar couvre les différents challenges liés à la sécurité ainsi que les solutions associées telles que l'encryption, l'authentification, les certificats, la sécurisation du stockage et le stockage à tolérance de panne, les environnements étanches. Les architectes de Sun Startup Essentials présenteront des implémentations économiques basées sur des composents standards et ouverts tel qu'Apache, MySQL et ZFS. Ce webinar fait partie de l'accompagnement du programme Sun Startup Essentials et est réservé à ses membres.

Votre société a moins de 6 ans et moins de  150 employés : Rejoignez Sun Startup Essentials >>

Looking for a project management tool?

Check out MyDashboardPro.com, a simple, yet powerful and secure project management tool that allows you to collaborate with as many people as you want, inside OR outside of your company intranet!

Best part: the tool is free to use and comes with 100MB accounts - try it out at: www.MyDashboardPro.com.

A little background: 25,000+ Sun employees use a similar tool worldwide, so this is industrial-strength! The basic code of this tool was open-sourced in 2007 and became part of OpenEco.org where it underwent severe security audits. Enjoy!

Recently I had the good fortune to do some testing on an Amber Road. More officially known as the Sun Storage 7410 Unified Storage System.

The machine had 64 GB of internal memory and two quad core CPUs. A single JBOD holding 24 disks, of which one was a SSD optimized for write access (a writezilla.) The other 23 disks were "normal" 7200 RPM disks of one TB each. For connectivity some of the four gigabit NICs were used.
Besides the above mentioned SSD, this system also had two other SSDs optimized for reading (a readzilla.)
This appliance used ZFS as a file system. The writezilla was used to hold the ZFS intent log. The readzilla's were used as a level 2 ZFS (L2ARC) file system cache.

For more technical information about this product please check the website at:

http://www.sun.com/storage/disk_systems/unified_storage/7410

Introduction

And indeed the throughput numbers we received were indeed not what we expected.

In order to get a better understanding a 7410 was setup together with some load generating equipment in one of our labs. The ISV was able to send us a copy of a part of their email archive we could use for testing. The Amber Road was configured in a comparable way to the one at the ISV location. The initial tests were indeed below expectations. The quest to find the bottleneck began.

Approach

There were quite a few elements that were "unknown" to me one of which were the internals of the Amber Road, including the hardware and the software (operating system, ZFS pool and file system, network interfaces , etc.)
The NICs, were they somehow dependent on each other? How much traffic can a single NIC handle? In throughput, but also in the number of packets?
The volume(s) in the Amber Road were used with NFS over gigabit ethernet. The Linux box was used to simulate the mail machine. Could this one handle the load required? The Linux volume manager was used. Using ext3 as a file system. Could this one deliver the load required? The tool used to make the backup, rsync, was this one able to drive all the resources?

I started with the front end: the Linux box. Especially the volume manager. Using

find <mail archive on lvm volume> -print | cpio -oc >/dev/null

as a way to simulate the reading part. I had a second machine on which Solaris 10 was installed. Exactly the same hardware as the Linux box. And of course couldn't resist to do the same reading tests on this one. I was not impressed with the load that could be generated from a couple of internal disks. Fortunately I had a couple of disk arrays "lying around".
I used a Sun StorageTek 6140 array holding 12 spindles spinning at 15K rpm. The array has a gigabyte of cache and two controllers.

The result of the above command:

1: 6140 Linux lvm (2 6140 volumes): 33 min
2: 6140 Solaris UFS on 2 6140 Disk Suite volume: 16 min
3: 6140 Linux nolvm (1 6140 volume): 34 min
4: 6140 Linux nolvm (4 6140 vols in parallel): 24 min

Tests 1 and 2 are performed with the same 6140 configuration. Same hardware. Different OS-es and volume managers. To check the Linux result I used half the disk capacity in the array (test 3) without lvm and 4 volumes without lvm (test 4). This last one had one volume mounted under /data1, the second volume on /data2, etc. To parallellize multiple find /dataX -print | cpio -o >/dev/null sessions were done on each file system in parallel.

After this result we decided to look into the other elements of the black box using Solaris only. Just to ensure we could generate load once we started testing the Amber Road.

Network

The next part to check was the Amber Road network. Or better, how much load can the Amber Road NICs sustain. A single volume on the Amber Road was NFS mounted with the following options:

rw,vers=3,rsize=8192,wsize=8192,soft,intr,proto=tcp,timeo=600,retrans=2,sec=sys

Since the rsync backup includes many file system operations (read, write, stat, etc) and is optimized to not write a file to backup if the file in question happens to reside on the backup volume already I changed the rsync use to a sequence of 'find ... cpio' ran in parallel. Each find command (see above) used its own private mail archive. After some analysis it was clear NFS was not a bottleneck. Using the analytics from the Amber Road the following screen dump shows the three NICs used were all running close to the theoretical speed of gigabit ethernet. In total "pumping" 300MB/sec look perfect to me, when using three interfaces. Each interface handling close to 15,000 NFSOPS/sec.

With these results I concluded that both client and network are running fine. I needed to focus on the 7410 now. I had 24 disks in this appliance. The GUI was not my favorite tool in order to look at the disk utilization and other statistics. Now I must admit, that I am in this business for quite some years. Still prefer the command line. Although it might be good to understand that the punch cards period was even before my time.....

Method

The ISV remark included a statement about other applications suffering when a rsync was running. The rsync themselves also took a remarkable long time. This would translate to an average of 50MB/sec transfer rate.
I changed the approach a little bit. I first checked what could be seen as reasonable for the Amber Road. Fortunately there is a load generating tool called vdbench available. This open source tool can be downloaded from http://sourceforge.net/projects/vdbench. It has many possibilities. One of these is a simulation of file system tests. For this it creates a directory structure with as many files as specified. This is all done with a parameter file. Here is one example, for a pure read only test:

fsd=fsd1,anchor=/data_remote/test,depth=2,width=8,files=10000,sizes=(8k,60,32k,25,512k,10,4m,5)
fwd=fwd1,fsd=fsd1,operation=read,skew=50,xfersize=8k,fileio=sequential,fileselect=random,threads=48
fwd=fwd2,fsd=fsd1,operation=read,skew=50,xfersize=8k,fileio=random,fileselect=random,threads=48
rd=rd1,fwd=(fwd1,fwd2),fwdrate=10000,format=no,elapsed=72000,interval=1

The generator runs on a client machine. The Amber Road volume is NFS mounted under /data_remote/test. There is a directory structure of 2 levels deep. Each directory at the end of this tree holds 10,000 files. Files of different sizes: 60% of them are 8KB, 25% is 32KB, 10% are 51KB and 5% are 4MB. A total a 96 threads are being used to generate the load. The test runs for 72,000 seconds.

The example above tries to create a continuous load of 10,000 IOPS (fwdrate).

Tests had to be done over a long time. The file system cache (ZFS ARC) was approximately 60 GB. It was already shown that the behavior of the Amber Road was perfect as long as capacity was available in ARC. Once the ARC was fully utilized the behavior started to show. For my tests this steady state behavior was needed in order to be comparable to the ISV situation.

The load test was done with a RAID-Z2 default setup, a RAID-Z2 narrow and a mirrored setup. Here's a picture with the output. Time versus number of IOPS. After the volume was created the directory structure and the files were created. After this a load run was started with 50% read.

Besides some spikes there is not much difference between these three. They all can cope with a 10,000 IOPS load very well. However this was during the first 10 minutes after all the files were created. Next graph shows the behavior over a ten times longer interval.

This data are from the same three tests. However here we see that at 700 seconds the default setup drops down to approximately 6,000 IOPS. Looking into the data presented by the vdbench log output (not shown here) also shows that there is no 50% read on average any more.
The RAID-Z2 narrow cab handle the 10,000 IOPS load longer. At 2700 seconds the behavior starts to change. It averages around 10,000 IOPS but the behavior is far less constant.
The mirrored setup performs the 10,000 IOPS nicely over the whole test period.

The tests described were executed to create a background load for running rsync. The bursts in the blue graph are points where the rsync is started. These were scheduled at 1800, 3600, and 5400 seconds. No ill effects during the test on a mirrored setup.

Here is a close up around the 900 second period.

One of the things that vdbench tries to do is to make up for missing IOs. The first big dip after T=700 is followed by a much bigger load request. The final result is that the system seems to throttle down to a 6000 IOPS level. During the first 600 seconds the average response time was around the 1 milli second. 200 seconds later this became 20 milli seconds. During the first 600 seconds the system was able to push 80 MB/sec. After the 800 second mark there was only 40-50 MB/sec left. Finally the amount of read ops settled around 85% of the total.

Finally to make this comparison complete, a close up of the graph around T=2900

Results

The above mentioned test was redone for the three setups but now with a read write ratio of 75:25 and again with a 100% read test.

In order to see the effect on the run time of an rsync command that would create a backup of a 2 GB "mail-archive" this command was executed at 1800, 3600 and 5400 (rsync --delete -a <mail archive> <destination directory>.) The completion times are in the following tables. The times for the rsync during the default setup test were so long, that this test needed to be redone for a much longer time (14400 seconds.)

Graphing Solaris Performance Stats with gnuplot

It is not unusual to see an engineer import text from "vmstat" or "iostat" to a spreadsheet application such as Microsoft Office Excel or OpenOffice Calc to visualize the data.  This is a fine approach when used periodically but impractical when used frequently.  The process of transferring the data to a laptop, manually massaging the data, launching the office application, importing the data and selecting the columns to chart is too cumbersome when used as a daily process or if there are a large number of machines that are being monitored.  It my case, I needed to visualize the performance from a few servers that were under test, and needed a few graphs from the servers, a few times a day.  I used some traditional Unix scripts and gnuplot (http://www.gnuplot.info) from the Companion CD (http://www.sun.com/software/solaris/freeware) to quickly graph the data.

The right tool for graphing Solaris data depends on your use case scenario:

• One or two graphs, now and then: Import the data into your favorite spreadsheet application.
• Historic data, more graphs, more frequently: use gnuplot
  
• Many graphs, real-time or historic data, for more machines, such as a grid of servers being managed by Sun Grid Engine:  a formal tool such a Ganglia (http://ganglia.info, http://www.sunfreeware.com/) is recommended. An advantage of Ganglia is that performance data is exposed via a web interface to a potentially large number of viewers in real time.

That being said, here are some scripts that I used to view Solaris Performance data with gnuplot.

1. Gathering data.  For each benchmark run, a script was used to start gathering performance data:

#!/usr/bin/ksh

dir=$1
mkdir $dir
vmstat 1        > $dir/vmstat.out        2>&1 &
zpool iostat 1  > $dir/zpool_iostat.out  2>&1 &
nicstat 1       > $dir/nicstat.out       2>&1 &
iostat -nmzxc 1 > $dir/iostat.out        2>&1 &
/opt/DTraceToolkit-0.99/Bin/iopattern 1 > $dir/iopattern.out   2>&1 &

The statistics gathering processes were all killed at the end of the benchmark run. Hence, each test had a directory with a comprehensive set of statistics files.

Next it was necessary to write a set of scripts to operate on the directories.

2. Graphing CPU utilization from "vmstat".

This script was fairly short and straightforward.  The "User CPU Utilization" and "System CPU Utilization" are in the 20th and 21st columns.  I added an optional argument to truncate the graph after a specific amount of time to account for the cases where the vmstat process was not killed immediately after the benchmark.  A bash "here document" is used to enter gnuplot commands.

#!/usr/bin/bash

dir=$1
file=$1/vmstat.out

if [ $# == 2 ] ; then
  minutes=$2
  (( seconds = minutes * 60 ))
  cat $file | head -$seconds > /tmp/data
  file=/tmp/data
fi

gnuplot -persist <<EOF
set title "$dir"
plot "$file" using 20 title "%user" with lines, \
     "$file" using 21 title "%sys" with lines

EOF

3. Graphing IO throughput from "iostat -nmzxc 1" data

This script was a little bit more complicated for three reasons:

• The data file contains statistics for several filesystems that are not interesting and will be filtered out.  The script needs to be launched with an argument that will be used to select one device.
• I used the 'z' option to iostat which does not print traces when the device is idle (Zero I/O).  The 'z' option makes a smaller file that is more human readable, but it it not good for graphing.  Thus I needed synthesize the zero traces before passing the data to gnuplot.
• I wanted to include a smooth line for the iostat "%w" and "%b" columns with a scale of 0 to 100.
  

#!/usr/bin/bash

# This script is used to parse "iostat -nmzxc" data which is formatted like this:
#
#                     extended device statistics
#     r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
#     0.0    0.9    0.8    3.8  0.0  0.0    0.0    0.5   0   0 c0t1d0
#     0.0    0.0    0.0    0.0  0.0  0.0    0.0    2.4   0   0 sge_master:/opt/sge6-2/default/common
#     0.0    0.8    1.9  184.5  0.0  0.0    4.1   31.1   0   1 192.168.2.9:/jbod



if [ $# -lt 2 -o $# -gt 3 ] ; then
  echo "Usage: $0 pattern dir [minutes]"
  exit 1
fi

pattern=$1
dir=$2
(( minutes = 24 * 60 )) #default: graph 1 day

if [ $# == 3 ] ; then
  minutes=$3
fi

(( seconds = minutes * 60 ))
all_data=$dir/iostat.out
plot_data=/tmp/plot_data

if [ ! -r $all_data ] ; then
  echo "can not read $all_data"
  exit 1
fi

# For each time interval, either:
#   print the trace for the device that matches the pattern, or
#   print a "zero" trace if there is not one in the data file 
# You can tell that there was no trace for the device during an
# interval if you reach the "extended device statistics" line 
# without finding a trace
gawk -v pattern=$pattern '
$0 ~ pattern {
  printf("%s\n",$0);
  found = 1 ;
}

/extended/ {
  if (found == 0)
    printf("    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 192.168.2.9:/jbod \n")
  found = 0;
} ' $all_data | head -$seconds > $plot_data

gnuplot -persist <<EOF
set title "$pattern - $dir"
set ytics nomirror
set y2range [0:100]
set y2tics 0, 20
plot "$plot_data" using  3 title "read (kb/sec)" axis x1y1 with lines, \
     "$plot_data" using  4 title "write (kb/sec)" axis x1y1 with lines, \
     "$plot_data" using  9 title "%w" axis x1y2 smooth bezier with lines, \
     "$plot_data" using 10 title "%b" axis x1y2 smooth bezier with lines

EOF

I created the following graph with the command "graph_iostat.bash jbod NFS_client_10GbE 5" to select data only from the "jbod" NFS mount, where the data is stored in the directory named "NFS_client_10GbE" and only graph the first 5 minutes worth of data.

The iostat data was collected on an NFS client connected with a 10 gigabit network.  There is some write activity (green) at the start of the 5 minute sample period, followed by several minutes of intense reading (red) where the client hits speeds of 600-700MB/sec. The purple "%b" line, with values on the right x1y2 axis, indicates that during the intense read phase, the mount point is busy about 90% of the time.  

4. Graphing I/O Service time from "iostat -nmzxc" data.

I also find that columns 6 and 7 from iostat are very interesting and can be graphed using a simplification of the previous script.

• actv: average number of transactions actively being serviced
• svc_t: average response time  of  transactions,  in  milliseconds

#!/usr/bin/bash

#                     extended device statistics
#     r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
#     0.0    0.9    0.8    3.8  0.0  0.0    0.0    0.5   0   0 c0t1d0
#     0.0    0.0    0.0    0.0  0.0  0.0    0.0    2.4   0   0 sge_master:/opt/sge6-2/default/common
#     0.0    0.8    1.9  184.5  0.0  0.0    4.1   31.1   0   1 192.168.2.9:/jbod


if [ $# -lt 2 -o $# -gt 3 ] ; then
  echo "Usage: $0 pattern dir [minutes]"
  exit 1
fi

pattern=$1
dir=$2
(( minutes = 24 * 60 )) #default: graph 1 day

if [ $# == 3 ] ; then
  minutes=$3
fi

(( seconds = minutes * 60 ))
all_data=$dir/iostat.out
plot_data=/tmp/plot_data

# For each time interval, either:
#   print the trace for the device that matches the pattern, or
#   print a "zero" trace if there is not one in the data file 
# You can tell that there was no trace for the device during an
# interval if you reach the "extended device statistics" line 
# without finding a trace
gawk -v pattern=$pattern '
$0 ~ pattern {
  printf("%s\n",$0);
  found = 1 ;
}

/extended/ {
  if (found == 0)
    printf("    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 192.168.2.9:/jbod \n")
  found = 0;
} ' $all_data | head -$seconds > $plot_data

gnuplot -persist <<EOF
set title "$pattern - $dir"
set log y
plot "$plot_data" using  6 title "wsvc_t" with lines, \
     "$plot_data" using  7 title "asvc_t" with lines \

EOF

Here is the graph produced by the command "graph_iostat_svc_t.bash jbod NFS_client_10GbE 5"

5. Graphing network throughput data from "nicstat"

Another very valuable Solaris performance statistics tool is "nicstat".  For the download link, see http://blogs.sun.com/timc/entry/nicstat_the_solaris_and_linux .  A script to graph the data from nicstat follows the same pattern.

#!/usr/bin/bash

if [ $# -lt 2 -o $# -gt 3 ] ; then
  echo "Usage: $0 interface dir [minutes]"
  exit 1
fi

interface=$1
dir=$2
(( minutes = 24 * 60 )) #default: graph 1 day

if [ $# == 3 ] ; then
  minutes=$3
fi

(( seconds = $minutes * 60 ))
all_data=$dir/nicstat.out
plot_data=/tmp/plot_data

if [ ! -r $all_data ] ; then
  echo "can not read $all_data"
  exit 1
fi

grep $interface $all_data | head -$seconds > $plot_data

gnuplot -persist <<EOF
set title "$interface - $dir"
plot "$plot_data" using 3 title "read" with lines, \
     "$plot_data" using 4 title "write" with lines
EOF

 "graph_nicstat.bash ixgbe2 NFS_server_10GbE 5"

6. Graphing IO throughput from "zpool iostat" data

The challenge for plotting "zpool iostat" data is that the traces are not in constant units and therefore it is necessary to re-compute the data in constant units, in this example, MB/sec. 

#!/usr/bin/bash

if [ $# -lt 2 -o $# -gt 3 ] ; then
  echo "Usage: $0 pattern dir [minutes]"
  exit 1
fi

pool=$1
dir=$2
(( minutes = 24 * 60 )) #default: graph 1 day

if [ $# == 3 ] ; then
  minutes=$3
fi

(( seconds = minutes * 60 ))
all_data=$dir/zpool_iostat.out
plot_data1=/tmp/plot_data1
plot_data2=/tmp/plot_data2

if [ ! -r $all_data ] ; then
  echo "can not read $all_data"
  exit 1
fi

grep $pool $all_data | awk '{printf("%s/1048576\n",$6)}' | sed -e 's/K/*1024/g' -e 's/M/*1048576/g' -e 's/G/*1073741824/g' | bc | head -$seconds > $plot_data1
grep $pool $all_data | awk '{printf("%s/1048576\n",$7)}' | sed -e 's/K/*1024/g' -e 's/M/*1048576/g' -e 's/G/*1073741824/g' | bc | head -$seconds > $plot_data2

gnuplot -persist <<EOF
set title "$pool - $dir"
set log y
plot "$plot_data1" using 1 title "read (MB/sec)" with lines, \
     "$plot_data2" using 1 title "write (MB/sec)" with lines

EOF

Graphing the IO throughput of the zpool named "jbod" using the command  "graph_iostat_svc_t.bash jbod NFS_client_10GbE 5" shows that zpool can deliver data at speeds of close to one gigabyte per second.

It is easy to modify the scripts above to graph the output of many tools that output a table of data in text format.

The PC sits on top of the enclosure on my bookshelf taking up 8.5" x 5.0" x 6.5" amount of space. This is not only power efficient, but space efficient since I am using 4 x 1TB drives. 4TB total (theoretical), ~2.6TB in a ZFS raidz. If I were to have purchased the 2TB drives, it would be even better.

Doug's blog on the FIT-PC2 gives a good overview on the features of the device and what works. There is no wifi driver and Xorg doesn't work, so you may want to install OpenSolaris on another machine before installing the internal HDD. My server is headless and uses the built-in gigabit ethernet, so I don't care about those issues.

• CompuLab FIT-PC2 Diskless Nettop PC
  $273
  
• Mediasonic (HF2-SU2S2) Pro Box 4 Bay Enclosure
  $135
  
• Samsung 1TB 7200 RPM
  4 x $85 = $340
  

Powered by ScribeFire.

Sun Startup Essentials est à nouveau partenaire de la Startup Academy pour l'édition 2010. Ce programme d'accompagnement de startups en est à sa troisième session. Près de 250 startups ont présenté leur candidature depuis la première édition en novembre 2008 pour pouvoir bénéficier d'un accompagnement complet et personnalisé par un panel d'experts. Désormais, le programme est permanent. Les entrepreneurs peuvent poser leur candidature à tout moment pour participer à la compétition. Celle-ci aura lieu du 5 mars (date limite de dépôt des candidatures) au 13 avril (date d'annonce des résultats). Chaque candidature est publiée sur le blog de la Startup Academy, ce qui offre aux entreprises qui concourent une visibilité prolongée et une opportunité supplémentaire de mieux se faire connaître du public, des investisseurs et des partenaires potentiels.

"Canary In A Coalmine"
The Police
from the album "Zenyatta Mondatta", 1980

Poor Steve A.^[1] ... This entry is not about Steve A. though. It is about the new PeopleSoft NA Payroll benchmark result that Sun published today.

First things first. Here is the direct URL to our latest benchmark results:

        PeopleSoft Enterprise Payroll 9.0 using Oracle for Solaris on a Sun SPARC Enterprise M4000 (16 job streams^[2] -- simply referred as 'stream' hereonwards)

The summary of the benchmark test results is shown below only for the 16 stream benchmarks. These numbers were extracted from the very first page of the benchmark results white papers where Oracle|PeopleSoft highlights the significance of the results and the actual numbers that are of interest to the customers. The results in the following table are sorted by the hourly throughput (payments/hour) in the descending order. The goal is to achieve as much hourly throughput as possible. Click on the link that is underneath the hourly throughput values to open corresponding benchmark result.

This is all public information. Feel free to compare the hardware configurations and the data presented in both of the rows and draw your own conclusions. Since both Sun and HP used the same benchmark toolkit, workload and ran the benchmark with the same number of job streams, comparison should be pretty straight forward.

If you want to compare the 8 stream results, check the other blog entry: PeopleSoft North American Payroll on Sun Solaris with F5100 Flash Array : A blog Reprise. Sun used the same hardware to run both benchmark tests with 8 and 16 streams respectively. We could have gotten away with 20+ Flash Modules (FMODs), but we want to keep the benchmark environment consistent with our prior benchmark effort around the same benchmark workload with 8 job streams. Due to the same hardware setup, now we can easily demonstrate the advantage of parallelism (simply by comparing the test results from 8 and 16 stream benchmarks) and how resilient and scalable the F5100 Flash array is.

Our benchmarks showed an improvement of ~55% in overall throughput when the number of job streams were increased from 8 to 16. Also our 16 stream results showed ~55% improvement in overall throughput over HP's published results with the same number of streams at a maximum average CPU utilization of 45% compared to HP's maximum average CPU utilization of 89%. The half populated Sun Storage F5100 Flash Array played the key role in both of those benchmark efforts by demonstrating superior I/O performance over the traditional disk based arrays.

Before concluding, I would like to highlight a few known facts (just for the benefit of those people who may fall for the PR trickery):

1. 8 job streams != 16 job streams. In other words, the results from an 8 stream effort is not comparable to that of a 16 stream result.
2. The throughput should go up with increased number of job streams [ only up to some extent -- do not forget that there will be a saturation point for everything ]. For example, the throughput with 16 streams might be higher compared to the 8 stream throughput.
3. The Law of Diminishing Returns applies to the software world too, not just for the economics. So, there is no guarantee that the throughput will be much better with 24 or 32 job streams.

Other blog posts and documents of interest:

1. Best Practices for Oracle PeopleSoft Enterprise Payroll for North America using the Sun Storage F5100 Flash Array or Sun Flash Accelerator F20 PCIe Card
2. PeopleSoft Enterprise Payroll 9.0 using Oracle for Solaris on a Sun SPARC Enterprise M4000 (8 streams benchmark white paper)
3. PeopleSoft North American Payroll on Sun Solaris with F5100 Flash Array : A blog Reprise
4. App benchmarks, incorrect conclusions and the Sun Storage F5100
5. Oracle PeopleSoft Payroll (NA) Sun SPARC Enterprise M4000 and Sun Storage F5100 World Record Performance

Notes:

[1] Steve A. tried so hard and his best to make everyone else believe that HP's 16 job stream NA Payroll 240K EE benchmark results are on par with Sun's 8 stream benchmark results. Apparently Steve A. failed and gave up after we showed the world a few screenshots from a published and eventually withdrawn benchmark [ by HP ]. You can read all his arguments, comparisons etc., in the comments section of my other blog entry PeopleSoft North American Payroll on Sun Solaris with F5100 Flash Array : A blog Reprise as well as in Joerg Moellenkamp's blog entries around the same topic.

[2] In PeopleSoft terminology, a job stream is something that is equivalent to a thread.

The LiveScribe Developer Challenge has been extended to run through March 13th.

Livescribe is Hosting a Developer Challenge

Submit your original smartpen app to the Livescribe Developer Challenge for a chance to win over $10,000 in prizes and special promotion within the Application Store and through media outlets. The contest runs worldwide December 7th 2009 – Mar 13th 2010. For official rules and to enter, please visit the Livescribe Developer Challenge website

"Yet another way the Sun Startup Essentials program can help our startups (and the economy) is to provide current opportunities to our large readership & member-base. We are not trying to be recruitment agents, nor will we get involved in any part of that process, but we have a great platform to assist startups. "

Following on from the last three startup job postings here are some more hot off the press!

• iPlatform hiring Python & django see http://theiplatform.com/jobs/
• Techfluff.tv looking for new London correspondent-Get a new Kodak Zi8! http://tinyurl.com/ydaoda7
• Moonfruit hiring! Key front end developer for rapidly growing business. Dynamic team Great location http://bit.ly/7Vj9NA
• Spoonfed is looking for a tech-savvy customer services executive! Join the amazing team, job spec here: http://bit.ly/7Zy21G