SSL Support to Tomcat 6 on Windows/Linux

Quick Set Up Of SSL on Tomcat 6

Setting up SSL on Tomcat 6 is easy and you don’t have to do much for converting your web application to work with the Https protocol. But however, the problem you would find to set up SSL is the documentation available over the web. The documentation source is available on the Apache site but it starts off good and ends with a lot of confusion. Especially I was confused on the OpenSSL part where it says to use OpenSSL.

It might be good in a production environment to use OpenSSL but if you just want to test out SSL with Tomcat 6 alone then it is more than enough to just have your JDK and Tomcat setups. So I would make you walk through the same steps which I did while getting SSL up and running and building a secured web app within a matter of minutes.

The things which I have used to setup SSL consists of:

• JDK 1.6
• Tomcat 6

Even though I have used the latest version I don’t see any problems which you might face in carrying out the same set of steps for JDK 1.5 which I am about to explain. JDK comes shipped with a keytool executable which is required to generate a keystore. The keytool can be found in the earlier version of JDK too. The 3 steps which would make you to get started with setting up SSL are:

1. Generating the Keystore file
2. Configuring Tomcat for using the Keystore file
3. Configuring your web application to work with SSL

Let’s get this party started now.

1. Generating the KeyStore file

The keystore file is the one which would store the details of the certificates necessary to make the protocol secured. Certificates contain the information as to who is the source from which you are receiving the application data and to authenticate whether it is the intended party or not. To make this keystore you would have to use the keytool. So open command prompt in Windows or the shell in Linux and type:

cd %JAVA_HOME%/bin on Windows

cd $JAVA_HOME/bin on Linux

You would land up in the Java bin directory. Now time to run the keytool command. You have to provide some parameters to the command as follows :

keytool -genkey -alias techtracer -keypass ttadmin -keystore techtracer.bin -storepass ttadmin

The highlighted words are the ones which you would have to change according to your requirements. But keep one thing in mind that both the keypass and storepass passwords should be the same. The .bin file is actually your keystore file. It would now start a questionnaire. So fill in the relevant details accordingly. Look below for a reference as to what to answer for the questions.

What is your first and last name?
[Unknown]: shekhar raj
What is the name of your organizational unit?
[Unknown]: home
What is the name of your organization?
[Unknown]: techtracer
What is the name of your City or Locality?
[Unknown]: bangalore
What is the name of your State or Province?
[Unknown]: karnataka
What is the two-letter country code for this unit?
[Unknown]: IN
Is CN=shekhar raj, OU=home, O=techtracer, L= bangalore, ST=karnataka, C=IN correct?
[no]: yes

The command would then conclude. It would make a .bin file with the name you had provided inside the bin directory itself. In my case it was techtracer.bin which was located in

C:\Program Files\Java\jdk1.6.0_02\bin\

Put the .bin file in the webapps directory of Tomcat. This is required to avoid the need to give an absolute path of the file in the next step.

2. Configuring Tomcat 6 for using the Keystore file

Here we would be making some changes to the server.xml file inside tomcat to tell it to use the keystore which was created in the earlier step for configuring SSL. Open the file server.xml which can be found as:

/conf/server.xml

Now you have to modify it. Find the Connector element which has port=”8443″ and uncomment it if already not done. Add two lines. The highlighted lines are the newly added ones.

keystoreFile=”../webapps/techtracer.bin”
keystorePass=”ttadmin” />

You can notice that I have given the path to the keystoreFile property as relative to tomcat bin directory because the startup command will look for the .bin file. Now all you have to do is start your server and check the working of SSL by pointing your browser to the URL to:

https://localhost:8443/

Now that you have your tomcat running in the SSL mode you are ready to deploy an application to test its working. You must note that still your tomcat can run in normal mode too at the same time i.e on port 8080 with http. So it is but obvious that any application deployed to the server will be running on http and https at the same time. This is something that we don’t want. We want our application to run only in the secured mode.

3. Configuring your web application to work with SSL

In order to do this for our test, take any application which has already been deployed successfully in Tomcat and first access it through http and https to see if it works fine. If yes, then open the web.xml of that application and just add this XML fragment before web-app ends i.e

securedapp
/*


CONFIDENTIAL

Explanation of the fragment is beyond the scope of this tutorial but all you should notice is that the /* indicates that now, any resource in your application can be accessed only with https be it Servlets or JSP’s. The term CONFIDENTIAL is the term which tells the server to make the application work on SSL. If you want to turn the SSL mode for this application off then just turn don’t delete the fragment. Just put the value as NONE instead of CONFIDENTIAL. That’s it!

Conclusion

These were the 3 easy steps in which you can make Tomcat 6 to work in the SSL mode and also it tells you how easily you can turn the SSL mode on and off. If you find any difficulty or are not clear on any of the above steps feel free to drop in your queries. If you like this tutorial it would be nice of you to drop in a comment of appreciation or feedback as to how this tutorial can be improved.

SSL Support to Tomcat 6

I recently had an enjoyable experience trying to figure out how to configure SSL support within Tomcat. I figured it would be pretty easy, but I bumped into some troubles along the way so it ended up taking a couple hours. The key issue is that there isn't a way to use Java's keytool to import a private key for an SSL certificate. Now that I have a handle on things and SSL is working, I figured I would post a walkthrough to help anybody who may be running into the same roadblock.

My Environment

• I am currently using Apache Tomcat 6 as a stand-alone web server
• I am using Java JDK 6, Standard Edition
• My web host provides Windows Server 2003 64-bit (Getting Tomcat to run as a 64-bit Windows service is a story for another day)

If you are running in a similar hosting environment, these steps should guide you through setting up SSL.

Step 1

The first step is to get a copy of your private key and a certificate for that key. You will also need a copy of the root certificate from your CA (such as VeriSign or GeoTrust). These were provided to me in Base64 format by my web host, and you should save these all as .pem files.

• My web host first provided me with a private key. I downloaded it as a .txt file, and then proceeded to open it using notepad. The file started with
  -----BEGIN PRIVATE KEY----- and ended with
  -----END PRIVATE KEY----- with a block of text in between. Take this file and save it as private.pem (you can either just rename the file you download, or copy and past the text into a new file).
• My web host then ordered the SSL certificate for me, and a little later I was sent an e-mail containing a similar chunk of text as before. The difference was that it started with
  -----BEGIN CERTIFICATE----- and ended with
  -----END CERTIFICATE-----. Again, open notepad, paste the text, and save this file as cert.pem
• Lastly, you will need to obtain a copy of the root certificate. My SSL certificate came from GeoTrust, and after a bit of searching I found their root certificates are available from their web site. Downloaded the certificate in Base-64 encoded X.509 format and save this file as root.pem

Step 2

We now have the keys and certificates that we need to configure SSL on our server. However, we need to convert them into a format that Tomcat supports. For this, I used a free tool called OpenSSL. Among other capabilities, OpenSSL will help you convert keys between different formats, which is exactly what we're looking for. I'm lazy and didn't want to compile the source code myself, so after a bit of searching I found that Shining Light Productions provides the binaries for Windows. Download this package and use the installer to get everything setup.

Step 3

Once OpenSSL is installed, you must take the three .pem files and combine them into a single .pem file. First, open notepad and copy the contents of root.pem into the file. Next, copy the contents of cert.pem on to the next line. Last, copy the contents of private.pem at the end. The data should look something like this (with more text in between):

-----BEGIN CERTIFICATE-----
MIIDIDCCAomgAwIBAgIENd70zzANB
1voqZiegDfqnc1zqcPGUIWVEX/r87
yloqaKHee9570+sB3c4
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDIDCCAomgAwIBAgIENd70zzANB
1voqZiegDfqnc1zqcPGUIWVEX/r87
yloqaKHee9570+sB3c4
-----END CERTIFICATE-----
-----BEGIN PRIVATE KEY-----
MIIDIDCCAomgAwIBAgIENd70zzANB
1voqZiegDfqnc1zqcPGUIWVEX/r87
yloqaKHee9570+sB3c4
-----END PRIVATE KEY-----

Save this file as ssl.pem.

Step 4

Open up a command prompt window and navigate into the "bin" directory of your OpenSSL installation. We want to take the ssl.pem file and convert it into a PKCS12 keystore, which we will call ssl.p12. This can be accomplished by running the following command:

openssl pkcs12 -export -in ssl.pem -out ssl.p12 -name tomcat

Note: Be sure to use the correct paths for your ssl.pem and ssl.p12 files. You will be prompted to create a password for this keystore.

Step 5 (Optional)

You can verify the PKCS12 conversion worked by using Java's keytool command. First switch to the "bin" directory of your JDK and run:

keytool -v -list -keystore ssl.p12 -storetype pkcs12

Note: Be sure to use the correct path for your ssl.p12 file. You will be prompted to enter the password you created earlier. Keytool will then list out the contents of the keystore. Look towards the top of the output to ensure that the keystore type is PKCS12, that the keystore contains 1 entry, the entry type is a PrivateKeyEntry, and the certificate chain length is 2.

Keystore type: PKCS12
Keystore provider: SunJSSE

Your keystore contains 1 entry

Alias name: tomcat
Creation date: Jun 16, 2007
Entry type: PrivateKeyEntry
Certificate chain length: 2


Step 6

Now we just need to configure Tomcat's server.xml file to use this keystore. First, take the ssl.p12 file and store it on your web server. Then navigate to the "conf" folder within your Tomcat installation directory. Open the server.xml file in notepad and add the following connector:

Note: Be sure to use the correct path to your keystore and the correct password. Depending on your setup, you may need to use a different connector configuration. In particular, Tomcat needs a different setup if you are using the Tomcat Native Library. Please consult the Tomcat documentation for more details.

Step 7

Reboot the Tomcat server and try accessing your web site using https. If everything is configured correctly, you should be able to successfully view your web site without any errors or warnings.

Conclusion

Hopefully you found this walkthrough to be helpful. I am aware that most of these steps are pretty specific to my environment and how my web host provided me with my SSL certificate. However, I would think with some hacking around you can adapt these steps to suit your needs. If you run across any difficulties or these steps don't work for your situation, feel free to post a comment describing your environment and I will do my best to provide some assistance .

Apache Tomcat Tuning Tips

Tomcat native with user-specified open SSL libraries

Looks like that Tomcat native can not work without the openSSL support. so we are going to build ssl also as part of our APR package. As usual we will start with APR.
Download the apr 1.2.7 bundle
(1)
./configure --prefix=/oad/x/apr
make
make install
(2)
Then go a grab a copy of openssl sources from www.openssl.org. I get 0.9.7f tarball, the version that corresponds to my Fedora core 4 install.

./config --prefix=/oad/x/apr --openssldir=/home/rajeevj01/x/tmp shared
make
make test
make install

The documents and tests would be installed in x/tmp location and libraries are installed in /oad/x/apr location.
(3)

Next, we un-tar and build the tomcat-nativve1.1.2 source ball with the following options.

[rajeevj01@bdc31035e native]$ ./configure --prefix=/oad/x/apr --with-ssl=/oad/x/apr --with-apr=/oad/x/apr --with-java-home=/home/rajeevj01/sw/jdk1.5 --with-java-platform=2

(4) Now, set up the tomcat server to use this native library , Add the following in $TOMCAT_HOME/bin/setenv.sh script (or equivalent)
------------------------------------------------------------------------------------------
LD_LIBRARY_PATH=/oad/x/apr/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH
-------------------------------------------------------------------------------------------


rajeevj01 at 2:41:57 AM EDT Link to this entry | Blog about this entry | Notify AOL
This entry has 0 comments: Add your own


Building APR library on Feodra core 4 Worried

The libtcnative (The native tomcat connector) requires the openSSL and APR libraries headers to be built. so the first step is : Download and install the APR. I head over to http://apr.apache.org and grab the latest tarball. untar it and just run configure with the required prefix
1051 ./configure --prefix=/oad/x/apr
1052 make
1053 make install

APR is now installed. we are planning to use AJP connector and so we build the non-SSL version of tomcat AJP/13 connector.

[rajeevj01@bdc31035e native]$ ./configure --prefix=/home/rajeevj01/x/tcnative126 --with-apr=/home/rajeevj01/x/apr126 --without-ssl --with-java-home=/home/rajeevj01/sw/jdk1.5 --with-java-platform=2

if we do not want to use SSL, we can go ahead and remove the -lssl flag from the makefile also. But if you do that, the tomcat can not find the APR installed. SSL must be enabled for APR (why?)
There are some version issues with tomcat 1.1.1(bundled with tomcat 5.5.16) and apr 1.2.7 , so download the tomcat 1.1.2 sources from
http://tomcat.heanet.ie/native/1.1.2/source/

1100 make
1101 make install

This builds the native tc library. Once we have the library, the next big question is how to use it.


rajeevj01 at 4:18:18 AM EDT Link to this entry | Blog about this entry | Notify AOL
This entry has 0 comments: Add your own

Ethereal for http packet captures Silly

we need some packet capture tool to analyze the server headers and what the browser is sending to the server. After using couple of useless freeware tool, now i have settled on ethereal. it can do a lot of stuff and options seem quite daunting but the good thing is i can make it work to my purpose with minimal fuss.

Download and install ethereal and read the user guide anyway. To get up to speed very quickly,
1) start ethereal , to start capture , Go to Menu | Capture |Options
2) on this screen, select the Interface (like VPN adapter or intel ethernet card etc)
3) in filter box, type tcp port 5400
one more filter that you can try is tcp port 5400 and host 10.141.11.189
etc.

This will capture all traffic to and from that host going through your designated interface.

4) I do not like to see lots of colors , so Menu | View | colorize packet List
This lets you switch on/off the packet colors in display.

5) You can save the packets captured during the session with Menu | File | Export

6) In the capture window, each packet can be probed further for frame details etc.

All in all a really useful tool and w/o any freeware shit!!

rajeevj01 at 7:29:51 AM EDT Link to this entry | Blog about this entry | Notify AOL
This entry has 0 comments: Add your own


webchat GC tuning Loopy

okay, so we have done one round of our JVM garbage collection data capture and this is what i can infer:

+ The eden space size is small , so minor collections are happening every now and then.
+ survivor space is not big, so after minor collections overflow is happening to tenured space at a very young age

So this means, i should start with a bigger young generation. Plus i am running a server application, so i will go with same -Xms and -Xmx options.

Here are my GC parameters
==============================================================
-Xss256K -Xms512M -Xmx512M -XX:NewRatio=2 -XX:SurvivorRatio=6 -XX:+UseParallelGC
=============================================================
I may want to try with -XX:NewSize=256m -XX:MaxNewSize=256m

so that young generation size is half the heap size to start with. Right now, i will settle for young:tenure ratio as 1:2.

with young generation = half the heap size we get
Young = 256 MB, Eden = 6 * (256/8) == 192 MB. survivor space is 32 MB each.

=================================================================
To check the jvm stats, we use
$jstat -gcutil > jstat.log
like:
$jstat -gcutil 200 > jstat.log

we can get the vmid using the $jps command.

rajeevj01 at 7:22:48 AM EDT Link to this entry | Blog about this entry | Notify AOL
This entry has 0 comments: Add your own


How to make the eclipse profiler work for you
Silly



First things first. we want to do remote profiling and we do not want lots of over heads. The option was there to run one of the standard tools like optimizeIt or Jboss profiler etc. However, i could not set up Jboss profiler quickly enough and optimizeIt would have required tomcat to start with its own audit system.

So finally i settled for eclipse plug-in profiler. This plugin can attach to remote servers. It just needs one library file and few command line options on server.

Required server changes

A) catalina.sh script or the script that starts you server will require the following chages

this one is for CPU profiling, see the link for more details on memory profiling etc. http://eclipsecolorer.sourceforge.net/index_profiler.html

JAVA_OPTS="-XrunProfilerDLL:1 -Xbootclasspath/a:jakarta-regexp.jar:profiler_trace.jar:commons-lang.jar \
-D__PROFILER_PACKAGE_FILTER=__A__%MAINCLASS%:__M__sun.:__M__com.sun.:__M__java.:__M__javax.:__M__org.apache. \
-D__PROFILER_TIMING_METHOD=1"

For our setup, we need to include these options in post.tcl file along with other java command line options . This is a jvmpi plugin so some of the GC options you require may not be supported.

B) copy common-lang.jar, jakarta-regex.jar and profiler_trace.jar to $tomcat_home/bin folder.
in our case, this may mean the folder from where we bootstrap the tomcat application. so copy the 3 jar files to the port folder like to 48800/ folder.

C) compile the agent using the following script ( Better to compile on target server ). The default agent is compiled with gcc 3.2 and it may not work on all systems.

g++ -O0 -DLINUX -shared -Wall -I. -I/home/rajeevj01/sw/jdk1.5/include
-I/home/rajeevj01/sw/jdk1.5/include/linux ProfilerDLL.cpp -o libProfilerDLL.so


D)copy the agent file to JDK_home/jre/lib/i386 folder
or you can put it in LD_LIBRARY_PATH also.

After making the changes, start the server. check logs for any errors. Once server startup is successful, we are ready to attach to this server to get profiling data.

Eclipse changes :

You need to do few changes to your eclipse before you can open the profiler window.


A) we require a new plugin. download it from the web link http://eclipsecolorer.sourceforge.net/index_profiler.html and copy to your eclipse plug-ins folder. Then from menu | windows | open perspective | profiler
B) you need to create a launcher configuration to connect to remote server. To do that , open Menu | Run and choose Run .. option. under remote profiler node, create a new configuration by clicking New button. Add the server name and click on Run button. thats it!!!!

C) you have to patch the plugin for eclipse 3.1 version.
link is
http://sourceforge.net/tracker/index.php?func=detail&aid=1202373&group_id=48823&atid=454283

D) You may require UNRAR on windoze machines. see unxutils zip on sourceforge. http://unxutils.sourceforge.net/


rajeevj01 at 8:15:30 AM EDT Link to this entry | Blog about this entry | Notify AOL
This entry has 0 comments: Add your own


Roadmap : for doing server performance tuning
Silly



1) chat is working okay for so-called 400 simultaneous users. we do not know when we get the apache on chdl-m05 blue box. so in the mean-while this is the plan i have for performance testing

A) we need to profile the application and see what is taking time. Right now, there are too many synchronization blocks around!! For profiling , we can take a look at JProbe/OptimizeIt or Jboss profiler, whatever works for us.

B) Shut down apache debugging and DNS lookup etc. point is you run a very lean server. should we compile the mod_status module at this point ? Linux server data should give enough indications at this point!

C) we need to do tomcat and JVM monitoring. we can do the monitoring using MC4J.
D) we capture the linux box stats for different combinations.

We need snapshots of following data
0) remove all the debugging options and make a lean server.
1) Active thread count

2) open sockets
This number should be less than my $ulimit -n output.
3) cpu utilization
4) memory ( I should not be swapping )

5) JVM heap status
we can use the jstat tool. what we want to understand is how much of the heap we are actually using under projected loads. This will give us a better idea of our heap usage. Also, i would like to know if i am doing lots of GC around.

The link for jstat is http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jstat.html
see the effect of using -server option etc.

6) Apache status
we can use mod_status for this purpose (?)

The parameters to vary are

1) apache server threads + number of child processes
2) JVM heap size, -server option, -Xincgc option
3) Thread stack size for apache as well as JVM threads



This entry has 3 comments: Hide Recent | Add your own

kernel-imposed limits on the system
-------------------------------------------
1.fd-max can be got at: cat /proc/sys/fs/file-max
2. threads-max can be got at: cat /proc/sys/kernel/threads-max

Comment from shrutikbdc - 6/5/06 7:56 AM

top command
-----------------

top -b -d 2 > top_stats

This redirects the output of top to a file in plain text format every 2 seconds


Comment from shrutikbdc - 6/5/06 7:45 AM

Memory

vmstat ; say "vmstat 2" gives stats every 2 seconds

The following can be captured from vmstat output:

Procs
r: The number of processes waiting for run time.
b: The number of processes in uninterruptable sleep.
w: The number of processes swapped out but otherwise runnable. This
field is calculated, but Linux never desperation swaps.

Memory
swpd: the amount of virtual memory used (kB).
free: the amount of idle memory (kB).
buff: the amount of memory used as buffers (kB).

Swap
si: Amount of memory swapped in from disk (kB/s).
so: Amount of memory swapped to disk (kB/s).

IO
bi: Blocks sent to a block device (blocks/s).
bo: Blocks received from a block device (blocks/s).

System
in: The number of interrupts per second, including the clock.
cs: The number of context switches per second.

CPU
These are percentages of total CPU time.
us: user time
sy: system time
id: idle time
mod_jk connector to plug jetty/tomcat in apache Loopy



we are going to use mod_jk to connect apache and jetty.
step1) download and copy mod_jk.so to /usr/lib/httpd/modules
Download link is:
http://tomcat.apache.org/download-connectors.cgi
Beware : do not download jk2 module, download jk module. This is very confusing!
jk is the one that we need not jk2. You may want to rename the jk .so file.

step2) add LoadModule directive to the apache httpd.conf file
[ Add where other LoadModule directives are ]
LoadModule jk_module modules/mod_jk.so

Beware : The way to add modules may differ in apache 1.3 and apache 2. I dont think you require anything more than above one line with apache 2 that has been enabled with DSO support.

Question: How do i check if my apache has been compiled with DSO support?
[rajeevj01@bdc31035e installs]$ /usr/sbin/httpd -l
should list mod_so.c

step3) go to /etc/httpd/conf.d directory
[ comment: this may be specific to FC4 , if such a folder is not there then you have to specify the path to mod_jk.conf in apache httpd.conf file using an Include directive, like,
Include mod_jk.conf
]

create a mod_jk.conf file there with the following content
--------------------------x------------------------------------------------
# rjha: adding the mod_jk options
JkWorkersFile /etc/httpd/conf/workers.properties
JkLogFile /home/rajeevj01/logs/mod_jk.log
JkLogLevel debug
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
JkMount /servlet/* ajp13

------------------------------x-----------------------------------------------

Replace /servlet/* with your dymanic scripts context path.

step 4) create the logs dir if that does not exist
step5) create the workers.properties file with following info
-------------------------x-----------------------------------------------------
ps=/

# rjha: sample worker.properties lifted from GERONIMO
worker.list=ajp13
worker.ajp13.port=8009
worker.ajp13.host=localhost
worker.ajp13.type=ajp13

step 6) Now configure your servlet container to listen on AJP13 port.

A) for jetty do the changes in $jetty_home/etc/jetty.xml

rjha : uncommenting the AJP13 listener -->

8009
5
20
0
443

B) for tomcat , do the changes in server.xml file. comment out the HTTP connector and uncomment the AJP13 connector portions.

Thats it. Now next set of things to do :
1) know more about jetty logging options
2) know more about mod_jk logging options
3) how servlets are deployed on jetty, the context mappings

Then we can start changing and debugging the jetty code.

Deploying the dit dot servlet Happy

The dit dot servlet just prints one dot every 1/2 seconds. Point is to replace the chat pipe servlet with this dit dot servlet on my machine and do the stress testing. I am going to deploy the dit dot servlet on my development box, with tomcat (ditch jetty for the moment). so the next steps in my wave of experiments is

1) Download and install the latest tomcat (preferably from sources ?) no ? binary should do at the moment. download sources as a side-kick.
2) deploy the dit dot servlet with tomcat container
3) deploy the rest of the chat apps
4) Now do the JVM stats monitoring , things like, how much of memory i am munching, how many threads etc. Need to find out more about this.

============= web.xml file ==============================

VanillaPipeServlet
com.bobo.chat.VanillaPipeServlet
VanillaPipeServlet
/pipe

We want to monitor our tomcat application JVM. The development boxes i have are linux boxes. On these machines, we just have to add the following 4 lines to JAVA_OPTS in catalina.sh script
====================================================
# rjha: supply this argument to JVM on start-up
# we want to monitor tomcat JVM remotely
# development system, no password!
JAVA_OPTS="-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=9421 \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false"
===================================================
No authentication, no passwords, i dont give a Damn at the moment.
Then you should have the JDK 1.5 installed on your machines.
on the monitoring machine, just fire jconsole and connect to target VM.
Right now, remote monitoring not working, only local monitoring is working.
I can monitor the number of threads now!!!!

Areas of performance monitoring of Tomcat Silly

Sometimes you would like to know how your tomcat server is doing. How many threads are open on container etc. There are two ways you can do this

1) Use the server status link on tomcat manager application
log into tomcat manager application. you will see a Server Status link on top of application's list.

2) Use one of the JMX monitoring tools. catalina container properties can be exposed as JMX beans and we can use one of the JMX monitoring tools. I cover JMX tools list in a separate document.

MC4J one of the tools that we are going to use (covered in separate blog) . we need to check out visualGC also. plus there is a tool called HAT for heap analysis. I need to pay attention to JVM tunining next. But for the starters, here is the points to ponder when tuning tomcat

1) server configuration file. Need more info on this from the tomcat user's guide.
http://tomcat.apache.org/tomcat-5.5-doc/config/index.html
The config parameters should tell me about
maximum processes, acceptCount, max-threads etc. Need to do RTFM here.

2) JVM parameters may play a big role here, so we need some info on JVM tuning also
+ Thread stack size
+ head min/max size
+ GC (incremental/ how much allocated to generational space)
+ -X arguments to JVM @ startup
-Xincgc (for incremental GC)

3) Threading support
How do i find out if my VM is using NPTL libs or not ? start with the linux kernel version
$uname -r
if you are running a 2.6.x kernel then you should be ok.
use this small program to print java library paths
================================================
public class PrintLibraryPath {
public static void main(String[] args) {
System.out.println(System.getProperty("java.library.path"));
}
}
====================================================
Then we can use ldd to see if NPTL is linked in our JVM or not.
[rajeevj01@bdc31035e jdk1.5]$ ldd /home/rajeevj01/bin/jdk1.5/jre/lib/i386/client/libjvm.so
linux-gate.so.1 => (0x00835000)
libm.so.6 => /lib/libm.so.6 (0x00111000)
libdl.so.2 => /lib/libdl.so.2 (0x003e4000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00219000)
libc.so.6 => /lib/libc.so.6 (0x0022b000)
/lib/ld-linux.so.2 (0x00864000)
[rajeevj01@bdc31035e jdk1.5]$ nm /lib/libpthread.so.0 | grep nptl
NPTL is linked!!
---------------------------------------------------------------------------------------------------

+ switch off servlet reloading
+ compile jsps a-priori in production usage. use JSPC compiler.
+ we are not using tomcat in stand-alone mode, so *comment* out that connector.

we need something to monitor the tomcat! something to monitor the JVM.
Use jconsole to see how the JVM is performing. Use servlet statistics page to see how the tomcat installation is doing.

Tomcat configurations Loopy

+ you should not allow any tracing/debugging options
+ you should disable the servlet re-loading
+ The number of threads should be tuned on the server. is there a way to increase the maxThreads programatically ?

maxKeepAliveRequests

The maximum number of HTTP requests which can be pipelined until the connection is closed by the server. Setting this attribute to 1 will disable HTTP/1.0 keep-alive, as well as HTTP/1.1 keep-alive and pipelining. Setting this to -1 will allow an unlimited amount of pipelined or keep-alive HTTP requests. If not specified, this attribute is set to 100.
maxSpareThreads

The maximum number of unused request processing threads that will be allowed to exist until the thread pool starts stopping the unnecessary threads. The default value is 50.
maxThreads

The maximum number of request processing threads to be created by this Connector, which therefore determines the maximum number of simultaneous requests that can be handled. If not specified, this attribute is set to 200.
minSpareThreads

The number of request processing threads that will be created when this Connector is first started. The connector will also make sure it has the specified number of idle processing threads available. This attribute should be set to a value smaller than that set for maxThreads. The default value is 4.

DescriptionacceptCount

The maximum queue length for incoming connection requests when all possible request processing threads are in use. Any requests received when the queue is full will be refused. The default value is 10.

java.sun.com has lots of articles on how to tune JVM settings. crux of the matter is that you should be able to
+ size up a heap for your purposes
+ thread stack size and numbers for your purpose

Right now, we are running with
-Xms128M -Xmx256M -Xincgc -Xss128k

you can do the required arithmetic. try tuning with server mode also, the warm-up may take some time but after that it should be ok!
use some JMX remote management tool like MC4J etc to monitor the health of the JVM as well as tomcat instance running on top of it.

This is a good reference to see what all options are available with JVM
http://blogs.sun.com/roller/resources/watt/jvm-options-list.html
this one discusses the JVM threads
http://java.sun.com/docs/hotspot/threads/threads.html
and this one tuning the GC
http://developers.sun.com/techtopics/mobility/midp/articles/garbagecollection2/
and finally we have the JVM tuning whitepaper
http://java.sun.com/performance/reference/whitepapers/tuning.html

Threads tuning params on linux 2.6

The issues here is to find out the system limits and application limits imposed on a running program on linux 2.6 kernel series. Later we see how we can tune the parameters to spawn a lot of threads and do fast context switching as well.

* Application limit tuning *
==================
1) ulimit -s and ulimit -u
ulimit has options to set thread stack size.
check by typing ulimit -a. looks like this is not system wide setting but just applies to applications started in that bash shell. you can not go beyond the hard-limits of a parameter. The hard limits can be checked by typing ulimit -aH.

2) you can set linux thread stack size using
$ulimit -s

Then you make sure that you use the same value with java -Xss option when starting your JVM. That way the thread runs in a limited stack space and more and more threads can be spawned.

* System limit tuning *
================
1) check the system wide settings in following places
$ cat /proc/sys/kernel/threads-max
$grep totalpages /var/log/dmesg (totalpages/16 is the max # threads possible)
$ cat /proc/sys/kernel/pid_max
$cat /proc/sys/vm/max_map_count (* need to figure out what exactly is this *)

* test program *
=============

3) Run the test program to see how many threads are supported on your machine

Idea is , final limit on # max threads should only be imposed by the amount of RAM in system.

4) How to do system wide ulimit settings ?
check the contents of file
/etc/security/limits.conf

Bumping the ulimit Happy

The ulimit command has two variants. ulimit-aH and ulimit -aS. H is for the hard options (what is set in the config files for the user) and soft is the current value.
Now if you are root its all pretty ok,you can just use ulimit command to increases whatever you want.

for other logins, we can not bump up the resources more than the hard limits. For such cases, we have to first increase the hard-limits by changing the /etc/security/limits.conf on linux 2.6 systems.
=================================================
Edit /etc/security/limits.conf and add the lines:rajeevj01 soft nofile 1024 rajeevj01 hard nofile 65535

Also check the values in /proc/sys/fs/file-max.
[ on 2.6.x systems, they are pretty ok!]
==================================================

After bumping up the hard limits, use ulimit command for user login

$ ulimit -n unlimited

A: max address space (KB)
C: max core file size (KB)
D: max data size (KB)
F: maximum filesize (KB)
M: max locked-in-memory address space (KB)
N: max number of open files
R: max resident set size (KB)
S: max stack size (KB)
T: max CPU time (MIN)
U: max number of processes
K: file creation mask, set by umask(2) .
L: max number of logins for this user
P: process priority, set by setpriority(2) .

For example, L2D2048N5 is a valid LIMITS_STRING. For reading convenience, the following entries are equivalent:

username L2D2048N5
username L2 D2048 N5

So far not able to locate any remote monitoring tools for linux box. Most of the scripts there can be put as part of cron and flushed to log files. I can use a user-specific crontab files!!
+ Install the redhat sysstat package $yum install sysstat
+ need to do some research on SAR and crond to achieve the desired results.
other useful commands are
ps,iostat, vmstat, top ,mpstat [ But the hitch is i have to be on the machine.]

One good link is http://people.redhat.com/alikins/system_tuning.html
Apart from that try reading the RedHat sysadmin guide also
http://www.redhat.com/docs/manuals/linux/RHL-9-Manual/admin-primer/index.html
Need to do some research on flushing cron jobs

So its agreed , to use netstat to monitor the number of open sockets and the above utilities to monitor the system health. since i am going to run all the experiments on LAN, i am not going to monitor the bandwidth usage as such!

I still can not locate any tool that allows for remote monitoring. anyways, i am not going to break my head over it. mostly, this is what i need to do
A) How to monitor
+ open processes
+ open threads
+ memory
+ I/O, open sockets and open file descriptors.

I think the most important is memory monitoring. when i check my /proc parameters, I dont think i am going to run out of fds and processes in a hurry! So kernel-limits are all ok. So we got to watch memory and CPU after bumping the user limits of fd and thread stack size using ulimit command in a shell.
To see the open sockets on my machine , I use this command lifted from red hat's site:
=====================x===================================

One of the most twisted, but useful invocations is:
$netstat -a -n|grep -E "^(tcp)"| cut -c 68-|sort|uniq -c|sort -n
This will show you a sorted list of how many sockets are in each connection state.
=====================x====================================
To see the memory and CPU usage , i am relying on top. for a quick snapshot , you may consider using vmstat also.
$vmstat 1 5
gives 5 readings with 1 sec interval.

B) gnome-system-monitor is a graphic utility that can be installed using yum if it is not there on your system. I have FC4 and gnome-system-monitor is installed. This shows the process list and memory usage.

C) top, watch, free , ps etc. are usual unix commands.

The following gives a recipe to bump the number of allowed file descriptors.
1) setting up the hard limit in /etc/limits.conf
rajeevj01 nofile hard 2048
2) Then executing ulimit -n 2048 in a shell should increase the number of file descriptors. on recent systems , /proc/sys/fs/file-max is good enough for all the purposes (limited by hardware only). on my machine it is ~ 101k.
3) apache or tomcat uses some start-up scripts like apachectl or catalina.sh etc (or if started from /etc/rc.d/httpd etc)
There, in the beginning of such scripts we can set the soft ulimit. (make sure the hard-limits have been bumped for apachectl user)
=========================================
ulimit -S -n `ulimit -H -n`
=========================================
if you are using PAM, then you also need to do the following to honor the limits.conf
edit /etc/security/limits.conf and add to any config file in /etc/pam.d/ that handles logins ("login" and "ssh" example):

session required /lib/security/pam_limits.so

1) packet analyzer/tcp mon
To see what the server is sending and what the browser is receiving. should show the files/size/transfer time plus all the headers. Useful for debugging.
2) bandwidth monitor and throttler
what is my download speed right now, how can i make my speed simulate a modem speed.
3) network load/stress monitor
How is my network pipe doing. what kinds of stress it is subjected to. is my webserver saturating my network card?
4) Javascript debugger
how to use venkman. microsoft script debugger. get rid of alert debugging.
5) Javascript compiler/size muncher
6) web server log analyzer, web server and servlet container tuning
Tomcat, Apache, Weblogic
7) OS resources monitoring and tuning.

Apache connection tuning Silly

The apache 2 has 256 MaxClients limit out-of-box. The compiled-in hard-limit of MaxClients(ServerLimit) is 20,000 (way beyond our purpose). First we will look at some of the apache directives used in httpd.conf to tune the server.

1) ServerLimit : This goes hand-in-hand with the MaxClients.
For prefork MPM , this is number of MaxClients. in case of worker MPM , this together with ThreadsPerChild determines the MaxClients
for worker MPM
MaxClients = ThreadsPerChild * ServerLimit

2) ThreadPerChild
number of threads created for a apache child process. Default is 25. Hard Limit can be 64 determined by the ThreadLimit.

3) MaxClients : This is the number of simultaneous requests that apache can handle.

Below are some of the real world setting examples (This is one of the busiest sourceforge mirrors)
=============================================

ServerLimit 900
StartServers 10
MaxClients 28800
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 32
MaxRequestsPerChild 100

=================================================
As you can see, MaxClients = ServerLimit * ThreadsPerChild
=================================================

StartServers 100
MinSpareServers 10
MaxSpareServers 10
ServerLimit 50000
MaxClients 50000
MaxRequestsPerChild 2000

=================================================
I do not know if we can start so many apache processes on a machine, even theoretically.
On my Linux machine,
[rajeevj01@bdc31035e ~]$ cat /proc/sys/kernel/pid_max
32768
So this number needs to be tuned first. Plus of course there is

+ this issue of open fds
+ and linux thread stack size
+ and increasing the number of processes limit in shell using the ulimit command.

The next example config is taken from AOL search team
=====================================================

## Worker MPM settings
# AOL: Fixed 2 servers and 200 threads each for a total of 400 max
# concurrent requests. Can be expanded to 1000 * 2 = 2000 via a
# graceful httpd controlled restart.

ServerLimit 2
ThreadLimit 1000
StartServers 2
MaxClients 400
MinSpareThreads 200
MaxSpareThreads 400
ThreadsPerChild 200

====================================================

After going through all such examples lets find a configuration that will allow us to run 1k simultaneous connections. This is our first goal. And perspective gained from here will determine how we go ahead with 2k simultaneous connections.


For our EC 2.0 case lets try this setup:
ServerLimit 8
ThreadLimit 256
StartServers 2
MaxClients 1024
MinSpareThreads 128
MaxSpareThreads 256
ThreadsPerChild 128

To achieve these numbers, we would need to
+ tune the number of file descriptors
+ and linux thread stack size.

Number of threads should not be a problem. we can try tuning serverLimit to 4 and increasing the ThreadsPerChild. But only more data can tells us how to proceed ahead with tuning.


Finally, we have installed apache 2.0 with worker MPM. all you need to do is --with-mpm=worker switch when using the configuration. using the worker MPM, we can start apache with 4k threads (MaxClients). see the blog about apache tuning to know more.

Idea is to start 4 AJP workers with 1k threads each and interface it with apache. Here we have a caveat. we have no way of distinguishing one apache request from another. so the standard tricks of 1 AJP worker for one URL is not going to work for EC2. So initially i am going to set up 1 apache - 1 AJP connector for 1k connections.

we need 4k + delta for apache instances and and 1k+delta /tomcat , so in total we are talking about 4 JVMS of 1200 threads each !!!

But before we dive into JVM arithmetic, we need to know how to setup AJP13 with 1k connections. here , the trick is to use MaxThreads setting for AJP connector in server.xml file. But then what maxProcessors are doing ?

There were some discussions on tomcat list about maxProcessor attrib. But the long and short of it is that I just need to set MaxThreads and i should be okay. I dont see a need to use APR (apache portable runtime), because in my case i will be holding on to the connections. so threads are not wasted.

Now, arithmetic

1 JVM - 1200 threads
Thread stack size 256k => 8 threads /MB so 256 MB means 1k Threads
so we start JVM with minHeap 256 MB, MaxHeap 512 MB
512 MB x 4 = 2 GB for running 4 JVMs.
Thread stack size can be tuned to 128K also. ($ulimit -s 128)

Lets try with one apache + one JVM first!

One apache that can handle 1k simultaneous is covered in apache tuning blog.


chat application is a push type application. we open a socket to server and browser keeps getting data on that open connection. Here, the browser and server should not cache at any point because you need to push data to users as soon as they are available.

To enable this behavior we need to set some response headers
You will need an equivalent of these apache server headers
========================================================
Header set Expires "Sat,6 May 1995 12:00:00 GMT"
Header set cache-control "no-store, no-cache"
Header append cache-control "must-revalidate , no-transform"
Header append cache-control "pre-check=0, post-check=0"
Header set Pragma "no-cache"
=========================================================

when we pluged mod_deflate module in our apache installation , we never realized that we would hit the caching issues because of compression ;o)

mod_deflate is to compress the response, in our case it can not work till the response is complete. So to do compression, server will keep caching the data coming from chat backend and only flush it when the response is complete.

So now we need to do compression based on some conditions like some Directories only etc.


To do List for server tuning

step 1) get a development box that has RH 4.0 installed. we need RH 4.0 because we need linux 2.6.x kernel series. verify the kernel with $uname -r

step2) we need to bump up the file descriptors limit. we need to raise a ticket for this. verify the hard limits and then again verify that you can actually bump up the file descriptors numbers. we can get 8k or maybe 16k descriptors /process/shell.

3) use this program to verify
======================================================================================
[rajeevj01@bdc31035e sandbox]$ cat fd.c
#include
#include
#include

void main(void) {
int k = 3;
while (dup(0) != -1)
k++;
printf("Opens = %d Fdsize = %d\n", k, sizeof(fd_set) * 8);
}

============================================================

4) change the thread stack size with $ulimit -s
change it to 128k or 256k. for the moment lets stick with 256k.
Again we can verify the number of threads we can spawn with a program.

5) Now fds and threads are okay. we need to move to apache settings. The apache setting that will let us use 1k simultaneous connections have been covered in a different blog.
Make sure we run a lean apache. check apache 2.0 perf docs also. Also make sure that our apache has been compiled with all the required compiler switches. we need to arrive at final configurations after doing the required tuning. turn off debugging options/lots of logging in production systems.

+ tune with various parameters with sendFile ON/OFF
+ we need to turn on keepalive ON

finally test your apache for 1k simultaneous connections using the AB tool. you may use grind/siege etc\. whatever you like, main thing is it should be verified.

6) Now we need to move to tomcat. we are using jwal team's nTomcat to run tomcat after starting the jwal layer. important things to remember are
+ turn off re-loading
+ turn off debugging
+ JVM settings
+ server.xml changes
+ shutdown http connector in production systems
+ tune with and w/o APR.

The AJP connector settings should change. This has been covered in a separate blog.

After doing the setup , verify the connections.


7) JVM settings tuning
+ min / max heap size
+ -Xss switch
other things like incremental gc, starting the JVM in server mode and other tweaks we can think of.

8) Now Run the TAS tool against our deployment and get the statistics.

================= programs to check max. threads =======================

[rajeevj01@bdc31035e sandbox]$ cat threads.c
/* compile with: gcc -lpthread -o thread-limit thread-limit.c */
/* originally from: http://www.volano.com/linuxnotes.html */

#include
#include
#include

#define MAX_THREADS 20000
int i;

void run(void) {
char c;
if (i < c =" %u" rc =" 0;" i =" 0;" rc ="=" rc =" pthread_create(&(thread[i])," rc ="=" 100 ="=">


Number of threads control how many concurrent requests can be served by this connector and that is a very important parameter for us.

Tomcat Performance Tuning Tips

Tomcat Performance Tuning Tips

Most companies I have worked for use Tomcat as Servlet Container. It is de facto standard just like how Apache been used as Web Server. However, most of us just drag our war file to the webapp folder and use Tomcat with all the settings as default out of the box. It works fine in development environment but may not in production. This article will give you advice in several areas:

1. Production Tomcat Architecture
2. Tuning tomcat for performance
3. Resolving problems which affect availability

Tomcat(Production)

In production Tomcat relies on a number of resources which can impact its overall performance. Understanding the overall system architecture is key to tuning performance and troubleshooting problems.

1. Hardware: CPU(s), memory, network IO and file IO
2. OS: SMP (symmetric multiprocessing) and thread support
3. JVM: version, tuning memory usage, and tuning GC
4. Tomcat: version (example, Tomcat 6 supports NIO)
5. Application: Application design can have the largest impact on overall performance
6. Database: concurrent db connection is allowed (pooling and object caching)
7. Web Server: Apache can sit in front of Tomcat and serves the static content. It also can do load balancing across multiple Tomcat instances.
8. Network: Network delays.
9. Remote Client: How fast is the communication protocol? Content can be compressed.

Performance Tuning

How to see performance

• Request latency is key b/c it reflects the responsiveness of your site for visitors.
• Test environment should match production as closely as possible.
• The data volume is important to simulate in database side.
• Test HTTP requests with different request parameters (test corner cases)
• Use load test to simulate the traffics (ex. JMeter)
• Final tests should be over longer periods like days because JVM performance changes over time and can actually improve if using HotSpot. Memory leaks, db temporary unavailable, etc can only be found when running longer tests.

JVM

• Sun Java and later releases inlcude some profiling optimizer customized for long running server application.
• Tomcat will freeze processing of all requests while the JVM is performing GC. On a poorly tuned JVM this can last 10’s of seconds. Most GC’s should take <>
• Tune the -Xms (min) and -Xmx (max) java stack memory (set them to the same value can improve GC performance)
• Make sure the java process always keeps the memory it uses resident in physical memory and not swapped out to virtual memory.
• Use -Xincgc to enable incremental garbage collection
• Try reducing -Xss thread stack memory usage

Tomcat

• Tomcat 6 supports NIO.
• Set "reloadable" false - remove unnecessary detection overhead
• Set "liveDeploy" to false - liveDeploy controls whether your webapps directory is periodically checked for new war files. This is done using background thread.
• Set "debug" to 0
• Set "swallowOutput" to true - This makes sure all output to stdout or stderr for a web application gets directed to the web application log rather than the console or catalina.out. This make it easier to troubleshoot problems.
• Connector configuration - minProcessor, maxProcessor, acceptCount, enableLookups. Don’t set the acceptCount too high b/c this sets the number of pending requests awaiting processing. It is better deny few requests than overload Tomcat and cause problems for all requests. Set "enableLookups" to false b/c DNS lookups can add significant delays.

dbcp

• We use connection pool provided by Spring instead
• Using middleware to persist and cache objects from your database can significantly improve performance b/c of fewer db calls, less thrashing of the JVM for creation and subsequent GC of object craeted for resultset.

Application design and profiling

• If the data used to generate a dynamic page rarely changes, modify it to a static page which you regenerate periodically.
• Cache dynamic page
• Use tool like JProble to profle your web applications during development phase
• Look for possible thread synchronization bottlenecks
• Date and Time thread synchronization bottleneck

Troubleshooting

Collecting and analyzing log data from production

Common problems on production

• Broken pipe - For HTTP Connector indicates that the remote client aborted the request. For web server JK Connector indicates that the web server process or thread was terminated. These are normal and rarely due to a problem with Tomcat. However, if you have long request, the connectionTimeout may close the connection before you send your response back.
• Tomcat freezes or pauses with no request being processed - usually due to a long pause of JVM GC. A long pause can cause a cascading effect and high load once Tomcat starts handling requests again. Don’t set the "acceptCount" too high and use java -verbose:gc startup argument to collect GC data.
• Out of Memory Exception - look into application code to fix the leak (profile tool can help). Increase available memory on the system via -Xmx. Restart tomcat!
• Database connection failure - connection used up when traffic is spike.
• Random connection close exception - when you close your connection twice. First close(), the connection returns to the pool. It may be picked up by another thread. Now, second close() may close a connection that is being used by other thread. Don’t close connection twice, use JDBC Template from Spring to avoid this problem.

Java Application Design and Performance Tips

Java Application Design Tips

Developing jav applications requires you to keep certain strategies in mind during the design phase. It is best to strategically design an application before you begin coding. Correcting the code because you failed to consider all of the "gotchas" before developing the application can be a painful process for Java Application Design.

Here are some design strategies to consider for Java Application Design:

• Keep it simple. Remove unnecessary features, possibly making those features a separate, secondary application. (Java Application Design)
  
  
• Smaller is better. This consideration should be a "no brainer" for all developers. Smaller applications use less memory.(Java Application Design)
  
  
• Minimize memory usage. To minimize the amount of memory used at run time, , do not depend on the garbage collector. You should manage the memory efficiently yourself by setting object references to null when you are finished with them. Another way to reduce run-time memory is to use lazy instantiation, only allocating objects on an as-needed basis. Other ways of reducing overall and peak memory release resources quickly, reuse objects, and avoid exceptions. (Java Application Design)
  

Java Application Performance Tips

Code for performance. Here are some ways to code with the aim to achieve the best Java Application performance:

• Use local variables. It is quicker to access local variables than to access class members.
  
  
• Avoid string concatenation. String concatenation decreases performance and can increase the application's peak memory usage.
  
  
• Use threads and avoid synchronization. Any operation that takes more than 1/10 of a second to run requires a separate thread. Avoiding synchronization can increase performance as well.
  
  
• Separate the model using the model-view-controller (MVC). MVC separates the logic from the code that controls the presentation.