Looks like that Tomcat native can not work without the openSSL support. so we are going to build ssl also as part of our APR package. As usual we will start with APR.
Download the apr 1.2.7 bundle
(1)
./configure --prefix=/oad/x/apr
make
make install
(2)
Then go a grab a copy of openssl sources from www.openssl.org. I get 0.9.7f tarball, the version that corresponds to my Fedora core 4 install.
./config --prefix=/oad/x/apr --openssldir=/home/rajeevj01/x/tmp shared
make
make test
make install
The documents and tests would be installed in x/tmp location and libraries are installed in /oad/x/apr location.
(3)
Next, we un-tar and build the tomcat-nativve1.1.2 source ball with the following options.
[rajeevj01@bdc31035e native]$ ./configure --prefix=/oad/x/apr --with-ssl=/oad/x/apr --with-apr=/oad/x/apr --with-java-home=/home/rajeevj01/sw/jdk1.5 --with-java-platform=2
(4) Now, set up the tomcat server to use this native library , Add the following in $TOMCAT_HOME/bin/setenv.sh script (or equivalent)
------------------------------------------------------------------------------------------
LD_LIBRARY_PATH=/oad/x/apr/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH
-------------------------------------------------------------------------------------------
rajeevj01 at 2:41:57 AM EDT Link to this entry | Blog about this entry | Notify AOL
This entry has 0 comments: Add your own
Building APR library on Feodra core 4 Worried
The libtcnative (The native tomcat connector) requires the openSSL and APR libraries headers to be built. so the first step is : Download and install the APR. I head over to http://apr.apache.org and grab the latest tarball. untar it and just run configure with the required prefix
1051 ./configure --prefix=/oad/x/apr
1052 make
1053 make install
APR is now installed. we are planning to use AJP connector and so we build the non-SSL version of tomcat AJP/13 connector.
[rajeevj01@bdc31035e native]$ ./configure --prefix=/home/rajeevj01/x/tcnative126 --with-apr=/home/rajeevj01/x/apr126 --without-ssl --with-java-home=/home/rajeevj01/sw/jdk1.5 --with-java-platform=2
if we do not want to use SSL, we can go ahead and remove the -lssl flag from the makefile also. But if you do that, the tomcat can not find the APR installed. SSL must be enabled for APR (why?)
There are some version issues with tomcat 1.1.1(bundled with tomcat 5.5.16) and apr 1.2.7 , so download the tomcat 1.1.2 sources from
http://tomcat.heanet.ie/native/1.1.2/source/
1100 make
1101 make install
This builds the native tc library. Once we have the library, the next big question is how to use it.
rajeevj01 at 4:18:18 AM EDT Link to this entry | Blog about this entry | Notify AOL
This entry has 0 comments: Add your own
Ethereal for http packet captures Silly
we need some packet capture tool to analyze the server headers and what the browser is sending to the server. After using couple of useless freeware tool, now i have settled on ethereal. it can do a lot of stuff and options seem quite daunting but the good thing is i can make it work to my purpose with minimal fuss.
Download and install ethereal and read the user guide anyway. To get up to speed very quickly,
1) start ethereal , to start capture , Go to Menu | Capture |Options
2) on this screen, select the Interface (like VPN adapter or intel ethernet card etc)
3) in filter box, type tcp port 5400
one more filter that you can try is tcp port 5400 and host 10.141.11.189
etc.
This will capture all traffic to and from that host going through your designated interface.
4) I do not like to see lots of colors , so Menu | View | colorize packet List
This lets you switch on/off the packet colors in display.
5) You can save the packets captured during the session with Menu | File | Export
6) In the capture window, each packet can be probed further for frame details etc.
All in all a really useful tool and w/o any freeware shit!!
rajeevj01 at 7:29:51 AM EDT Link to this entry | Blog about this entry | Notify AOL
This entry has 0 comments: Add your own
webchat GC tuning Loopy
okay, so we have done one round of our JVM garbage collection data capture and this is what i can infer:
+ The eden space size is small , so minor collections are happening every now and then.
+ survivor space is not big, so after minor collections overflow is happening to tenured space at a very young age
So this means, i should start with a bigger young generation. Plus i am running a server application, so i will go with same -Xms and -Xmx options.
Here are my GC parameters
==============================================================
-Xss256K -Xms512M -Xmx512M -XX:NewRatio=2 -XX:SurvivorRatio=6 -XX:+UseParallelGC
=============================================================
I may want to try with -XX:NewSize=256m -XX:MaxNewSize=256m
so that young generation size is half the heap size to start with. Right now, i will settle for young:tenure ratio as 1:2.
with young generation = half the heap size we get
Young = 256 MB, Eden = 6 * (256/8) == 192 MB. survivor space is 32 MB each.
=================================================================
To check the jvm stats, we use
$jstat -gcutil
like:
$jstat -gcutil
we can get the vmid using the $jps command.
rajeevj01 at 7:22:48 AM EDT Link to this entry | Blog about this entry | Notify AOL
This entry has 0 comments: Add your own
How to make the eclipse profiler work for you
Silly
First things first. we want to do remote profiling and we do not want lots of over heads. The option was there to run one of the standard tools like optimizeIt or Jboss profiler etc. However, i could not set up Jboss profiler quickly enough and optimizeIt would have required tomcat to start with its own audit system.
So finally i settled for eclipse plug-in profiler. This plugin can attach to remote servers. It just needs one library file and few command line options on server.
Required server changes
A) catalina.sh script or the script that starts you server will require the following chages
this one is for CPU profiling, see the link for more details on memory profiling etc. http://eclipsecolorer.sourceforge.net/index_profiler.html
JAVA_OPTS="-XrunProfilerDLL:1 -Xbootclasspath/a:jakarta-regexp.jar:profiler_trace.jar:commons-lang.jar \
-D__PROFILER_PACKAGE_FILTER=__A__%MAINCLASS%:__M__sun.:__M__com.sun.:__M__java.:__M__javax.:__M__org.apache. \
-D__PROFILER_TIMING_METHOD=1"
For our setup, we need to include these options in post.tcl file along with other java command line options . This is a jvmpi plugin so some of the GC options you require may not be supported.
B) copy common-lang.jar, jakarta-regex.jar and profiler_trace.jar to $tomcat_home/bin folder.
in our case, this may mean the folder from where we bootstrap the tomcat application. so copy the 3 jar files to the port folder like to 48800/ folder.
C) compile the agent using the following script ( Better to compile on target server ). The default agent is compiled with gcc 3.2 and it may not work on all systems.
g++ -O0 -DLINUX -shared -Wall -I. -I/home/rajeevj01/sw/jdk1.5/include
-I/home/rajeevj01/sw/jdk1.5/include/linux ProfilerDLL.cpp -o libProfilerDLL.so
D)copy the agent file to JDK_home/jre/lib/i386 folder
or you can put it in LD_LIBRARY_PATH also.
After making the changes, start the server. check logs for any errors. Once server startup is successful, we are ready to attach to this server to get profiling data.
Eclipse changes :
You need to do few changes to your eclipse before you can open the profiler window.
A) we require a new plugin. download it from the web link http://eclipsecolorer.sourceforge.net/index_profiler.html and copy to your eclipse plug-ins folder. Then from menu | windows | open perspective | profiler
B) you need to create a launcher configuration to connect to remote server. To do that , open Menu | Run and choose Run .. option. under remote profiler node, create a new configuration by clicking New button. Add the server name and click on Run button. thats it!!!!
C) you have to patch the plugin for eclipse 3.1 version.
link is
http://sourceforge.net/tracker/index.php?func=detail&aid=1202373&group_id=48823&atid=454283
D) You may require UNRAR on windoze machines. see unxutils zip on sourceforge. http://unxutils.sourceforge.net/
rajeevj01 at 8:15:30 AM EDT Link to this entry | Blog about this entry | Notify AOL
This entry has 0 comments: Add your own
Roadmap : for doing server performance tuning
Silly
1) chat is working okay for so-called 400 simultaneous users. we do not know when we get the apache on chdl-m05 blue box. so in the mean-while this is the plan i have for performance testing
A) we need to profile the application and see what is taking time. Right now, there are too many synchronization blocks around!! For profiling , we can take a look at JProbe/OptimizeIt or Jboss profiler, whatever works for us.
B) Shut down apache debugging and DNS lookup etc. point is you run a very lean server. should we compile the mod_status module at this point ? Linux server data should give enough indications at this point!
C) we need to do tomcat and JVM monitoring. we can do the monitoring using MC4J.
D) we capture the linux box stats for different combinations.
We need snapshots of following data
0) remove all the debugging options and make a lean server.
1) Active thread count
2) open sockets
This number should be less than my $ulimit -n output.
3) cpu utilization
4) memory ( I should not be swapping )
5) JVM heap status
we can use the jstat tool. what we want to understand is how much of the heap we are actually using under projected loads. This will give us a better idea of our heap usage. Also, i would like to know if i am doing lots of GC around.
The link for jstat is http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jstat.html
see the effect of using -server option etc.
6) Apache status
we can use mod_status for this purpose (?)
The parameters to vary are
1) apache server threads + number of child processes
2) JVM heap size, -server option, -Xincgc option
3) Thread stack size for apache as well as JVM threads
This entry has 3 comments: Hide Recent | Add your own
kernel-imposed limits on the system
-------------------------------------------
1.fd-max can be got at: cat /proc/sys/fs/file-max
2. threads-max can be got at: cat /proc/sys/kernel/threads-max
Comment from shrutikbdc - 6/5/06 7:56 AM
top command
-----------------
top -b -d 2 > top_stats
This redirects the output of top to a file in plain text format every 2 seconds
Comment from shrutikbdc - 6/5/06 7:45 AM
Memory
vmstat
The following can be captured from vmstat output:
Procs
r: The number of processes waiting for run time.
b: The number of processes in uninterruptable sleep.
w: The number of processes swapped out but otherwise runnable. This
field is calculated, but Linux never desperation swaps.
Memory
swpd: the amount of virtual memory used (kB).
free: the amount of idle memory (kB).
buff: the amount of memory used as buffers (kB).
Swap
si: Amount of memory swapped in from disk (kB/s).
so: Amount of memory swapped to disk (kB/s).
IO
bi: Blocks sent to a block device (blocks/s).
bo: Blocks received from a block device (blocks/s).
System
in: The number of interrupts per second, including the clock.
cs: The number of context switches per second.
CPU
These are percentages of total CPU time.
us: user time
sy: system time
id: idle time
mod_jk connector to plug jetty/tomcat in apache Loopy
we are going to use mod_jk to connect apache and jetty.
step1) download and copy mod_jk.so to /usr/lib/httpd/modules
Download link is:
http://tomcat.apache.org/download-connectors.cgi
Beware : do not download jk2 module, download jk module. This is very confusing!
jk is the one that we need not jk2. You may want to rename the jk .so file.
step2) add LoadModule directive to the apache httpd.conf file
[ Add where other LoadModule directives are ]
LoadModule jk_module modules/mod_jk.so
Beware : The way to add modules may differ in apache 1.3 and apache 2. I dont think you require anything more than above one line with apache 2 that has been enabled with DSO support.
Question: How do i check if my apache has been compiled with DSO support?
[rajeevj01@bdc31035e installs]$ /usr/sbin/httpd -l
should list mod_so.c
step3) go to /etc/httpd/conf.d directory
[ comment: this may be specific to FC4 , if such a folder is not there then you have to specify the path to mod_jk.conf in apache httpd.conf file using an Include
Include mod_jk.conf
]
create a mod_jk.conf file there with the following content
--------------------------x------------------------------------------------
# rjha: adding the mod_jk options
JkWorkersFile /etc/httpd/conf/workers.properties
JkLogFile /home/rajeevj01/logs/mod_jk.log
JkLogLevel debug
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
JkMount /servlet/* ajp13
------------------------------x-----------------------------------------------
Replace /servlet/* with your dymanic scripts context path.
step 4) create the logs dir if that does not exist
step5) create the workers.properties file with following info
-------------------------x-----------------------------------------------------
ps=/
# rjha: sample worker.properties lifted from GERONIMO
worker.list=ajp13
worker.ajp13.port=8009
worker.ajp13.host=localhost
worker.ajp13.type=ajp13
step 6) Now configure your servlet container to listen on AJP13 port.
A) for jetty do the changes in $jetty_home/etc/jetty.xml
rjha : uncommenting the AJP13 listener -->
B) for tomcat , do the changes in server.xml file. comment out the HTTP connector and uncomment the AJP13 connector portions.
Thats it. Now next set of things to do :
1) know more about jetty logging options
2) know more about mod_jk logging options
3) how servlets are deployed on jetty, the context mappings
Then we can start changing and debugging the jetty code.
Deploying the dit dot servlet Happy
The dit dot servlet just prints one dot every 1/2 seconds. Point is to replace the chat pipe servlet with this dit dot servlet on my machine and do the stress testing. I am going to deploy the dit dot servlet on my development box, with tomcat (ditch jetty for the moment). so the next steps in my wave of experiments is
1) Download and install the latest tomcat (preferably from sources ?) no ? binary should do at the moment. download sources as a side-kick.
2) deploy the dit dot servlet with tomcat container
3) deploy the rest of the chat apps
4) Now do the JVM stats monitoring , things like, how much of memory i am munching, how many threads etc. Need to find out more about this.
============= web.xml file ==============================
We want to monitor our tomcat application JVM. The development boxes i have are linux boxes. On these machines, we just have to add the following 4 lines to JAVA_OPTS in catalina.sh script
====================================================
# rjha: supply this argument to JVM on start-up
# we want to monitor tomcat JVM remotely
# development system, no password!
JAVA_OPTS="-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=9421 \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false"
===================================================
No authentication, no passwords, i dont give a Damn at the moment.
Then you should have the JDK 1.5 installed on your machines.
on the monitoring machine, just fire jconsole and connect to target VM.
Right now, remote monitoring not working, only local monitoring is working.
I can monitor the number of threads now!!!!
Areas of performance monitoring of Tomcat Silly
Sometimes you would like to know how your tomcat server is doing. How many threads are open on container etc. There are two ways you can do this
1) Use the server status link on tomcat manager application
log into tomcat manager application. you will see a Server Status link on top of application's list.
2) Use one of the JMX monitoring tools. catalina container properties can be exposed as JMX beans and we can use one of the JMX monitoring tools. I cover JMX tools list in a separate document.
MC4J one of the tools that we are going to use (covered in separate blog) . we need to check out visualGC also. plus there is a tool called HAT for heap analysis. I need to pay attention to JVM tunining next. But for the starters, here is the points to ponder when tuning tomcat
1) server configuration file. Need more info on this from the tomcat user's guide.
http://tomcat.apache.org/tomcat-5.5-doc/config/index.html
The config parameters should tell me about
maximum processes, acceptCount, max-threads etc. Need to do RTFM here.
2) JVM parameters may play a big role here, so we need some info on JVM tuning also
+ Thread stack size
+ head min/max size
+ GC (incremental/ how much allocated to generational space)
+ -X arguments to JVM @ startup
-Xincgc (for incremental GC)
3) Threading support
How do i find out if my VM is using NPTL libs or not ? start with the linux kernel version
$uname -r
if you are running a 2.6.x kernel then you should be ok.
use this small program to print java library paths
================================================
public class PrintLibraryPath {
public static void main(String[] args) {
System.out.println(System.getProperty("java.library.path"));
}
}
====================================================
Then we can use ldd to see if NPTL is linked in our JVM or not.
[rajeevj01@bdc31035e jdk1.5]$ ldd /home/rajeevj01/bin/jdk1.5/jre/lib/i386/client/libjvm.so
linux-gate.so.1 => (0x00835000)
libm.so.6 => /lib/libm.so.6 (0x00111000)
libdl.so.2 => /lib/libdl.so.2 (0x003e4000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00219000)
libc.so.6 => /lib/libc.so.6 (0x0022b000)
/lib/ld-linux.so.2 (0x00864000)
[rajeevj01@bdc31035e jdk1.5]$ nm /lib/libpthread.so.0 | grep nptl
NPTL is linked!!
---------------------------------------------------------------------------------------------------
+ switch off servlet reloading
+ compile jsps a-priori in production usage. use JSPC compiler.
+ we are not using tomcat in stand-alone mode, so *comment* out that connector.
we need something to monitor the tomcat! something to monitor the JVM.
Use jconsole to see how the JVM is performing. Use servlet statistics page to see how the tomcat installation is doing.
Tomcat configurations Loopy
+ you should not allow any tracing/debugging options
+ you should disable the servlet re-loading
+ The number of threads should be tuned on the server. is there a way to increase the maxThreads programatically ?
maxKeepAliveRequests
The maximum number of HTTP requests which can be pipelined until the connection is closed by the server. Setting this attribute to 1 will disable HTTP/1.0 keep-alive, as well as HTTP/1.1 keep-alive and pipelining. Setting this to -1 will allow an unlimited amount of pipelined or keep-alive HTTP requests. If not specified, this attribute is set to 100.
maxSpareThreads
The maximum number of unused request processing threads that will be allowed to exist until the thread pool starts stopping the unnecessary threads. The default value is 50.
maxThreads
The maximum number of request processing threads to be created by this Connector, which therefore determines the maximum number of simultaneous requests that can be handled. If not specified, this attribute is set to 200.
minSpareThreads
The number of request processing threads that will be created when this Connector is first started. The connector will also make sure it has the specified number of idle processing threads available. This attribute should be set to a value smaller than that set for maxThreads. The default value is 4.
DescriptionacceptCount
The maximum queue length for incoming connection requests when all possible request processing threads are in use. Any requests received when the queue is full will be refused. The default value is 10.
java.sun.com has lots of articles on how to tune JVM settings. crux of the matter is that you should be able to
+ size up a heap for your purposes
+ thread stack size and numbers for your purpose
Right now, we are running with
-Xms128M -Xmx256M -Xincgc -Xss128k
you can do the required arithmetic. try tuning with server mode also, the warm-up may take some time but after that it should be ok!
use some JMX remote management tool like MC4J etc to monitor the health of the JVM as well as tomcat instance running on top of it.
This is a good reference to see what all options are available with JVM
http://blogs.sun.com/roller/resources/watt/jvm-options-list.html
this one discusses the JVM threads
http://java.sun.com/docs/hotspot/threads/threads.html
and this one tuning the GC
http://developers.sun.com/techtopics/mobility/midp/articles/garbagecollection2/
and finally we have the JVM tuning whitepaper
http://java.sun.com/performance/reference/whitepapers/tuning.html
Threads tuning params on linux 2.6
The issues here is to find out the system limits and application limits imposed on a running program on linux 2.6 kernel series. Later we see how we can tune the parameters to spawn a lot of threads and do fast context switching as well.
* Application limit tuning *
==================
1) ulimit -s and ulimit -u
ulimit has options to set thread stack size.
check by typing ulimit -a. looks like this is not system wide setting but just applies to applications started in that bash shell. you can not go beyond the hard-limits of a parameter. The hard limits can be checked by typing ulimit -aH.
2) you can set linux thread stack size using
$ulimit -s
Then you make sure that you use the same value with java -Xss option when starting your JVM. That way the thread runs in a limited stack space and more and more threads can be spawned.
* System limit tuning *
================
1) check the system wide settings in following places
$ cat /proc/sys/kernel/threads-max
$grep totalpages /var/log/dmesg (totalpages/16 is the max # threads possible)
$ cat /proc/sys/kernel/pid_max
$cat /proc/sys/vm/max_map_count (* need to figure out what exactly is this *)
* test program *
=============
3) Run the test program to see how many threads are supported on your machine
Idea is , final limit on # max threads should only be imposed by the amount of RAM in system.
4) How to do system wide ulimit settings ?
check the contents of file
/etc/security/limits.conf
Bumping the ulimit Happy
The ulimit command has two variants. ulimit-aH and ulimit -aS. H is for the hard options (what is set in the config files for the user) and soft is the current value.
Now if you are root its all pretty ok,you can just use ulimit command to increases whatever you want.
for other logins, we can not bump up the resources more than the hard limits. For such cases, we have to first increase the hard-limits by changing the /etc/security/limits.conf on linux 2.6 systems.
=================================================
Edit /etc/security/limits.conf and add the lines:rajeevj01 soft nofile 1024 rajeevj01 hard nofile 65535
Also check the values in /proc/sys/fs/file-max.
[ on 2.6.x systems, they are pretty ok!]
==================================================
After bumping up the hard limits, use ulimit command for user login
$ ulimit -n unlimited
A: max address space (KB)
C: max core file size (KB)
D: max data size (KB)
F: maximum filesize (KB)
M: max locked-in-memory address space (KB)
N: max number of open files
R: max resident set size (KB)
S: max stack size (KB)
T: max CPU time (MIN)
U: max number of processes
K: file creation mask, set by umask(2) .
L: max number of logins for this user
P: process priority, set by setpriority(2) .
For example, L2D2048N5 is a valid LIMITS_STRING. For reading convenience, the following entries are equivalent:
username L2D2048N5
username L2 D2048 N5
So far not able to locate any remote monitoring tools for linux box. Most of the scripts there can be put as part of cron and flushed to log files. I can use a user-specific crontab files!!
+ Install the redhat sysstat package $yum install sysstat
+ need to do some research on SAR and crond to achieve the desired results.
other useful commands are
ps,iostat, vmstat, top ,mpstat [ But the hitch is i have to be on the machine.]
One good link is http://people.redhat.com/alikins/system_tuning.html
Apart from that try reading the RedHat sysadmin guide also
http://www.redhat.com/docs/manuals/linux/RHL-9-Manual/admin-primer/index.html
Need to do some research on flushing cron jobs
So its agreed , to use netstat to monitor the number of open sockets and the above utilities to monitor the system health. since i am going to run all the experiments on LAN, i am not going to monitor the bandwidth usage as such!
I still can not locate any tool that allows for remote monitoring. anyways, i am not going to break my head over it. mostly, this is what i need to do
A) How to monitor
+ open processes
+ open threads
+ memory
+ I/O, open sockets and open file descriptors.
I think the most important is memory monitoring. when i check my /proc parameters, I dont think i am going to run out of fds and processes in a hurry! So kernel-limits are all ok. So we got to watch memory and CPU after bumping the user limits of fd and thread stack size using ulimit command in a shell.
To see the open sockets on my machine , I use this command lifted from red hat's site:
=====================x===================================
One of the most twisted, but useful invocations is:
$netstat -a -n|grep -E "^(tcp)"| cut -c 68-|sort|uniq -c|sort -n
This will show you a sorted list of how many sockets are in each connection state.
=====================x====================================
To see the memory and CPU usage , i am relying on top. for a quick snapshot , you may consider using vmstat also.
$vmstat 1 5
gives 5 readings with 1 sec interval.
B) gnome-system-monitor is a graphic utility that can be installed using yum if it is not there on your system. I have FC4 and gnome-system-monitor is installed. This shows the process list and memory usage.
C) top, watch, free , ps etc. are usual unix commands.
The following gives a recipe to bump the number of allowed file descriptors.
1) setting up the hard limit in /etc/limits.conf
rajeevj01 nofile hard 2048
2) Then executing ulimit -n 2048 in a shell should increase the number of file descriptors. on recent systems , /proc/sys/fs/file-max is good enough for all the purposes (limited by hardware only). on my machine it is ~ 101k.
3) apache or tomcat uses some start-up scripts like apachectl or catalina.sh etc (or if started from /etc/rc.d/httpd etc)
There, in the beginning of such scripts we can set the soft ulimit. (make sure the hard-limits have been bumped for apachectl user)
=========================================
ulimit -S -n `ulimit -H -n`
=========================================
if you are using PAM, then you also need to do the following to honor the limits.conf
edit /etc/security/limits.conf and add to any config file in /etc/pam.d/ that handles logins ("login" and "ssh" example):
session required /lib/security/pam_limits.so
1) packet analyzer/tcp mon
To see what the server is sending and what the browser is receiving. should show the files/size/transfer time plus all the headers. Useful for debugging.
2) bandwidth monitor and throttler
what is my download speed right now, how can i make my speed simulate a modem speed.
3) network load/stress monitor
How is my network pipe doing. what kinds of stress it is subjected to. is my webserver saturating my network card?
4) Javascript debugger
how to use venkman. microsoft script debugger. get rid of alert debugging.
5) Javascript compiler/size muncher
6) web server log analyzer, web server and servlet container tuning
Tomcat, Apache, Weblogic
7) OS resources monitoring and tuning.
Apache connection tuning Silly
The apache 2 has 256 MaxClients limit out-of-box. The compiled-in hard-limit of MaxClients(ServerLimit) is 20,000 (way beyond our purpose). First we will look at some of the apache directives used in httpd.conf to tune the server.
1) ServerLimit : This goes hand-in-hand with the MaxClients.
For prefork MPM , this is number of MaxClients. in case of worker MPM , this together with ThreadsPerChild determines the MaxClients
for worker MPM
MaxClients = ThreadsPerChild * ServerLimit
2) ThreadPerChild
number of threads created for a apache child process. Default is 25. Hard Limit can be 64 determined by the ThreadLimit.
3) MaxClients : This is the number of simultaneous requests that apache can handle.
Below are some of the real world setting examples (This is one of the busiest sourceforge mirrors)
=============================================
ServerLimit 900
StartServers 10
MaxClients 28800
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 32
MaxRequestsPerChild 100
=================================================
As you can see, MaxClients = ServerLimit * ThreadsPerChild
=================================================
StartServers 100
MinSpareServers 10
MaxSpareServers 10
ServerLimit 50000
MaxClients 50000
MaxRequestsPerChild 2000
=================================================
I do not know if we can start so many apache processes on a machine, even theoretically.
On my Linux machine,
[rajeevj01@bdc31035e ~]$ cat /proc/sys/kernel/pid_max
32768
So this number needs to be tuned first. Plus of course there is
+ this issue of open fds
+ and linux thread stack size
+ and increasing the number of processes limit in shell using the ulimit command.
The next example config is taken from AOL search team
=====================================================
## Worker MPM settings
# AOL: Fixed 2 servers and 200 threads each for a total of 400 max
# concurrent requests. Can be expanded to 1000 * 2 = 2000 via a
# graceful httpd controlled restart.
ServerLimit 2
ThreadLimit 1000
StartServers 2
MaxClients 400
MinSpareThreads 200
MaxSpareThreads 400
ThreadsPerChild 200
====================================================
After going through all such examples lets find a configuration that will allow us to run 1k simultaneous connections. This is our first goal. And perspective gained from here will determine how we go ahead with 2k simultaneous connections.
For our EC 2.0 case lets try this setup:
ServerLimit 8
ThreadLimit 256
StartServers 2
MaxClients 1024
MinSpareThreads 128
MaxSpareThreads 256
ThreadsPerChild 128
To achieve these numbers, we would need to
+ tune the number of file descriptors
+ and linux thread stack size.
Number of threads should not be a problem. we can try tuning serverLimit to 4 and increasing the ThreadsPerChild. But only more data can tells us how to proceed ahead with tuning.
Finally, we have installed apache 2.0 with worker MPM. all you need to do is --with-mpm=worker switch when using the configuration. using the worker MPM, we can start apache with 4k threads (MaxClients). see the blog about apache tuning to know more.
Idea is to start 4 AJP workers with 1k threads each and interface it with apache. Here we have a caveat. we have no way of distinguishing one apache request from another. so the standard tricks of 1 AJP worker for one URL is not going to work for EC2. So initially i am going to set up 1 apache - 1 AJP connector for 1k connections.
we need 4k + delta for apache instances and and 1k+delta /tomcat , so in total we are talking about 4 JVMS of 1200 threads each !!!
But before we dive into JVM arithmetic, we need to know how to setup AJP13 with 1k connections. here , the trick is to use MaxThreads setting for AJP connector in server.xml file. But then what maxProcessors are doing ?
There were some discussions on tomcat list about maxProcessor attrib. But the long and short of it is that I just need to set MaxThreads and i should be okay. I dont see a need to use APR (apache portable runtime), because in my case i will be holding on to the connections. so threads are not wasted.
Now, arithmetic
1 JVM - 1200 threads
Thread stack size 256k => 8 threads /MB so 256 MB means 1k Threads
so we start JVM with minHeap 256 MB, MaxHeap 512 MB
512 MB x 4 = 2 GB for running 4 JVMs.
Thread stack size can be tuned to 128K also. ($ulimit -s 128)
Lets try with one apache + one JVM first!
One apache that can handle 1k simultaneous is covered in apache tuning blog.
chat application is a push type application. we open a socket to server and browser keeps getting data on that open connection. Here, the browser and server should not cache at any point because you need to push data to users as soon as they are available.
To enable this behavior we need to set some response headers
You will need an equivalent of these apache server headers
========================================================
Header set Expires "Sat,6 May 1995 12:00:00 GMT"
Header set cache-control "no-store, no-cache"
Header append cache-control "must-revalidate , no-transform"
Header append cache-control "pre-check=0, post-check=0"
Header set Pragma "no-cache"
=========================================================
when we pluged mod_deflate module in our apache installation , we never realized that we would hit the caching issues because of compression ;o)
mod_deflate is to compress the response, in our case it can not work till the response is complete. So to do compression, server will keep caching the data coming from chat backend and only flush it when the response is complete.
So now we need to do compression based on some conditions like some Directories only etc.
To do List for server tuning
step 1) get a development box that has RH 4.0 installed. we need RH 4.0 because we need linux 2.6.x kernel series. verify the kernel with $uname -r
step2) we need to bump up the file descriptors limit. we need to raise a ticket for this. verify the hard limits and then again verify that you can actually bump up the file descriptors numbers. we can get 8k or maybe 16k descriptors /process/shell.
3) use this program to verify
======================================================================================
[rajeevj01@bdc31035e sandbox]$ cat fd.c
#include
#include
#include
void main(void) {
int k = 3;
while (dup(0) != -1)
k++;
printf("Opens = %d Fdsize = %d\n", k, sizeof(fd_set) * 8);
}
============================================================
4) change the thread stack size with $ulimit -s
change it to 128k or 256k. for the moment lets stick with 256k.
Again we can verify the number of threads we can spawn with a program.
5) Now fds and threads are okay. we need to move to apache settings. The apache setting that will let us use 1k simultaneous connections have been covered in a different blog.
Make sure we run a lean apache. check apache 2.0 perf docs also. Also make sure that our apache has been compiled with all the required compiler switches. we need to arrive at final configurations after doing the required tuning. turn off debugging options/lots of logging in production systems.
+ tune with various parameters with sendFile ON/OFF
+ we need to turn on keepalive ON
finally test your apache for 1k simultaneous connections using the AB tool. you may use grind/siege etc\. whatever you like, main thing is it should be verified.
6) Now we need to move to tomcat. we are using jwal team's nTomcat to run tomcat after starting the jwal layer. important things to remember are
+ turn off re-loading
+ turn off debugging
+ JVM settings
+ server.xml changes
+ shutdown http connector in production systems
+ tune with and w/o APR.
The AJP connector settings should change. This has been covered in a separate blog.
After doing the setup , verify the connections.
7) JVM settings tuning
+ min / max heap size
+ -Xss switch
other things like incremental gc, starting the JVM in server mode and other tweaks we can think of.
8) Now Run the TAS tool against our deployment and get the statistics.
================= programs to check max. threads =======================
[rajeevj01@bdc31035e sandbox]$ cat threads.c
/* compile with: gcc -lpthread -o thread-limit thread-limit.c */
/* originally from: http://www.volano.com/linuxnotes.html */
#include
#include
#include
#define MAX_THREADS 20000
int i;
void run(void) {
char c;
if (i < c =" %u" rc =" 0;" i =" 0;" rc ="=" rc =" pthread_create(&(thread[i])," rc ="=" 100 ="=">
Number of threads control how many concurrent requests can be served by this connector and that is a very important parameter for us.
Posts
1 comment:
such a nice article
Post a Comment