load balancing / failover

I've determined that Coldfusion clusters suck.  They're just DUMB. If one instance in a cluster freezes, other people still get referenced to that instance and get no response.  CF clustering provides no real "failover" from one instance to another, so it seems. Am I wrong on this? I don't have a setup that warrants having multiple physical servers and a hardware load balancer so I'm wondering if there are any decent software load balancing solutions that will provide more reliability than CF clustering does. Basically I'd like to have two IIS servers each using a separate CF instance, and have a software load balancer distribute the requests to each (maintaining sticky sessions), and if one instance stops responding (due to failure or high load), force requests onto the available instance. Suggestions? -- Rick Root New Brian Vander Ark Album, songs in the music player and cool behind the scenes video at www.myspace.com/brianvanderark

Actually, I think what your seeing here is not that the clustering is breaking, but rather that the JRun/IIS connector piece is dying.  I'm running clusters in all of my shared environments and experience this very infrequently.  All of them are Win2k3 RC2 (though they were previously Win2k).  The only time I ever have issues is if IIS just plain hangs.  But when IIS hangs, it's never in a good way ;).  If it's still possible to open a the communication port, our hardware load balancer thinks that the server is alive.  But since IIS is stuffed up, the JRun connector piece never really gets called.  If the JRun connector doesn't get called, clustering won't function.  We've now taken to having a {server}/alive.htm file that is pinged every 5 seconds by the hardware load balancer to ensure that IIS hasn't died in addition to the software JRun clustering.  If the hardware load balancer can't get through to the alive page then the server is taken out of the hardware balancing cluster. Matthew Williams Geodesic GraFX www.geodesicgrafx.com/blog

Rick, I believe the ideal way to do this would be to have X number of IIS server all connecting to a single 'cluster' type (run the web connector, and do not connect to a single cluster member, but to the named customer itself). Basically split your web into 2 tiers, the physical web tier with no CF installed (this is great for security too btw), and then a tier with all your CF servers inside of it.  If you have one IIS box hooked to instance1, and the other to instance 2, and you have a cluster named 'myCluster', the cluster is not actually being used unless you re-ran the web connector and hooked IIS to the myCluster object instead of instance1. I have had the failover work great for me in a CF cluster. Chris I've determined that Coldfusion clusters suck.  They're just DUMB. If one instance in a cluster freezes, other people still get referenced to that instance and get no response.  CF clustering provides no real "failover" from one instance to another, so it seems. Am I wrong on this? I don't have a setup that warrants having multiple physical servers and a hardware load balancer so I'm wondering if there are any decent software load balancing solutions that will provide more reliability than CF clustering does. Basically I'd like to have two IIS servers each using a separate CF instance, and have a software load balancer distribute the requests to each (maintaining sticky sessions), and if one instance stops responding (due to failure or high load), force requests onto the available instance. Suggestions? -- Rick Root New Brian Vander Ark Album, songs in the music player and cool behind the scenes video at www.myspace.com/brianvanderark

Rick Root wrote: > I've determined that Coldfusion clusters suck. ColdFusion clusters don't exists. You mean a JRun cluster? > If one instance in a cluster freezes, other people still get referenced to > that instance and get no response.  CF clustering provides no real > "failover" from one instance to another, so it seems. > > Am I wrong on this? JRun clusters will only failover when the JRun connector (jrun.dll loaded into the IIS process) detects there is something wrong with an instance. The detection time can be between 15 seconds for a clean shutdown, 4 minutes for some types of network problems and never for some internal errors. > Basically I'd like to have two IIS servers each using a separate CF > instance, and have a software load balancer distribute the requests to each > (maintaining sticky sessions), and if one instance stops responding (due to > failure or high load), force requests onto the available instance. NLB + JRun/JBoss cluster. Jochem

----- Excess quoted text cut - see Original Post for more ----- Last time I checked, NLB didn't provide any failover either, unless the actual system is down (or the nic is not responding). Why not use a real loadbalancer - either hardware or software like LVS?   Russ

Russ wrote: ----- Excess quoted text cut - see Original Post for more ----- Correct. > Why not use a real loadbalancer - either hardware or software like LVS?   Because sometimes you should give people the solution they ask for instead of the solution you would build for yourself. Jochem

You are correct that the JRun clustering is not application aware.  As a technicality, I wouldn't blame ColdFusion.  I'm pretty certain CF is just making use of the J2EE clustering abilities.  Read: Every other J2EE app written in Java would have the same problems.  (I think) The JRun connectors will NOT send traffic to an instance if it is turned off, stopped, or otherwise not accessible.   It WILL however try to send requests to that instance even if it is: * Busy * Low on memory * Has a full queue * At 100% CPU utilization * Has some other sort of higher level application error All you get is basic round robin. You can assign priority to beefier servers, but the same rules above still apply. If you want to actually manage your traffic based on the state of the application running on each instance, you have to get a Hardware load balancer capable of hitting a "heartbeat" page which outputs some metrics about that instance and how busy it is.   I know, I was disappointed too when I first messed with clustering in Enterprise, but that's the way it is.  If your budget doesn't allow for anything more fancy smancy-- your best bet is just to monitor the instances well and restart one if it hangs and use a program like SeeFusion to auto-kill any long running stuff that might be taking down that instance. Ultimate high-availability clustering with thorough redundancy at every level= $,$$$,$$$ The rest of us just get as close as we can. ~Brad I've determined that Coldfusion clusters suck.  They're just DUMB. If one instance in a cluster freezes, other people still get referenced to that instance and get no response.  CF clustering provides no real "failover" from one instance to another, so it seems. Am I wrong on this? I don't have a setup that warrants having multiple physical servers and a hardware load balancer so I'm wondering if there are any decent software load balancing solutions that will provide more reliability than CF clustering does. Basically I'd like to have two IIS servers each using a separate CF instance, and have a software load balancer distribute the requests to each (maintaining sticky sessions), and if one instance stops responding (due to failure or high load), force requests onto the available instance. Suggestions? -- Rick Root New Brian Vander Ark Album, songs in the music player and cool behind the scenes video at www.myspace.com/brianvanderark

> Why not use a real loadbalancer - either hardware or software like LVS?   Because sometimes you should give people the solution they ask for instead of the solution you would build for yourself. ======================================= Then why did you suggest "NLB + JRun/JBoss cluster" when neither of those provide application aware balancing? Rick asked for this: "if one instance stops responding (due to failure or high load), force requests onto the available instance." And this is from Microsoft's site: "NLB doesn't detect application failure. For example, a Web Server service may stop but NLB will still send TCP/IP requests to that server." Russ, I think Rick already basically answered the question of why he isn't using something more advanced when he said "I don't have a setup that warrants having multiple physical servers and a hardware load balancer".  Even LVS would not only require him to use Linux, but have an additional physical server to control the load balancing.   ~Brad

> Russ, I think Rick already basically answered the question of why he > isn't using something more advanced when he said "I don't have a setup > that warrants having multiple physical servers and a hardware load > balancer".  Even LVS would not only require him to use Linux, but have > an additional physical server to control the load balancing. > Sorry I wasn't following the thread closely.  Having two instances on the same server won't provide a huge amount of redundancy, but if he wants to run everything on the same server, there's again either a shared hardware loadbalancer which can be had fairly cheap at a place like HMS, or set up LVS or something in VMWare on the same box.  If he's already on Linux, he can probably even use the same box. I believe Apache also has some rudimentary load balancing built in, but it probably wouldn't work for this. Russ

So if money was not object, what would be the perfect set up for a ColdFusion Environment? Do other vendor's provide a better J2EE server that has better clustering that JRun? When it comes to connecting to the JRun (CF) cluster, does IIS work differently than Apache? (for clustering) is one better than the other? For what we have been experiencing (IIS [separate server] | CF702/JRun 4 updater 7 [2 servers, 2 instances each]; recently JVM1.5), it seems to me that there is too much dependence between IIS and individual CF instances. - at times, if one instance does off line, the whole site goes down. - if an instance is off-line when IIS comes up, it is not added to the cluster when it comes back on-line - IIS needs to be restarted if there is a change in the cluster Has any one noticed any improvements in clustering in CF8? Updater 7 for JRun was meant to have clustering fixes, but we have not noticed it :-( On Thu, May 15, 2008 at 6:06 AM, Russ <cflists@ruslansivak.com> wrote: ----- Excess quoted text cut - see Original Post for more -----

I can't speak for what is the best solution, but we've had fairly good experiences using my custom Apache connector that connects to only a single cluster instance, and having 2 front end Apache servers (which are load balanced using a hardware load balancer) and 2 backend Apache/CF servers which are also load balanced using the same hardware load balancer.  The image requests get served by the front end servers, and the CF requests get forwarded to the backend servers through the load balancer.  If an instance is offline, it gets taken offline by the load balancer and requests are no longer sent to it until it comes back up.   I had to modify the Apache connector a bit to make this work.  I basically added a new config command which restricts which instances it's allowed to connect to.   Russ ----- Excess quoted text cut - see Original Post for more -----

Just to clarify Russ, You still have a (JRun) cluster, even though you are connecting directly to a CF instance? ----- Excess quoted text cut - see Original Post for more -----

Yes, we have a cluster, but the patch I made to mod_jrun stops it from connecting to other instances of the cluster.  This lets us do external loadbalancing, while still having session replication. Russ ----- Excess quoted text cut - see Original Post for more -----

AJ Mercer wrote: > When it comes to connecting to the JRun (CF) cluster, does IIS work > differently than Apache? Not significantly. They both depend on a library that gets plugged in to the server, they both bootstrap etc. > For what we have been experiencing (IIS [separate server] | CF702/JRun 4 > updater 7 [2 servers, 2 instances each]; recently JVM1.5), > it seems to me that there is too much dependence between IIS and individual > CF instances. >  - at times, if one instance does off line, the whole site goes down. >  - if an instance is off-line when IIS comes up, it is not added to the > cluster when it comes back on-line I have experienced both issues quite frequently. Luckily so far this behavior is pretty consistent and detectable befor you bring systems online. Once you have a working setup it keeps working. >  - IIS needs to be restarted if there is a change in the cluster For some changes that is correct. For instance, if you switch on sticky sessions the connector library in the webserver needs to re-initialize so it knows it has to start paying attention to the jsessionid. For that you need to restart the webserver. > Has any one noticed any improvements in clustering in CF8? > > Updater 7 for JRun was meant to have clustering fixes, but we have not > noticed it :-( I have noticed differences between a JRun installed with the CF 7.02 installer + JRun Updater 7 + CF 8.01 EAR files + JDK 1.6U5 versus a clean CF 8.01 install with the same CF 8.01 EAR files. In my setup (and I do some tricky things to get EAR file autodeployment working) I see more errors in the upgraded setup as in the clean setup. Jochem