ColdFusion MX Still crashing after updater *pulling hair out*

As much as I hate to mention things like this.. In a previous e-mail I mentioned that I had an issue with Coldfusion MX crashing on a high hit site.   At the time,   Coldfusion MX was un-patched.. Yesterday I ran the ColdFusion MX Updater 2.   and, is still crashing at least three times a day..   if not more. There is on average 200+ people on this server...   with up to 5 requests simultaneously running(most of the time..  it is 1-2). We have ASP a few ASP scripts running on the same machine. The issue is..    DLLHOST.exe will suddenly take up 100% of the processor power and not leave anything for the jrun process..      then the Jrun process queues up users..  until the DLLHOST.exe is finished processing.. or coldfusion gets 230+ queued requests..      then coldfusion quits entirely.   The service is still running after it quits..  ..    but no response from .cfm pages..   and the processor usage goes down to 1%. It was suggested that there might be a runaway ASP process that is executing.. I examined that possibility with iistracer..    and it is, indeed, a coldfusion script that is running.   (But..   it doesn't hang like this through most of the requests..       just every so often..  with no pattern)... Coldfusion is set to time out all requests after 90000 milliseconds(90 seconds)..     and yet..    iistracer is reporting scripts running for a couple of hours..       during that time..   DLLHOST jumps around..      and frequently stays at 100% processor utilization(causing there not to be enough processing power and coldfusion to queue up the requests). Also, ColdFusion reports that it has no processes running during this time. Also ..   stopping and re-starting the CF application service doesn't get rid of the process that is running..    ..  you have to stop and restart IIS entirely to stop the process that seems to 'every so often' not end. I get this error in my <cfmxroot>\runtime\logs\default-err.log; 03/21 23:18:23 error Connection reset by peer: JVM_recv in socket input stream read java.net.SocketException: Connection reset by peer: JVM_recv in socket input stream read I sometimes get this error on my <cfmxroot>\logs\exception.log; "Error","jrpp-25494","03/13/03","15:46:20","i3replication_session","null The specific sequence of files included or processed is: D:\Webs\FreeStoreClub\www\default.cfm " java.lang.NullPointerException   at coldfusion.runtime.ClientScopeKey.hashCode(Unknown Source)   at java.util.Hashtable.get(Unknown Source)   at coldfusion.runtime.ClientScopeServiceImpl.GetClientScope(Unknown Source) There is at least 250MB of free physical ram... on a Win2K Server with 512MB of RAM Processor speed:800Mhz (single processor) Running the Standard MS SQL JDBC drivers..      SQL Server is running fine.. no issues there. At least a GB of free space on the WIN2K drive..     and at least 6 GB of space on the CFMX drive. Coldfusion performance monitor counters are enabled. Client variables are stored in the database 1 processor = 5 processing pages maximum JRE settings are set to default. I don't use a single application/session variable in the application..    so that rules CFLOCK out. We are not using any COM objects..     so that rules COM Objects out.. I'm looking for a solution here..         any Ideas? -Zine

You mentioned that you applied Updater 2 yesterday.  Have you tried the newly-released Updater 3?  We've found significant improvements across the board -- especially with respect to stability and performance. Also, your setup: > There is at least 250MB of free physical ram... > on a Win2K Server with 512MB of RAM > Processor speed:800Mhz (single processor) Isn't this too choked for a server running both SQL Server _and_ CFMX Server?  I ran a configuration similar to yours (with fast SCSI drives) back in CF4.5 + SQL7 days, but CFMX really ought to have a lot more RAM available to it. Respectfully, Adam Phillip Churvis Member of Team Macromedia Advanced Intensive ColdFusion MX Training ColdFusion MX and SQL Server 2000 Class: April 28, 2003 - May 2, 2003 http://www.ColdFusionTraining.com > As much as I hate to mention things like this.. > > In a previous e-mail I mentioned that I had an issue with Coldfusion MX > crashing on a high hit site.   At the time,   Coldfusion MX was un-patched.. ----- Excess quoted text cut - see Original Post for more ----- and > frequently stays at 100% processor utilization(causing there not to be > enough processing power and coldfusion to queue up the requests). > Also, ColdFusion reports that it has no processes running during this time. > Also ..   stopping and re-starting the CF application service doesn't get > rid of the process that is running..    ..  you have to stop and restart IIS ----- Excess quoted text cut - see Original Post for more ----- input > stream read > > I sometimes get this error on my <cfmxroot>\logs\exception.log; > "Error","jrpp-25494","03/13/03","15:46:20","i3replication_session","null The > specific sequence of files included or processed is: > D:\Webs\FreeStoreClub\www\default.cfm " > java.lang.NullPointerException > at coldfusion.runtime.ClientScopeKey.hashCode(Unknown Source) > at java.util.Hashtable.get(Unknown Source) > at coldfusion.runtime.ClientScopeServiceImpl.GetClientScope(Unknown Source) > > There is at least 250MB of free physical ram... > on a Win2K Server with 512MB of RAM > Processor speed:800Mhz (single processor) > Running the Standard MS SQL JDBC drivers..      SQL Server is running fine.. ----- Excess quoted text cut - see Original Post for more ----- so ----- Excess quoted text cut - see Original Post for more -----

----- Excess quoted text cut - see Original Post for more ----- Are you calling any external objects?  CF can only timeout a thread/request that it has control over - if it's given control of the thread to a another source (database, COM, CFX, etc) then it must wait until that source returns control to it before it can time out the thread. That fact that CF isn't timing out leads me to believe that it not in control (or that it's just plain toast - but since you say that JRUN is still queueing requests it seems to be working). Jim Davis

Thanks for your response Jim LOL,   I know..    definitely odd..     I appreciate you responding. there are no CFX tags at all..   The only outside resource it uses is the Database. It also uses some of the CGI variables and verifies they are not null(or a zero length string) before using them... If I check the CGI variable   Http_host several times..   is it going to cause DLLHOST to shoot up?   Perhaps DLLHOST is providing that information to JRUN?.. Is there any way that that request that is running behind the scenes.. could be how CFMX stores a cached query?     Just a thought.. Recent news: I removed the CFFORM tag in one page on the application..  and I am now relying on the ColdFusion isnumeric() function..  to test and make sure that the value is numeric. It seems to be running better..      It was definitely causing the NULL Pointer JVM error..     I have since(removing it[cfform])..   not seen any of the java.lang.NullPointer errors in the exception.log file. That's good news at least..             and..  at least it is now stable.. Still..   DLLHOST shoots up..   now and then..     but it doesn't hold the processor at 100% for several minutes like it did yesterday(causing the CF overload).   Perhaps the <CFFORM> tag has some problems? Thanks again -Zine out* ----- Excess quoted text cut - see Original Post for more ----- Are you calling any external objects?  CF can only timeout a thread/request that it has control over - if it's given control of the thread to a another source (database, COM, CFX, etc) then it must wait until that source returns control to it before it can time out the thread. That fact that CF isn't timing out leads me to believe that it not in control (or that it's just plain toast - but since you say that JRUN is still queueing requests it seems to be working). Jim Davis

----- Excess quoted text cut - see Original Post for more ----- That's weird... Unfortunately (or perhaps fortunately) I'm still stuck at CF 4.5 for my busy production sites so my suggestions may be wrong for MX.  As far as I know the same service/process/whatever that manages the queue deals with monitoring the timeouts - so if the queue is still loading then the service that timeouts out unresponsive threads should be working too. I'm sure you've already said it, but what DB driver are you using? Native or ODBC Bridge? > It also uses some of the CGI variables and verifies they are > not null(or a zero length string) before using them... > If I check the CGI variable   Http_host several times..   is > it going to > cause DLLHOST to shoot up?   Perhaps DLLHOST is providing > that information > to JRUN?.. I really doubt that... As far as I understand it IIS provides the CGI variable set as part of it's "handoff" to JRUN.  I doubt that JRUN itself is calling anything at the system level for that - but IIS would be (but I can't see this as being a problem - at the very least HTTP_HOST HAS to be cached...) > Is there any way that that request that is running behind the scenes.. > could be how CFMX stores a cached query?     Just a thought.. I don't think it's as "formal" as a full request for Caching - however the CF Scheduler definitely does run behind the scenes and consumes a thread.  Any tasks running? ----- Excess quoted text cut - see Original Post for more ----- Well - that's definitely suspect at least... If you have a comparable test server you might want to run some load tests to see if you can replicate the problem with and without the CFFORM you were using. My gut says that the whole CFFORM tag can't be a problem (or else we'd have heard it from somebody else by now) but perhaps it and something else or an oddity with the way the form was constructed? In any case it's definitely worth checking out DLLHOST and see what it's responsibilities to CFMX are (if any). Sorry I haven't been much help. Jim Davis