Coldfusion restarting itself

Windows 2k3, Coldfusion 8.0.1 Standard on JDK 1.6. Is it normal behavior for coldfusion to restart itself for no apparent reason?  And by restarting, I mean starting.. restarting implies it's stopping.. there's no indication in the server log of it stopping.  It must be crashing and crashing hard. It happened to me today.  A lot. Nothing of interest in the coldfusion logs or the system event logs... the jrun logs show this error though, which is interesting: 07/03 23:13:53 error The request has been canceled by the administrator or by the server. coldfusion.monitor.event.MonitoringServletFilter$StopThreadException: The request has been canceled by the administrator or by the server.   at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:65)   at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)   at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)   at com.seefusion.Filter.doFilter(Filter.java:49)   at com.seefusion.SeeFusion.doFilter(SeeFusion.java:1500)   at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)   at jrun.servlet.FilterChain.service(FilterChain.java:101)   at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)   at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)   at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)   at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)   at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)   at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)   at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)   at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)   at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66) 5 seconds later... it looks like CF is starting again. 07/03 23:13:58 info No JDBC data sources have been configured for this server (see jrun-resources.xml) 07/03 23:13:59 info JRun Proxy Server listening on *:51800 07/03 23:13:59 info Deploying enterprise application "Adobe_ColdFusion_8" from: file:/C:/ColdFusion8/ 07/03 23:14:02 user ColdFusionStartUpServlet: init 07/03 23:14:02 user ColdFusionStartUpServlet: ColdFusion: Starting application services 07/03 23:14:02 user ColdFusionStartUpServlet: ColdFusion: VM version = 10.0-b19 In the jrun coldfusion-out.log, I found this: # # An unexpected error has been detected by Java Runtime Environment: # # java.lang.OutOfMemoryError: requested 23970816 bytes for jbyte in C:\BUILD_AREA\jdk6_04\hotspot\src\share\vm\prims\jni.cpp. Out of swap space? # #  Internal Error (allocation.inline.hpp:42), pid=4160, tid=3312 #  Error: jbyte in C:\BUILD_AREA\jdk6_04\hotspot\src\share\vm\prims\jni.cpp # # Java VM: Java HotSpot(TM) Server VM (10.0-b19 mixed mode windows-x86) # An error report file with more information is saved as: # C:\ColdFusion8\runtime\bin\hs_err_pid4160.log # # If you would like to submit a bug report, please visit: #   http://java.sun.com/webapps/bugreport/crash.jsp # Starting Macromedia JRun 4.0 (Build 108673), coldfusion server So... I looked at the file referenced... and you can view it here: http://cfm.pastebin.com/f208c730d Certainly looks like a memory problem but I've never seen CF behave this way. Anyone care to offer insight? -- Rick Root New Brian Vander Ark Album, songs in the music player and cool behind the scenes video at www.myspace.com/brianvanderark

This has happened twice in the last 5 minutes.  Exact same issue as described below. I have fusionreactor system metrics running and it showed only about 15% memory usage. On Thu, Jul 3, 2008 at 11:45 PM, Rick Root <rick.root@webworksllc.com> wrote: ----- Excess quoted text cut - see Original Post for more ----- -- Rick Root New Brian Vander Ark Album, songs in the music player and cool behind the scenes video at www.myspace.com/brianvanderark

Try using the JVM startup parameters described in the blog entry to see if that stops the issue from occurring. ----- Excess quoted text cut - see Original Post for more -----

James, that looks like a very distinct possibility. I'll give that a try and report back in a day or so... If it works.. I owe ya! Rick -- Rick Root New Brian Vander Ark Album, songs in the music player and cool behind the scenes video at www.myspace.com/brianvanderark

Also try to find the files hs_err_pid*.log on your disk. Not sure where they get stored on windows boxes (your java dir I would expect; on unix it's in the holding id's user directory). There is usually a good dump there that tells you what was going on. ----- Excess quoted text cut - see Original Post for more -----

> Also try to find the files hs_err_pid*.log on your disk. Not sure where they > get stored on windows boxes (your java dir I would expect; on unix it's in > the holding id's user directory). There is usually a good dump there that > tells you what was going on. Actually.. I did find that file and mentioned it in my original email: The contents are here: http://cfm.pastebin.com/f208c730d Now... Having made the change to attempt to disable hotspot.. I'm still getting the exact same error. "java.lang.OutOfMemoryError: requested 30233088 bytes for jbyte in C:\BUILD_AREA\jdk6_04\hotspot\src\share\vm\prims\jni.cpp. Out of swap space?" It's happened 15 or so times since I made the change this morning and restarted the server.  frak! So... more googling turned up this: http://kb.adobe.com/selfservice/viewContent.do?externalId=tn_18548 i'm gonna try increasing my maxPermSize again.  The site has approximately 330,000 .cfm files ... for now (until they implement a virtual directory system instead of using real directories) -- Rick Root New Brian Vander Ark Album, songs in the music player and cool behind the scenes video at www.myspace.com/brianvanderark

> So... more googling turned up this: > > http://kb.adobe.com/selfservice/viewContent.do?externalId=tn_18548 > > i'm gonna try increasing my maxPermSize again.  The site has > approximately 330,000 .cfm files ... for now (until they implement a > virtual directory system instead of using real directories) I can't set that value to 512mb.  I had it set to 392MB.  But at 512 coldfusion won't start. I also can't increase my JVM heap past 1280.  CF won't start in either condition. -- Rick Root New Brian Vander Ark Album, songs in the music player and cool behind the scenes video at www.myspace.com/brianvanderark

Have you cleared out the contents of the cfclasses folder? Try stopping CF, deleting everything in that folder (C:\CFusionMX\wwwroot\WEB-INF\cfclasses or the equivalent) and then starting CF. This might make the original error go away. ----- Excess quoted text cut - see Original Post for more -----

> Have you cleared out the contents of the cfclasses folder? Try > stopping CF, deleting everything in that folder > (C:\CFusionMX\wwwroot\WEB-INF\cfclasses or the equivalent) and then > starting CF. This might make the original error go away. Trying that now, I'll let you know. CF has restarted itself 28 times in the last 6 hours.  wtf! -- Rick Root New Brian Vander Ark Album, songs in the music player and cool behind the scenes video at www.myspace.com/brianvanderark

----- Excess quoted text cut - see Original Post for more ----- Okay that bought me 45 minutes of stability, after I deleted almost 50,000 class files. After 45 minutes, there were about 4000 class files... now there are about 5500 class files in that directory.... Rick -- Rick Root New Brian Vander Ark Album, songs in the music player and cool behind the scenes video at www.myspace.com/brianvanderark

----- Excess quoted text cut - see Original Post for more ----- CF actually restarted 3 times in about 10-15 minutes there but it has been running for 85 minutes now... of course, the site load is decreasing since it's pretty late now... only averaging 2-3 requests per second now. -- Rick Root New Brian Vander Ark Album, songs in the music player and cool behind the scenes video at www.myspace.com/brianvanderark

The only other suggestion I have is to try the 1.5 JDK. ----- Excess quoted text cut - see Original Post for more -----

Update.... server has been up over 17 hours now without a restart, serving slightly over 180,000 requests for .cfm pages. Ultimately, we backed off cfimage and switched to CFX_ImageCR3 and we seem to have considerably FEWER problems in terms of performance. Rick

Bad news! The server, running in interpreted mode, was much more reliable but the CPU usage remained quite high.  The machine has two 3.0ghz dual core CPUs. Now that the load is picking up ... the server can't keep up and the requests pile up and then CF crashes hard. I removed the jvm -Xint argument to re-enable hotspot compiling.. and the CPU usage is down and the server is able to keep up, but CF started crashing again.  Regularly. Now I've restarted CF with hotspot compilingdisabled again (put the -Xint arg back in).. and the CPU is averaging 95%, and the requests are piling up again. CRAP! -- Rick Root New Brian Vander Ark Album, songs in the music player and cool behind the scenes video at www.myspace.com/brianvanderark

Rick - Random idea, you don't have anything that is causing some code to go into an infinite loop under high load? Just figuring if you're hitting high load, and then getting multiple loops, it would cause the server to crash. Just an idea. Mark ----- Excess quoted text cut - see Original Post for more -----

This looks like a bug in the Sun java hotspot compiler (the thing that converts Java bytecode into native code after a certain number of calls to the same class). http://www.talkingtree.com/blog/index.cfm/2006/4/28/Understanding-HotSpot-in-Plain-English ----- Excess quoted text cut - see Original Post for more ----- -- mxAjax / CFAjax docs and other useful articles: http://www.bifrost.com.au/blog/

What does a stack trace look like when things start getting busy? How many requests a minute are you serving at peak usage? ~Brad Bad news! The server, running in interpreted mode, was much more reliable but the CPU usage remained quite high.  The machine has two 3.0ghz dual core CPUs. Now that the load is picking up ... the server can't keep up and the requests pile up and then CF crashes hard. I removed the jvm -Xint argument to re-enable hotspot compiling.. and the CPU usage is down and the server is able to keep up, but CF started crashing again.  Regularly. Now I've restarted CF with hotspot compilingdisabled again (put the -Xint arg back in).. and the CPU is averaging 95%, and the requests are piling up again. CRAP!

Also, did you get a chance to give the 1.5 JDK a whirl like James suggested? ~Brad The only other suggestion I have is to try the 1.5 JDK.

> What does a stack trace look like when things start getting busy? Nothing particularly unusual, just a lot of request activity.  WE monitor with Fusion Reactor, and I don't see anything unusual in terms of the activity OTHER Than when I disabled hot spot compilation, the CPU usage increased dramatically - as I would expect.  The increase in CPU means that things run slower. > How many requests a minute are you serving at peak usage? Right now the site is pretty busy and serving 466 page requests per minute over the last 28.5 minutes.  I have restricted CF to 30 active requests in the CF Administrator. > Also, did you get a chance to give the 1.5 JDK a whirl like James suggested? Not yet.  Personally, I'd like to avoid switching JDKs if possible. 1.6 is the "Default" and I'm hoping to stick with it. > Random idea, you don't have anything that is causing some code to go > into an infinite loop under high load? Nope, I'd definately see that in fusion reactor. The good news is that I'm doing an upgrade tonight to CF Enterprise, and I'm going to cluster 2 instances at 1.3gb of RAM each, and re-enable the hotspot compiler. I think it makes sense.  I turned on iis web logging and in the last 20 hours, the web server has logged almost 2 million hits and 259,194 CFM page requests. Rick

While Java 6 is the default, it breaks CF in a few respects: 1) The Java 6 classloader bug mans that it loads classes much slower than Java 1.5; a real problems for frameworks like Model-Glue etc. 2) JRun's internal webserver SSL doesn't work with Java 6. 3) It appears the hotspot compiler doesn't like your code either. We rolled back to the latest supported Java 1.5 JDK and we've had no problems so far (our servers are still just in the testing stage). > >> Also, did you get a chance to give the 1.5 JDK a whirl like James suggested? > > Not yet.  Personally, I'd like to avoid switching JDKs if possible. > 1.6 is the "Default" and I'm hoping to stick with it. -- mxAjax / CFAjax docs and other useful articles: http://www.bifrost.com.au/blog/

was switching to the 1.5 JDK "difficult"? Rick ----- Excess quoted text cut - see Original Post for more -----

Alright, I just "downgraded" to JVM 1.5_15 I re-enabled hotspot compilation (removed the -Xint arg) Had to delete all the existing class files in the cfclasses directory (prsumably because they were compiled under java 6)... *EXACT* same behavior.  Actually worse.  With hotspot compiling enabled it wouldn't stay up more than a minute, without it, it was even slower.  =( Looks like we're still on for the CF Enterprise upgrade. -- Rick Root New Brian Vander Ark Album, songs in the music player and cool behind the scenes video at www.myspace.com/brianvanderark

Ah, sorry to hear that. Hopefully, a clean install of Enterprise helps. ----- Excess quoted text cut - see Original Post for more ----- -- mxAjax / CFAjax docs and other useful articles: http://www.bifrost.com.au/blog/

So... I resolved the problem.  I think. I converted the site to use virtual directories instead of real directories.  Before I did this, my cfclasses folder had 43,000 templates.  Now it has 630.  Now it can keep *ALL* of the templates in the template cache. After replacing the real directories with virtual directories and wiping out the cfclasses folder, everything got a WHOLE lot faster. Probably will still do the CF Enterprise upgrade though. -- Rick Root New Brian Vander Ark Album, songs in the music player and cool behind the scenes video at www.myspace.com/brianvanderark

Just curious... What made you think of doing that? m!ke So... I resolved the problem.  I think. I converted the site to use virtual directories instead of real directories.  Before I did this, my cfclasses folder had 43,000 templates.  Now it has 630.  Now it can keep *ALL* of the templates in the template cache. After replacing the real directories with virtual directories and wiping out the cfclasses folder, everything got a WHOLE lot faster.

> Just curious... > > What made you think of doing that? well, I realize that templates are compiled into class files, and compiling class files is CPU intensive, so the class files get cached - on disk and into memory. Of course, CF will only cache 1024 templates by default.. and i think this is where part of my problem was.  I had 50,000 class files, most of which were the same just residing in separate subdirectories (about 2200 subdirectories containing the same 150 files). After cleaning out the cfclasses folder, it doesn't take long for there to quickly be 5000 classes in there.  And caching templates uses memory, and swapping templates in and out of memory takes CPU and such too. So I figured.. if I had fewer class files, they could *ALL* remain cached in memory.  I'd probably reduce CPU even more if I enabled trusted cache. Rick

Ah... Excellent.  Thanks for the explanation. m!ke > Just curious... > > What made you think of doing that? well, I realize that templates are compiled into class files, and compiling class files is CPU intensive, so the class files get cached - on disk and into memory. Of course, CF will only cache 1024 templates by default.. and i think this is where part of my problem was.  I had 50,000 class files, most of which were the same just residing in separate subdirectories (about 2200 subdirectories containing the same 150 files). After cleaning out the cfclasses folder, it doesn't take long for there to quickly be 5000 classes in there.  And caching templates uses memory, and swapping templates in and out of memory takes CPU and such too. So I figured.. if I had fewer class files, they could *ALL* remain cached in memory.  I'd probably reduce CPU even more if I enabled trusted cache. Rick