ColdFusion 8/Apache2 2/Win2k3 stops responding

We have CF8.01(32bit) running on Win2k3(32bit).  We are using the Apache 2.2 pre-compiled binary.  CF is installed in the multi-instance (JRUN4) mode.  We also have FusionReactor 3.x installed. The behavior we are seeing is that on indeterminate time periods the site becomes unresponsive.  We originally had a custom Apache 2.2 build (security hardened), with a custom Windows user.  We have backed those out to allow for far more access and the problem still continues.  We've been running the jrun adapater in debug mode and get the following messages: [Mon Dec 12 10:45:47 2011] [notice] jrApache[2968: 12770]  returning error page for Connection aborted [Mon Dec 12 10:45:47 2011] [notice] jrApache[2968: 12770]   proxyerror: statusMsg: JRun Connector protocol error, errorMsg: JRun Connector Protocol Error [Mon Dec 12 10:45:47 2011] [notice] jrApache[2968: 12770]   setStatus(): 503 and no error url [Mon Dec 12 10:45:47 2011] [notice] jrApache[2968: 12770]     request_rec->content_type = text/html [Mon Dec 12 10:45:47 2011] [notice] jrApache[2968: 12770]     ap_send_http_header() [Mon Dec 12 10:45:47 2011] [notice] jrApache[2968: 12770]     ap_rwrite(142 bytes) [Mon Dec 12 10:45:47 2011] [debug] mod_headers.c(756): headers: ap_headers_output_filter() [Mon Dec 12 10:45:47 2011] [notice] jrApache[2968: 12770]  127.0.0.1:51020 destroying/closing socket [1472] error 10053 [Mon Dec 12 10:45:47 2011] [notice] jrApache[2968: 12770]  127.0.0.1:51020 freeing pooled sockets Some other notes on it.  We have another vhost running (with CF) that isn't logging any errors.  the CFAdmin pages under CFIDE become unresponsive, but a custom monitoring page (has its own application.cfm) is still able to respond (sometimes).  I can see its activity in the jrun log in between actual user requests that result in 503 errors. When I've searched for any of the other errors we see in the logs they all point back to very old blog posts (MX era stuff).   Has anybody else seen this recently, and if so what did you do to resolve it?  Or does anybody have any other ideas that we can try to isolate the issue?

----- Excess quoted text cut - see Original Post for more ----- Would probably help if I had pasted the correct clipboard entry: [Fri Dec 09 09:40:52 2011] [notice] jrApache[2796: 2296]  jrun_trans: r is 0x1ac0f78, server is 0x5c7a98, virtual 1 [Fri Dec 09 09:40:52 2011] [notice] jrApache[2796: 2296]     HOST: filtered [Fri Dec 09 09:40:52 2011] [notice] jrApache[2796: 2296]  filtering /content/mer/thumbnail/dst_mbu_24monthrh.jpg (//filtered/content/mer/thumbnail/dst_mbu_24monthrh.jpg) HOST=filtered [Fri Dec 09 09:40:52 2011] [notice] jrApache[2796: 2296]  filterRequest:   no match [Fri Dec 09 09:40:52 2011] [notice] jrApache[2796: 2296]  jrun_service: r is 0x1ac0f78, server is 0x5c7a98, virtual 1 [Fri Dec 09 09:40:52 2011] [notice] jrApache[2796: 2296]  jrun_service: no handler, return DECLINED [Fri Dec 09 09:40:52 2011] [notice] jrApache[2796: 2296]  jrun_trans: r is 0x1ac0f78, server is 0x5c7a98, virtual 1 [Fri Dec 09 09:40:52 2011] [notice] jrApache[2796: 2296]     HOST: filtered [Fri Dec 09 09:40:52 2011] [notice] jrApache[2796: 2296]  filtering /content/mer/thumbnail/dst_mbu_gettherighttires.jpg (//filtered/content/mer/thumbnail/dst_mbu_gettherighttires.jpg) HOST=filtered [Fri Dec 09 09:40:52 2011] [notice] jrApache[2796: 2296]  filterRequest:   no match [Fri Dec 09 09:40:52 2011] [notice] jrApache[2796: 2296]  jrun_service: r is 0x1ac0f78, server is 0x5c7a98, virtual 1 [Fri Dec 09 09:40:52 2011] [notice] jrApache[2796: 2296]  jrun_service: no handler, return DECLINED [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  jrun_trans: r is 0x1ac0f78, server is 0x5c7a98, virtual 1 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     HOST: filtered [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  filtering / (c:/webfiles/b2c/nanosites/webroot/) HOST=filtered [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  filterRequest:   no match [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  jrun_trans: sub-request so DECLINED [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  jrun_trans: sub-request so DECLINED [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]   jrun_service: r is 0x1ac0f78, server is 0x5c7a98, virtual 1 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  jrun_service: found jrun-proxy-target index -1 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     HOST: filtered [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  filtering /index.cfm (/index.cfm) HOST=filtered [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  filterRequest:   matched *.cfm [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     cookie: CFID=32275; CFTOKEN=14275426; DT_Cookie=195799232.22528.0000; __utma=255268567.152711697.1323441627.1323441627.1323441627.1; __utmb=255268567.1.10.1323441627; __utmc=255268567; __utmz=255268567.1323441627.1.1.utmcsr=filtered|utmccn=(referral)|utmcmd=referral|utmcct=/ [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  127.0.0.1:51020 selectProxy not active [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]   127.0.0.1:51020 proxy_handleRequest() [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  PROXY_SERVICE_40 -> [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     QUERY_STRING: resource=ourProcedure [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     REQUEST_METHOD: GET [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     SCRIPT_NAME: /index.cfm [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     PATH_INFO: <null> [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     REMOTE_ADDR: filtered [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     REMOTE_ADDR: filtered [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     SERVER_NAME: filtered [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     SERVER_PORT: 80 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     SERVER_PROTOCOL: HTTP/1.1 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     PATH_TRANSLATED: <null> [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     SERVER_PORT_SECURE: 0 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     REMOTE_USER: <null> [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     AUTH_TYPE: <null> [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  127.0.0.1:51020 pooled socket [2428] returned [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]   sent/avail/result 1075/1075/1075 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  read/len/total 4/4/4 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  PROXY_GET_HEADER <- [2428] [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  read/len/total 4/4/4 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  read/len/total 15/15/15 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     Acrobat-Version: <null> [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  sent/avail/result 4/4/4 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  read/len/total 4/4/4 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  PROXY_GET_REALPATH <- [2428] [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]   read/len/total 4/4/4 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]   read/len/total 10/10/10 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     ap_escape_uri(/index.cfm) = /index.cfm [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  jrun_trans: sub-request so DECLINED [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     getRealPath(/index.cfm) = C:/webfiles/b2c/nanosites/webroot/index.cfm [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  sent/avail/result 47/47/47 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  read/len/total 4/4/4 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]   PROXY_GET_REALPATH <- [2428] [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  read/len/total 4/4/4 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  read/len/total 10/10/10 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     ap_escape_uri(/index.htm) = /index.htm [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  jrun_trans: sub-request so DECLINED [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     getRealPath(/index.htm) = C:/webfiles/b2c/nanosites/webroot/index.htm [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  sent/avail/result 47/47/47 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  read/len/total 4/4/4 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  PROXY_GET_HEADER <- [2428] [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]   read/len/total 4/4/4 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]   read/len/total 3/3/3 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     abc: <null> [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  sent/avail/result 4/4/4 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  read/len/total 4/4/4 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  PROXY_GET_HEADER <- [2428] [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  read/len/total 4/4/4 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  read/len/total 12/12/12 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     RESULTSTRUCT: <null> [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  sent/avail/result 4/4/4 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  read/len/total 4/4/4 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  PROXY_PUT_STATUS <- [2428] [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  recv failed[2428]: 10053 Connection aborted [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  read/len/total -1/4/0 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  handleJrppRequest aborting <-- [2428] [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  returning error page for Connection aborted [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  proxyerror: statusMsg: JRun Connector protocol error, errorMsg: JRun Connector Protocol Error [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  setStatus(): 503 and no error url [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     request_rec->content_type = text/html [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     ap_send_http_header() [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     ap_rwrite(142 bytes) [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  127.0.0.1:51020 destroying/closing socket [2428] error 10053 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  127.0.0.1:51020 freeing pooled sockets [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  jrCloseSocket [1428] [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  jrCloseSocket [2428] [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  127.0.0.1:51020 handleRequest failed, error 10053 Connection aborted [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296] 127.0.0.1:51020 removed proxy, seq 0 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  removeProxy - after [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     /flashservices/gateway*=coldfusion.bootstrap.BootstrapServlet [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]        127.0.0.1:51020 active: 0 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     /cfide/graphdata.cfm=coldfusion.bootstrap.BootstrapServlet [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]        127.0.0.1:51020 active: 0 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     /cfform-internal*=coldfusion.bootstrap.BootstrapServlet [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]        127.0.0.1:51020 active: 0 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     /cfide/graphdata=coldfusion.bootstrap.BootstrapServlet [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]        127.0.0.1:51020 active: 0 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     /jrunstatistics=jrun.jsp.JSPServlet [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]        127.0.0.1:51020 active: 0 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     /cffileservlet*=coldfusion.bootstrap.BootstrapServlet [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]        127.0.0.1:51020 active: 0 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     /cfformgateway*=coldfusion.bootstrap.BootstrapServlet [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]        127.0.0.1:51020 active: 0 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     /flex2gateway*=coldfusion.bootstrap.BootstrapServlet [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]        127.0.0.1:51020 active: 0 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     /servlet*=jrun.servlet.ServletInvoker [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]        127.0.0.1:51020 active: 0 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     *.cfml/*=coldfusion.bootstrap.BootstrapServlet [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]        127.0.0.1:51020 active: 0 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     *.cfm/*=coldfusion.bootstrap.BootstrapServlet [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]        127.0.0.1:51020 active: 0 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     *.cfc/*=coldfusion.bootstrap.BootstrapServlet [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]        127.0.0.1:51020 active: 0 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     *.cfswf=coldfusion.bootstrap.BootstrapServlet [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]        127.0.0.1:51020 active: 0 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     *.cfml=coldfusion.bootstrap.BootstrapServlet [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]        127.0.0.1:51020 active: 0 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     *.cfr=coldfusion.bootstrap.BootstrapServlet [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]        127.0.0.1:51020 active: 0 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     *.cfm=coldfusion.bootstrap.BootstrapServlet [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]        127.0.0.1:51020 active: 0 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     *.jst=jrun.jsp.JSTServlet [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]        127.0.0.1:51020 active: 0 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     *.jws=coldfusion.bootstrap.BootstrapServlet [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]        127.0.0.1:51020 active: 0 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     *.sws=coldfusion.bootstrap.BootstrapServlet [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]        127.0.0.1:51020 active: 0 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     *.jsp=jrun.jsp.JSPServlet [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]        127.0.0.1:51020 active: 0 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     *.swc=coldfusion.bootstrap.BootstrapServlet [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]        127.0.0.1:51020 active: 0 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     *.cfc=coldfusion.bootstrap.BootstrapServlet [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]        127.0.0.1:51020 active: 0 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]     *.as=coldfusion.bootstrap.BootstrapServlet [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]        127.0.0.1:51020 active: 0 [Fri Dec 09 09:40:53 2011] [notice] jrApache[2796: 2296]  jrun_service: OK

> The behavior we are seeing is that on indeterminate time periods the site becomes unresponsive. Is there anything in your ColdFusion logfiles? Jochem -- Jochem van Dieten http://jochem.vandieten.net/