The Woes of CFThread -- going out of my mind!

Well I fail. Can anybody point out what is fundamentally wrong with this code.  Such that it has at least a 50% chance of complete failure resulting in a non-responsive ColdFusion server that must be restarted.  What gets me is that sometimes it works completely correct and produces all the desired files.  The next time, with the exact same code and data, it fails and hangs the server, but generates no exceptions or errors.  At least none that I have been able to find. <cfsetting requesttimeout="1800"> <cfinclude template = "scripts.cfm"> <cfset Application.DSN = LIC_DSN> <h1>PROCESSING</h1> <cfset variables.pathDate = #dateFormat(now(),'yyyy')#/> <cfset variables.pIssueDate = #dateFormat(now(),'mm/dd/yy')#> <!--- GET RENEWAL ID ---> <cfstoredproc procedure="REPORT_UTILS.getBatchRenewals"   dataSource = "#LIC_DSN#">   <cfprocresult name = "variables.ResultFirms"> </cfstoredproc> <cfset variables.threads = ""> <ul> <cfloop index = "i" from="1" to="#ResultFirms.recordcount#">     <cfif i GT 10><cfbreak></cfif><!--- Development throttle ... only process first 15 rows --->     <cfset variables.threads = listAppend(variables.threads,'batchRenew_thread#i#')>     <li><cfoutput>batchRenew_thread#i#</cfoutput></li>        <cfthread name="batchRenew_thread#i#" threadIndex="#i#" threadPathDate="#pathDate#" action="run">     <cfset var vRenewalReport = "">     <cftry>                <!--- GET RENEWAL ID --->         <cfstoredproc procedure="REPORT_UTILS.getRenewalSequenceId"             dataSource = "#LIC_DSN#">             <cfprocparam type="out" variable="thread.vRenewalSeq" CFSQLType="CF_SQL_VARCHAR">         </cfstoredproc>                <!--- PATH --->         <cfset thread.vOutputFilePath = variables.RENEWALS_FILE_PATH &                variables.pathDate & "\" &                      Trim(variables.ResultFirms["firmno"][threadIndex]) & "\"/>                        <!--- CREATE OUTPUT PATH --->         <cfif DirectoryExists(thread.vOutputFilePath) is False>             <cfdirectory action="create" directory="#thread.vOutputFilePath#"/>         </cfif>                  <!--- FILE NAME --->         <cfset thread.vOutputFilePath &= GetFileName()/>                <!--- BUILD REPORT --->         <cfreport template="ren.cfr" format="pdf" name="vRenewalReport">                 <cfreportparam name="pFirmNo" value="#variables.ResultFirms['firmno'][threadIndex]#">             <cfreportparam name="pIssueDate" value="#DateFormat(variables.pIssueDate, 'mm/dd/yy')#">             <cfreportparam name="pRenewalId" value="#thread.vRenewalSeq#">         </cfreport>                <cfpdf action="write" source="vRenewalReport" destination="#thread.vOutputFilePath#" overwrite="yes">                <cfset vRenewalReport = "">                <!--- INSERT RENEWAL --->         <cfstoredproc procedure="REPORT_UTILS.insertRenewal"             dataSource = "#LIC_DSN#">             <cfprocparam type="in" value="#thread.vRenewalSeq#" CFSQLType="CF_SQL_NUMERIC">             <cfprocparam type="in" value="#variables.ResultFirms['firmno'][threadIndex]#" CFSQLType="CF_SQL_NUMERIC">             <cfprocparam type="in" value="#threadPathDate#" CFSQLType="CF_SQL_NUMERIC">         </cfstoredproc>                <cfstoredproc procedure="REPORT_UTILS.insertRenewalProductsForFirm"             dataSource = "#LIC_DSN#">             <cfprocparam type="in" value="#thread.vRenewalSeq#" CFSQLType="CF_SQL_NUMERIC">             <cfprocparam type="in" value="#ResultFirms['firmno'][threadIndex]#" CFSQLType="CF_SQL_NUMERIC">         </cfstoredproc>                <cfcatch type="any">             <cflock name="treadLogLock" timeout="5" type="exclusive">                     <cfset thread.cfcatch = duplicate(cfcatch)>             </cflock>         </cfcatch>     </cftry>     </cfthread>     <!--- Pause every x threads to allow garbage collection.  --->     <cfif i MOD 10 EQ 0>         <cfthread action="join" name="#variables.threads#" timeout="2500"/>                <!---  Loop over threads and kill any hanging threads --->         <cfloop collection="#cfThread#" item="th">             <cfif cfThread[th].status NEQ 'COMPLETED'>                 <cfthread action="terminate" name="#th#"/>             </cfif>         </cfloop> <h2>SLEEPING <cfoutput>#timeFormat(now(),"hh:mm:ss:l")#</cfoutput></h2>               <cfset sleep(10000)>         <cfset variables.threads = "">         </ul><ul>     </cfif> </cfloop> </ul> <h2>DONE</h2> <cfthread action="join" name="#structKeyList(cfThread)#" timeout="10000" /> <!---  Loop over threads and kill any hanging threads ---> <cfloop collection="#cfThread#" item="th">     <cfif cfThread[th].status NEQ 'COMPLETED'>         <cfthread action="terminate" name="#th#"/>     </cfif> </cfloop> <cfoutput> <ul> <cfloop collection="#cfThread#" item="th">     <li style="margin-top: .5em;">#th#         <ul>         <cfloop collection="#cfThread[th]#" item="key">                        <li>#key#:   <cfif isSimpleValue(cfThread[th][key])>#cfThread[th][key]#<cfelse><cfdump var="#cfThread[th][key]#"></cfif></li>         </cfloop>         </ul>     </li> </cfloop> </ul> </cfoutput>

Well, Did you check what is the thread doing when you say the Server is hung? Can you take a thread dump and post that here? You can take the thread dump using Server monitor or you can refer to my post http://coldfused.blogspot.com/2005/11/thread-dumps.html however, I see few potential problems in the code - In the loop, when you fire 10th threads, you join with timeout of 2.5 seconds, and if by that time, all the threads are not finished, you kill them all. So it might happen that threads would not have finished by that time - If you see the docs and if you see my presentation, everywhere it is said that you should use 'terminate' only when absolutely necessary. Don't overuse it as it might lead the thread in some inconsistent state. You should code it in such way that you don't need to terminate the threads most of the time - I don't think you should duplicate the catch. get the message from catch and set it to the thread. For test purpose you can remove catch and see what is the error that you get. Always remember that the thread does not write any output or error to the response. It will be available in the thread scope. Hope that helps. Rupesh

Well, Did you check what is the thread doing when you say the Server is hung? Can you take a thread dump and post that here? You can take the thread dump using Server monitor or you can refer to my post http://coldfused.blogspot.com/2005/11/thread-dumps.html however, I see few potential problems in the code - In the loop, when you fire 10th threads, you join with timeout of 2.5 seconds, and if by that time, all the threads are not finished, you kill them all. So it might happen that threads would not have finished by that time - If you see the docs and if you see my presentation, everywhere it is said that you should use 'terminate' only when absolutely necessary. Don't overuse it as it might lead the thread in some inconsistent state. You should code it in such way that you don't need to terminate the threads most of the time - I don't think you should duplicate the catch. get the message from catch and set it to the thread. For test purpose you can remove catch and see what is the error that you get. Always remember that the thread does not write any output or error to the response. It will be available in the thread scope. Hope that helps. Rupesh

Ian, ----- Excess quoted text cut - see Original Post for more ----- I wonder if the problem isn't with <cfthread /> but actually with heavy usage of either <cfreport /> or <cfpdf /> tags. Are you creating large reports? It could be a memory issue your running in to. How much RAM is available to the JVM? I would try simplifying things. Does it always work if you comment out the <cfreport /> and <cfpdf /> tags? If it does, what happens if you comment out the <cfreport /> tag and just write a very simply PDF? I could see how if you're generating some large PDFs, that without enough RAM it could drive the JVM nuts. -Dan

Excellent point Dan!

Dan G. Switzer, II wrote: ----- Excess quoted text cut - see Original Post for more ----- The memory usage of the PDF report generation is defiantly a problem.   It is what we where trying to address with the usage of the thread tag.   The idea being that by splitting the process into separate threads and throttle them down would allow the server time to clean up memory used in previous iterations. That is a topic for a seperate discussion.  How does <cfreport...> work in large iterations.  We found that no matter what we did, the memory used by the report tag would just climb and climb each iteration unless we broke up the process into completely separate HTTP requests. By creating a template that would process a small number of reports per request and then send a page to the browser with a META refresh to re-request the page in one second.  This works fine.

----- Excess quoted text cut - see Original Post for more ----- While certainly not ideal, what if you split things up like this: 1) You create one template that creates a report based on some kind of unique key identifier. 2) You create a second template that loops through all the reports you need to create and call the first template using CFHTTP. This should at least sort of emulate the "meta refresh" technique, but allows you to automate the process w/out relying on a browser for the meta refresh. -Dan