best practice for building a large text file?

All: What’s the best practice for building a large text file? Is it better to use CFFILE and APPLEND or use an array then once you have the information use arraytolist and then use CFFILE to create the text file? I have a 30 meg text file that I’m building line by line. Regards, James Blaha

Can you use your database to do it? ----- Excess quoted text cut - see Original Post for more -----

For a very large file, something in between.  If you build a big file within memory (in an array or text variable) then you risk bringing the machine to a standstill or even crashing it. If you append to a file line by line a couple of million times you'll also tax the system and won't be doing yourself any favors, since the program will probably take much longer to execute than if you built the file in memory first and merely dumped it to disk at the end. I'd figure out the average line length and pick some number of lines to dump to the file each time based on the specs of the machine. There's no real need to use arrays and lists.  Just use a simple variable and terminate each line with a CR/LF pair. <cfset outputfile = "C:\myfile.txt"> <cfset linesperwrite = 1000> <cfset crlf = Chr(13) & Chr(10)> <cffile action="write" file="#outputfile#" output="" addnewline="no"> <cfset x = ""> <cfset linecount = 0> <cfloop (until you're done)>   <cfset some_stuff = ...>   <cfset x = x & "#some_stuff#" & crlf>   <cfset linecount = linecount + 1>   <cfif linecount gte linesperwrite>     <cffile action="append" file="#outputfile#" output="#x#" addnewline="no">     <cfset x = "">     <cfset linecount = 0>   </cfif> </cfloop> <!--- Don't forget to dump any remaining lines to the file ---> <cffile action="append" file="#outputfile#" output="#x#" addnewline="no"> > All: > > What’s the best practice for building a large text file? Is it better to > use CFFILE and APPLEND or use an array then once you have the > information use arraytolist and then use CFFILE to create the text file? > > I have a 30 meg text file that I’m building line by line.

Jim, Thank you so much for your reply. I really like your approach. Can you please just explain this line below. I'm building the file looping through a query. <cfloop (until you're done)> Have a great weekend! Regards, JB Jim McAtee wrote: ----- Excess quoted text cut - see Original Post for more -----

I didn't know the looping condition for your particular application.  You'd just loop over your query. <cfloop query="myquery>   ... </cfloop> > Can you please just explain this line below. I'm building the file > looping through a query. > > <cfloop (until you're done)>

When I'm building files, I usually use build up a string/array (depending on the situation) in memory, and then flush it to the file periodically, because string concatenation gets expensive as the string gets large.  This example (a horribly contrived one) uses the string concatenation method, and has code to use the array append method commented out for illustrative purposes.  I didn't test, so it may not be perfect, but it should be close. <cfset filename = expandPath('./data_file.txt') /> <!--- data file ---> <cfset s = "" /> <!--- temp variable ---> <!--- <cfset a = arrayNew(1) /> ---> <cfset flush_frequency = 50 /> <!--- how often to flush to file ---> <cfloop query="get">    <cfset s = s & "name: #name#, dob: #dateOfBirth#" />    <!---    <cfset arrayAppend(a, "name: #name#, dob: #dateOfBirth#") />    --->    <cfif month(dateOfBirth) EQ month(now()) and day(dateOfBirth) EQ day(now())>       <cfset s = s & " happy birthday!" />       <!---       <cfset a[arrayLen(a)] = a[arrayLen(a)] & " happy birthday" />       --->    </cfif>    <cfif currentrow MOD flush_frequency EQ 0>       <cffile action="append"          file="#filename#"          output="#s#" />       <cfset s = "" />       <!---       <cffile action="append"          file="#filename#"          output="#arrayToList(a, chr(10))##chr(10)#" />       <cfset a = arrayNew(1) />       --->    </cfif> </cfloop> <cffile action="append"    file="#filename#"    output="#s#" /> <!--- <cffile action="append"    file="#filename#"    output="#arrayToList(a, chr(10))#" /> ---> I haven't done any formal performance testing, but it's not noticably differe for small data sets, and noticably faster for large data sets.  Just don't forget that <cfset s = "" /> after the flush, or you'll kick your server's ass, trust me.  ; ) HTH, barneyb --- Barney Boisvert, Senior Development Engineer AudienceCentral bboisvert@audiencecentral.com voice : 360.756.8080 x12 fax   : 360.647.5351 www.audiencecentral.com ----- Excess quoted text cut - see Original Post for more -----

Barney, Thank you so much for your reply. I need to digest this one. Have a great weekend! Regards, JB Barney Boisvert wrote: ----- Excess quoted text cut - see Original Post for more -----

It's all but identical to what Jim said, except I forgot to add newlines to my string, and I have the commented array usage in there. --- Barney Boisvert, Senior Development Engineer AudienceCentral bboisvert@audiencecentral.com voice : 360.756.8080 x12 fax   : 360.647.5351 www.audiencecentral.com ----- Excess quoted text cut - see Original Post for more -----

Wait - it's only Tuesday... Can I have your job? ;p > Sent: Tuesday, July 01, 2003 10:44 AM { ... } >Have a great weekend! >Regards, >JB

Jeff, Its been a long day! :-) Regards, JB Jeff Beer wrote: ----- Excess quoted text cut - see Original Post for more -----