Home of the ColdFusion Community
|
Mailing Lists
|
Home /
Groups /
ColdFusion Talk (CF-Talk)
best practice for building a large text file?
Author: James Blaha
Short Link: http://www.houseoffusion.com/groups/cf-talk/thread.cfm/threadid:25210#126769
Jeff,
Its been a long day! :-)
Regards,
JB
Jeff Beer wrote:
----- Excess quoted text cut - see Original Post for more -----
Author: Jeff Beer
Short Link: http://www.houseoffusion.com/groups/cf-talk/thread.cfm/threadid:25210#126766
Wait - it's only Tuesday...
Can I have your job? ;p
> Sent: Tuesday, July 01, 2003 10:44 AM
{ ... }
>Have a great weekend!
>Regards,
>JB
Author: Barney Boisvert
Short Link: http://www.houseoffusion.com/groups/cf-talk/thread.cfm/threadid:25210#126750
It's all but identical to what Jim said, except I forgot to add newlines to
my string, and I have the commented array usage in there.
---
Barney Boisvert, Senior Development Engineer
AudienceCentral
bboisvert@audiencecentral.com
voice : 360.756.8080 x12
fax : 360.647.5351
www.audiencecentral.com
----- Excess quoted text cut - see Original Post for more -----
Author: Jim McAtee
Short Link: http://www.houseoffusion.com/groups/cf-talk/thread.cfm/threadid:25210#126749
I didn't know the looping condition for your particular application. You'd
just loop over your query.
<cfloop query="myquery>
...
</cfloop>
> Can you please just explain this line below. I'm building the file
> looping through a query.
>
> <cfloop (until you're done)>
Author: James Blaha
Short Link: http://www.houseoffusion.com/groups/cf-talk/thread.cfm/threadid:25210#126747
Barney,
Thank you so much for your reply. I need to digest this one.
Have a great weekend!
Regards,
JB
Barney Boisvert wrote:
----- Excess quoted text cut - see Original Post for more -----
Author: James Blaha
Short Link: http://www.houseoffusion.com/groups/cf-talk/thread.cfm/threadid:25210#126745
Jim,
Thank you so much for your reply. I really like your approach.
Can you please just explain this line below. I'm building the file
looping through a query.
<cfloop (until you're done)>
Have a great weekend!
Regards,
JB
Jim McAtee wrote:
----- Excess quoted text cut - see Original Post for more -----
Author: Barney Boisvert
Short Link: http://www.houseoffusion.com/groups/cf-talk/thread.cfm/threadid:25210#126744
When I'm building files, I usually use build up a string/array (depending on
the situation) in memory, and then flush it to the file periodically,
because string concatenation gets expensive as the string gets large. This
example (a horribly contrived one) uses the string concatenation method, and
has code to use the array append method commented out for illustrative
purposes. I didn't test, so it may not be perfect, but it should be close.
<cfset filename = expandPath('./data_file.txt') /> <!--- data file
--->
<cfset s = "" /> <!--- temp variable --->
<!---
<cfset a = arrayNew(1) />
--->
<cfset flush_frequency = 50 /> <!--- how often to flush to file --->
<cfloop query="get">
<cfset s = s & "name: #name#, dob: #dateOfBirth#" />
<!---
<cfset arrayAppend(a, "name: #name#, dob: #dateOfBirth#") />
--->
<cfif month(dateOfBirth) EQ month(now()) and day(dateOfBirth) EQ
day(now())>
<cfset s = s & " happy birthday!" />
<!---
<cfset a[arrayLen(a)] = a[arrayLen(a)] & " happy birthday" />
--->
</cfif>
<cfif currentrow MOD flush_frequency EQ 0>
<cffile action="append"
file="#filename#"
output="#s#" />
<cfset s = "" />
<!---
<cffile action="append"
file="#filename#"
output="#arrayToList(a, chr(10))##chr(10)#" />
<cfset a = arrayNew(1) />
--->
</cfif>
</cfloop>
<cffile action="append"
file="#filename#"
output="#s#" />
<!---
<cffile action="append"
file="#filename#"
output="#arrayToList(a, chr(10))#" />
--->
I haven't done any formal performance testing, but it's not noticably
differe for small data sets, and noticably faster for large data sets. Just
don't forget that <cfset s = "" /> after the flush, or you'll kick your
server's ass, trust me. ; )
HTH,
barneyb
---
Barney Boisvert, Senior Development Engineer
AudienceCentral
bboisvert@audiencecentral.com
voice : 360.756.8080 x12
fax : 360.647.5351
www.audiencecentral.com
----- Excess quoted text cut - see Original Post for more -----
Author: Jim McAtee
Short Link: http://www.houseoffusion.com/groups/cf-talk/thread.cfm/threadid:25210#126743
For a very large file, something in between. If you build a big file within
memory (in an array or text variable) then you risk bringing the machine to a
standstill or even crashing it.
If you append to a file line by line a couple of million times you'll also tax
the system and won't be doing yourself any favors, since the program will
probably take much longer to execute than if you built the file in memory first
and merely dumped it to disk at the end.
I'd figure out the average line length and pick some number of lines to dump to
the file each time based on the specs of the machine.
There's no real need to use arrays and lists. Just use a simple variable and
terminate each line with a CR/LF pair.
<cfset outputfile = "C:\myfile.txt">
<cfset linesperwrite = 1000>
<cfset crlf = Chr(13) & Chr(10)>
<cffile action="write" file="#outputfile#" output="" addnewline="no">
<cfset x = "">
<cfset linecount = 0>
<cfloop (until you're done)>
<cfset some_stuff = ...>
<cfset x = x & "#some_stuff#" & crlf>
<cfset linecount = linecount + 1>
<cfif linecount gte linesperwrite>
<cffile action="append" file="#outputfile#" output="#x#"
addnewline="no">
<cfset x = "">
<cfset linecount = 0>
</cfif>
</cfloop>
<!--- Don't forget to dump any remaining lines to the file --->
<cffile action="append" file="#outputfile#" output="#x#" addnewline="no">
> All:
>
> What’s the best practice for building a large text file? Is it better to
> use CFFILE and APPLEND or use an array then once you have the
> information use arraytolist and then use CFFILE to create the text file?
>
> I have a 30 meg text file that I’m building line by line.
Author: David Brown
Short Link: http://www.houseoffusion.com/groups/cf-talk/thread.cfm/threadid:25210#126740
Can you use your database to do it?
----- Excess quoted text cut - see Original Post for more -----
Author: James Blaha
Short Link: http://www.houseoffusion.com/groups/cf-talk/thread.cfm/threadid:25210#126738
All:
What’s the best practice for building a large text file? Is it better to
use CFFILE and APPLEND or use an array then once you have the
information use arraytolist and then use CFFILE to create the text file?
I have a 30 meg text file that I’m building line by line.
Regards,
James Blaha
|
May 24, 2012
|
Latest Fusion Authority Articles
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
