Home of the ColdFusion Community
|
Mailing Lists
|
Home /
Groups /
ColdFusion Talk (CF-Talk)
loopy loop!
Author: Jim Davis
Short Link: http://www.houseoffusion.com/groups/cf-talk/thread.cfm/threadid:22822#115068
> file I/O operations are very fast - their speed tends to be
> under-estimated by developers. One thing you should watch
> out for are concurrency issues... I recommend using named
> locks whenever accessing the local file system.
If your process is taking several hours you may be able to speed things
up considerably by doing your initial I/O on a RAM drive. In general
for huge jobs like this writing directly to the file system faster
because CF does not have to maintain memory space for the entire result.
However it's still hamstrung (a little) by the physical speed of the
disk. In many corporate environments write-ahead caching is not allowed
for data integrity issues - this will really kill peformance if you're
used to having it turned on (often the case on a developer's local
machine).
A RAM disk effectively eliminates the physical disk lag, but maintains
the "append and forget" speed of direct-to-file processing. Once you're
done a single file operation moves the file from the RAM Disk to the
physical disk.
Very peppy.
Jim Davis
Author: Simon Horwith
Short Link: http://www.houseoffusion.com/groups/cf-talk/thread.cfm/threadid:22822#115054
file I/O operations are very fast - their speed tends to be under-estimated
by developers. One thing you should watch out for are concurrency issues...
I recommend using named locks whenever accessing the local file system.
~Simon
Simon Horwith
Macromedia Certified Instructor
Certified Advanced ColdFusion MX Developer
Certified Flash MX Developer
CFDJList - List Administrator
Fig Leaf Software
1400 16th St NW, # 220
Washington DC 20036
202.797.6570 (direct line)
http://www.figleaf.com
I prepare inventory files to upload to Half.com and Amazon marketplace.
Some 150k records or so.
I used to run the script so that I appended each row to a variable, then
when done, wrote this variable to a file - thinking that the variable
would take less time that writing to a file 150k times.
WRONG!
I rewrote the script to write to the file each iteration instead of memory
and the time it took to run when from 22 hours to 1 hour.
I wrote some sample code to compare the two loops and found that the time
it took to complete the variable->file loop increased almost exponentially
as the number of iterations went up while the file loop increased in a
nice linear fashion.
> I have this loop that runs ~22,000 iterations. It brings the server to
its
----- Excess quoted text cut - see Original Post for more -----
Author: Tony Schreiber
Short Link: http://www.houseoffusion.com/groups/cf-talk/thread.cfm/threadid:22822#115052
I prepare inventory files to upload to Half.com and Amazon marketplace.
Some 150k records or so.
I used to run the script so that I appended each row to a variable, then
when done, wrote this variable to a file - thinking that the variable
would take less time that writing to a file 150k times.
WRONG!
I rewrote the script to write to the file each iteration instead of memory
and the time it took to run when from 22 hours to 1 hour.
I wrote some sample code to compare the two loops and found that the time
it took to complete the variable->file loop increased almost exponentially
as the number of iterations went up while the file loop increased in a
nice linear fashion.
----- Excess quoted text cut - see Original Post for more -----
Author: Chris Edwards
Short Link: http://www.houseoffusion.com/groups/cf-talk/thread.cfm/threadid:22822#115046
> <cfscript>
> emails = replace(valuelist(list.email,"|"),"|",chr(13)&chr(10),"ALL");
> </cfscript>
before 182713 milliseconds--
After 77502 milliseconds
Thats better. Can I make it go faster? This is on a windows box( p3 1ghtz,
512ram ) CF5.
--
Chris Edwards
Web Application Developer
Outer Banks Internet, Inc.
252-441-6698
chris.edwards@obinet.com
http://www.OuterBanksInternet.com
Author: Dave Carabetta
Short Link: http://www.houseoffusion.com/groups/cf-talk/thread.cfm/threadid:22822#115034
----- Excess quoted text cut - see Original Post for more -----
It seems like it would make sense that WriteOutput takes less memory than
storing to a variables. If you think about it, once it does the
WriteOutput(), CF no longer needs to keep track of what's inside it -- it's
no longer in memory. But if you store it to a variable, CF needs to go find
that variable and then append to it and then store it again. That overhead
is minimal with a smaller number of iterations. But 22,000 times is a lot of
times!!
Are you sure that your approach is the best solution? I obviously have no
idea what your ultimate goal is, but it looks from your code like you're
trying to build a list of email address from a database with Windows
carriage returns. Have you tried maybe using an array instead? Perhaps they
have better performance than what you're doing? Or you could use ValueList()
on the query column using a delimiter of "#Chr(13)##Chr(10)#".
Those are just off the top of my head, but 22,000 of anything is going to
take a lot of processing time, so I wouldn't be so suprised to see what
you're seeing.
Regards,
Dave.
Author: Marlon Moyer
Short Link: http://www.houseoffusion.com/groups/cf-talk/thread.cfm/threadid:22822#115032
How about replacing it with:
Warning - Not tested !
<cfscript>
emails = replace(valuelist(list.email,"|"),"|",chr(13)&chr(10),"ALL");
</cfscript>
Marl
----- Excess quoted text cut - see Original Post for more -----
Author: Chris Edwards
Short Link: http://www.houseoffusion.com/groups/cf-talk/thread.cfm/threadid:22822#115027
I have this loop that runs ~22,000 iterations. It brings the server to its
knees. The RAM usage skyrockets above 800 Megs cutting into major virtual
memory. But when I just write the output instead of storing it to a
variable, it runs fine. I'm trying to retrieve a list of email addresses
and then write them to a file.
<cfscript>
emails = "";
for( i = 1; i lte list.recordcount; i=i+1 )
{
emails = emails & list.email[i] & chr(13) & chr(10);
}
</cfscript>
I can't imagine that the list in memory would take that much room. Does
anyone have a suggestion?
--
Chris Edwards
Web Application Developer
Outer Banks Internet, Inc.
252-441-6698
chris.edwards@obinet.com
http://www.OuterBanksInternet.com
|
May 21, 2013
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
