loopy loop!

I have this loop that runs ~22,000 iterations.  It brings the server to its knees.  The RAM usage skyrockets above 800 Megs cutting into major virtual memory.  But when I just write the output instead of storing it to a variable, it runs fine.  I'm trying to retrieve a list of email addresses and then write them to a file.     <cfscript>     emails = "";     for( i = 1; i lte list.recordcount; i=i+1 )       {       emails = emails & list.email[i] & chr(13) & chr(10);       }     </cfscript> I can't imagine that the list in memory would take that much room.  Does anyone have a suggestion? -- Chris Edwards Web Application Developer Outer Banks Internet, Inc. 252-441-6698 chris.edwards@obinet.com http://www.OuterBanksInternet.com

How about replacing it with: Warning - Not tested ! <cfscript> emails = replace(valuelist(list.email,"|"),"|",chr(13)&chr(10),"ALL"); </cfscript> Marl ----- Excess quoted text cut - see Original Post for more -----

> <cfscript> > emails = replace(valuelist(list.email,"|"),"|",chr(13)&chr(10),"ALL"); > </cfscript> before 182713 milliseconds-- After 77502 milliseconds Thats better.  Can I make it go faster?  This is on a windows box( p3 1ghtz, 512ram ) CF5. -- Chris Edwards Web Application Developer Outer Banks Internet, Inc. 252-441-6698 chris.edwards@obinet.com http://www.OuterBanksInternet.com

I prepare inventory files to upload to Half.com and Amazon marketplace. Some 150k records or so. I used to run the script so that I appended each row to a variable, then when done, wrote this variable to a file - thinking that the variable would take less time that writing to a file 150k times. WRONG! I rewrote the script to write to the file each iteration instead of memory and the time it took to run when from 22 hours to 1 hour. I wrote some sample code to compare the two loops and found that the time it took to complete the variable->file loop increased almost exponentially as the number of iterations went up while the file loop increased in a nice linear fashion. ----- Excess quoted text cut - see Original Post for more -----

----- Excess quoted text cut - see Original Post for more ----- It seems like it would make sense that WriteOutput takes less memory than storing to a variables. If you think about it, once it does the WriteOutput(), CF no longer needs to keep track of what's inside it -- it's no longer in memory. But if you store it to a variable, CF needs to go find that variable and then append to it and then store it again. That overhead is minimal with a smaller number of iterations. But 22,000 times is a lot of times!! Are you sure that your approach is the best solution? I obviously have no idea what your ultimate goal is, but it looks from your code like you're trying to build a list of email address from a database with Windows carriage returns. Have you tried maybe using an array instead? Perhaps they have better performance than what you're doing? Or you could use ValueList() on the query column using a delimiter of "#Chr(13)##Chr(10)#". Those are just off the top of my head, but 22,000 of anything is going to take a lot of processing time, so I wouldn't be so suprised to see what you're seeing. Regards, Dave.

file I/O operations are very fast - their speed tends to be under-estimated by developers.  One thing you should watch out for are concurrency issues... I recommend using named locks whenever accessing the local file system. ~Simon Simon Horwith Macromedia Certified Instructor Certified Advanced ColdFusion MX Developer Certified Flash MX Developer CFDJList - List Administrator Fig Leaf Software 1400 16th St NW, # 220 Washington DC 20036 202.797.6570 (direct line) http://www.figleaf.com I prepare inventory files to upload to Half.com and Amazon marketplace. Some 150k records or so. I used to run the script so that I appended each row to a variable, then when done, wrote this variable to a file - thinking that the variable would take less time that writing to a file 150k times. WRONG! I rewrote the script to write to the file each iteration instead of memory and the time it took to run when from 22 hours to 1 hour. I wrote some sample code to compare the two loops and found that the time it took to complete the variable->file loop increased almost exponentially as the number of iterations went up while the file loop increased in a nice linear fashion. > I have this loop that runs ~22,000 iterations.  It brings the server to its ----- Excess quoted text cut - see Original Post for more -----

> file I/O operations are very fast - their speed tends to be > under-estimated by developers.  One thing you should watch > out for are concurrency issues... I recommend using named > locks whenever accessing the local file system. If your process is taking several hours you may be able to speed things up considerably by doing your initial I/O on a RAM drive.  In general for huge jobs like this writing directly to the file system faster because CF does not have to maintain memory space for the entire result. However it's still hamstrung (a little) by the physical speed of the disk.  In many corporate environments write-ahead caching is not allowed for data integrity issues - this will really kill peformance if you're used to having it turned on (often the case on a developer's local machine). A RAM disk effectively eliminates the physical disk lag, but maintains the "append and forget" speed of direct-to-file processing.  Once you're done a single file operation moves the file from the RAM Disk to the physical disk. Very peppy. Jim Davis