size limit for cffile?

I have a client who has the need to read in real big text files daily. 100mb plus.  Anyone know what the functional limit is on cffile, if any?  Server has 2gb of RAM and is a dual Xeon. -- --m@Robertson-- Janitor, MSB Web Systems mysecretbase.com

Matt Robertson wrote: > I have a client who has the need to read in real big text files daily. >  100mb plus.  Anyone know what the functional limit is on cffile, if > any?  Server has 2gb of RAM and is a dual Xeon. > If I were on another list, I'd just tell you to search the archives. ;) but instead cuz this is a friendly list, I'll post some links to other threads where this is discussed.. http://www.houseoffusion.com/cf_lists/messages.cfm/forumid:4/threadid:13645 http://www.houseoffusion.com/cf_lists/messages.cfm/forumid:12/threadid:927 and others.. Rick

Well this isn't quite the same old question, and its my fault for not being clear.  I'm asking strictly about the READ attribute.  I already have the file placed on the server via SSH. And the plot has thickened since I first posted.  What I am finding is I can get the file into memory (an 89 mb file so far) but once its in there CF doesn't have any playing-around room and crashes with an out-of-memory error doing some after-read processing. Am looking into how I can increase how much memory CF can address now. Server has 2gb and looks like I can't take more than 1.128gb. On 7/12/06, Rick Root <rick.root@webworksllc.com> wrote:

With really big files you may not want to read the entire file into memory the way <cffile> does. I've seen some folks user Java file i/o calls to get around this, essentially inline Java calls to read the file line by line. This will be slower, but, it'll work. --- Ben Well this isn't quite the same old question, and its my fault for not being clear.  I'm asking strictly about the READ attribute.  I already have the file placed on the server via SSH. And the plot has thickened since I first posted.  What I am finding is I can get the file into memory (an 89 mb file so far) but once its in there CF doesn't have any playing-around room and crashes with an out-of-memory error doing some after-read processing. Am looking into how I can increase how much memory CF can address now. Server has 2gb and looks like I can't take more than 1.128gb. On 7/12/06, Rick Root <rick.root@webworksllc.com> wrote:

Sorry, Matt, I didn't read close enough either!  There are threads out there also relating to READING large files. However, I'd probably also suggest looking into using the java file I/O stuff to avoid the issues relating to load a 100MB file into memory and dealing with it. Because I love writing java code in CF, I wrote some sample code for you, and even tested it. <cfscript>   srcFile = "E:\Inetpub\ajaxCFC\documentation\license.txt";   // create a FileReader object   fr = createObject("java","java.io.FileReader");   // Call the constructure with the source file path   fr.init(srcFile);   // create a BufferedReader object   br = createObject("java","java.io.BufferedReader");   // call the constructor with the FileReader as the arg   br.init(fr);   // read the first line   str = br.readLine();   // loop ... str will be undefined if there are no more lines   while (isDefined("str")) {     // do stuff with the string     writeOutput(str);     // read the next line so we can continue the loop     str = br.readLine();   }   // close the buffered reader object   br.close(); </cfscript>

oooh.  Thanks Rick!  One of these days I need to learn that stuff. Reminds me of something else too.  Geez what a bum I am.  Check your paypal box in a bit. -- --m@Robertson-- Janitor, MSB Web Systems mysecretbase.com

Hmm..... Interesting idea.... Calling JAVA objects..... I should really get into that..... Power fo Java with Coldfusion ease.... On 7/12/06, Matt Robertson <websitemaker@gmail.com> wrote:

B V wrote: > Hmm..... Interesting idea.... Calling JAVA objects..... I should really get > into that..... Power fo Java with Coldfusion ease.... I always find it fun to do java integration like this =) I just blogged this whole thing too.... http://www.opensourcecf.com/1/2006/07/Reading-large-files-with-java-versus-CFFILE. cfm Rick

For what it's worth, I just ran some tests reading a 4MB file, looping through line by line and performing some simple math (cnt = cnt + 1) The java method was CONSISTENTLY around 375ms on my server.  I hit reload like 30 times and it was never below 350ms and never above 500ms,   and almost always below 400ms. Using CFFILE, it was much more erratic... coming in as low as 450ms but often coming in at 2000ms and higher. Rick Rick Root wrote:

I'll definetly try that..... Java objects are interesting..... I always wanted to get into Java, but it seemed to suck so much because of the syntax, but If I can just call objects from coldfusion  for the power....... Two Java Objects that would really make my life easier would be a). An object to monitor CPU and RAM usage/sessions b). An object that could Exec Shell (I hate doing it with PHP, because I have to implement a complicated security system.....) On 7/12/06, Rick Root <rick.root@webworksllc.com> wrote:

Rick what do you do if you can't run the whole operation from within a cfscript block?  I need to do a *whole* bunch of stuff (like cfqueries that select, insert and update) inside of that while loop and I can't seem to get a non-cfscript analog of that loop working.  I have to be missing something stupid as I could not get a conditional cfloop to work with an isdefined in the condition. -- --m@Robertson-- Janitor, MSB Web Systems mysecretbase.com

All the statements in that example have CF tag equivalents (CFScript syntax is much more similar to Java syntax, making it easier and clearer to port Java code examples). Assignment and method calls can be made into CFSet statements, and turn the while into cfloop, etc. - David On 7/12/06, Matt Robertson <websitemaker@gmail.com> wrote:

On 7/12/06, David <david.lakein@gmail.com> wrote: > All the statements in that example have CF tag equivalents (CFScript syntax Yes I know :-)  I'm saying that I cannot get the conditional loop to work, and I cannot recall ever needing an isdefined in a non-cfscripted loop so I have nothing to fall back upon. If I use <cfloop    condition="isdefined("foo")">    ...do stuff... </cfloop> you get an explosion due to the doubled-quotes.  If you substitute single quotes inside of the isdefined that throws an error.  If you dispense with the quotes so the value in isdefined is a variable rather than a sring literal, and set the variable outside the loop, like so <cfset arfarf="foo"> <cfloop    condition="isdefined(arfarf)">    ...do stuff... </cfloop> I don't get any errors but I also don't seem to be getting an expected result. -- --m@Robertson-- Janitor, MSB Web Systems mysecretbase.com

Matt Robertson wrote: > > Yes I know :-)  I'm saying that I cannot get the conditional loop to > work, and I cannot recall ever needing an isdefined in a > non-cfscripted loop so I have nothing to fall back upon. <cfset srcFile = "C:/JRun4/bin/jvm.config"> <cfset fr = createObject("java","java.io.FileReader")> <cfset fr.init(srcFile)> <cfset br = createObject("java","java.io.BufferedReader")> <cfset br.init(fr)> <cfset str = br.readLine()> <cfloop condition='isDefined("str")'>   <cfoutput>#str#<br></cfoutput>   <cfset str = br.readLine()> </cfloop> <cfset br.close()> Jochem

On 7/12/06, Jochem van Dieten <jochemd@oli.tudelft.nl> wrote: > <cfloop condition='isDefined("str")'> I am such an idiot... single quotes on the OUTSIDE.  The one thing I didn't try.  Thanks Jochem! -- --m@Robertson-- Janitor, MSB Web Systems mysecretbase.com

Does this work for binary files?  I suppose it should, but the reading line by line bit makes me curious.