CFLOCK issues in ColdFusion 8

Hi all, We recently upgraded our test server from CFMX 6.1 to ColdFusion 8, which causes our application much problems. Because of the nature of our application and the way it is build, we use cflockextensively. It seems that ColdFusion 8 has some serious issues with CFLOCK. After the upgrade, we found requests to be blocked for no obvious reason, while the CF monitor always shows  that the process is waiting for a lock (cflock is the final element in the stack trace). We are using nested locks, but there are no deadlocks (believe me, I checked). Anyways, the CF documentation says that a deadlock would end when a timeout is reached. I'm now looking at a request that has been blocked for over 3 days (lock timeout = 30 seconds). Here's the stack trace of the request: ************************************** /opt/bea/wls_sp6/user_projects/domains/domaineistc/extend7010/stage/cfusion-ear/cfusion-war/eistc/components/SharedDirectoryListing.cfc cflock at /opt/bea/wls_sp6/user_projects/domains/domaineistc/extend7010/stage/cfusion-ear/cfusion-war/eistc/components/SharedDirectoryListing.cfc getNextItem() : 138 at /opt/bea/wls_sp6/user_projects/domains/domaineistc/extend7010/stage/cfusion-ear/cfusion-war/eistc/components/SharedDirectoryListing.cfc check() : 116 at /opt/bea/wls_sp6/user_projects/domains/domaineistc/extend7010/stage/cfusion-ear/cfusion-war/eistc/tasks/archiving/setExpirydates.cfm cflock : 51 at /opt/bea/wls_sp6/user_projects/domains/domaineistc/extend7010/stage/cfusion-ear/cfusion-war/eistc/tasks/archiving/setExpirydates.cfm : 3 at /opt/bea/wls_sp6/user_projects/domains/domaineistc/extend7010/stage/cfusion-ear/cfusion-war/eistc/tasks/archiving/setExpirydates.cfm ************************************** This is the only active request. No other requests are running. When I start a new request that waits for the same lock, it waits for the lock for 30 seconds and then times out, which leads me to believe that the long running request has obtained the lock. This code worked without any issues on CFMX6, so there has to be something different in the way CF8 handles the locks. Is there anybody out there that had similar issues after upgrading to CF8? Is there a way to see all active locks on an application? The CF monitor does not seem to be able to give me the information I need to solve this issue. Any help would be much appreciated! Thank you, Wim Belgium

Hi all, We recently upgraded our test server from CFMX 6.1 to ColdFusion 8, which causes our application much problems. Because of the nature of our application and the way it is build, we use cflockextensively. It seems that ColdFusion 8 has some serious issues with CFLOCK. After the upgrade, we found requests to be blocked for no obvious reason, while the CF monitor always shows  that the process is waiting for a lock (cflock is the final element in the stack trace). We are using nested locks, but there are no deadlocks (believe me, I checked). Anyways, the CF documentation says that a deadlock would end when a timeout is reached. I'm now looking at a request that has been blocked for over 3 days (lock timeout = 30 seconds). Here's the stack trace of the request: ************************************** /opt/bea/wls_sp6/user_projects/domains/domaineistc/extend7010/stage/cfusion-ear/cfusion-war/eistc/components/SharedDirectoryListing.cfc cflock at /opt/bea/wls_sp6/user_projects/domains/domaineistc/extend7010/stage/cfusion-ear/cfusion-war/eistc/components/SharedDirectoryListing.cfc getNextItem() : 138 at /opt/bea/wls_sp6/user_projects/domains/domaineistc/extend7010/stage/cfusion-ear/cfusion-war/eistc/components/SharedDirectoryListing.cfc check() : 116 at /opt/bea/wls_sp6/user_projects/domains/domaineistc/extend7010/stage/cfusion-ear/cfusion-war/eistc/tasks/archiving/setExpirydates.cfm cflock : 51 at /opt/bea/wls_sp6/user_projects/domains/domaineistc/extend7010/stage/cfusion-ear/cfusion-war/eistc/tasks/archiving/setExpirydates.cfm : 3 at /opt/bea/wls_sp6/user_projects/domains/domaineistc/extend7010/stage/cfusion-ear/cfusion-war/eistc/tasks/archiving/setExpirydates.cfm ************************************** This is the only active request. No other requests are running. When I start a new request that waits for the same lock, it waits for the lock for 30 seconds and then times out, which leads me to believe that the long running request has obtained the lock. This code worked without any issues on CFMX6, so there has to be something different in the way CF8 handles the locks. Is there anybody out there that had similar issues after upgrading to CF8? Is there a way to see all active locks on an application? The CF monitor does not seem to be able to give me the information I need to solve this issue. Any help would be much appreciated! Thank you, Wim Belgium

Is it possible for you to take the thread dump and post it here?

> Because of the nature of our application and the way it is > build, we use cflockextensively. Why, exactly? In my experience, many applications overuse locks for various reasons, such as requirements imposed by CF5 and earlier versions. There shouldn't be that many things you need to lock from within CF (as opposed to your database, which will be where most of your concurrency issues occur and which you should solve by using whatever concurrency control your database provides). > We are using nested locks, but there are no deadlocks ... I've seen various problems with nested locks before. I strongly recommend that you replace them with a named lock scheme, or remove them entirely if you can. Dave Watts, CTO, Fig Leaf Software http://www.figleaf.com/ Fig Leaf Training: Adobe/Google/Paperthin Certified Partners http://training.figleaf.com/ WebManiacs 2008: the ultimate conference for CF/Flex/AIR developers! http://www.webmaniacsconference.com/

Though it is not directly related to the problem mentioned here, there is an interesting post by Simon on cflock http://www.horwith.com/index.cfm/2008/4/28/cflock-explained

Thank you all for your replies. The problem was caused (believe it or not) by the ColdFusion monitor in CF8, and more specifically the Memory Tracking. When I stopped the memory tracking, the issue does not occur anymore. I stopped it yesterday and the application is now running fine for over 24 hours. Whenever I activate the memory tracking again, we are having lock issues withing 3 minutes. When I stop it again, the problem goes away (after I kill the stuck threads). It seems that Adobe needs to look into the impact of the memory tracking some more. They mention that it will have a negative impact on the performance (which is totally understandable), but blocking an entire application from running is just plain ridiculous. Regards, Wim.

Hi Wim, I think there is a hotfix for the same issue that we have built. I will check on the same. Thanks, Hemant Adobe ColdFusion Team

Just a quick note, this bug is fixed in CF 8.0.1.

>Just a quick note, this bug is fixed in CF 8.0.1. Just a quick note, I AM running CF 8.0.1 Back to the drawing board ;o)

> The problem was caused (believe it or not) by the ColdFusion monitor in > CF8, and more specifically the Memory Tracking. This is not recommended to be on in production. Only the first one of the three is as the others have various levels of performance impact. You wouldn't leave debugging on on a production box either, for the same reason. -- Tom Chiverton **************************************************** This email is sent for and on behalf of Halliwells LLP. Halliwells LLP is a limited liability partnership registered in England and Wales under registered number OC307980 whose registered office address is at Halliwells LLP, 3 Hardman Square, Spinningfields, Manchester, M3 3EB.  A list of members is available for inspection at the registered office. Any reference to a partner in relation to Halliwells LLP means a member of Halliwells LLP.  Regulated by The Solicitors Regulation Authority. CONFIDENTIALITY This email is intended only for the use of the addressee named above and may be confidential or legally privileged.  If you are not the addressee you must not read it and must not use any information contained in nor copy it nor inform any person other than Halliwells LLP or the addressee of its existence or contents.   If you have received this email in error please delete it and notify Halliwells LLP IT Department on 0870 365 2500. For more information about Halliwells LLP visit www.halliwells.com.

@Tom I agree and we didn't activate this on production, but I don't think it's normal that this feature (which is built to help you solve issues) will cause this kind of issues in the first place. I can live with some performance decrease, but blocking the entire application makes this a "feature from hell" which should never be used, not even on a development or test system.

It may be a reason to re-examine what you're doing with so much locking going on in the first place. You didn't mention the reason for your use of so many CFLOCKS (which must be nested I assume based on what you're saying?) but as Dave alluded, there may be other ways to deal with whatever the underlying forces are. On Fri, May 16, 2008 at 8:57 AM, Wim Lemmens <didgiman@gmail.com> wrote: ----- Excess quoted text cut - see Original Post for more -----

@Brian It's clear that the locking is not the issue here, so I don't think there's much reason to elaborate on that. The problem is that a running process is blocked at a random point inside a lock (when the Memory Tracking is activated). This causes the lock to be active for ever (beyond the lock timeout), which prevents other requests from obtaining the lock. We are using named locks, so it doesn't really block the application, but just that particular process. However, this behavior is seen in all processes (for various named locks), so in practice, the whole application is blocked. Stopping the Memory Tracking solves the issue.

Ah OK, if this is just a simple named lock then that is definitely a common thing. It sounded like you had a large number of locks that might be nested, etc., which might be an indicator that some other approach could be considered. Regardless, it's true that the memory tracking should not be locking (ha ha) things up like this. I wasn't making excuses for the Server Monitor, but rather just trying to see if there was an underlying issue that might need attention regardless of this apparent bug in the Monitor. Regards, Brian On Fri, May 16, 2008 at 9:55 AM, Wim Lemmens <didgiman@gmail.com> wrote: ----- Excess quoted text cut - see Original Post for more -----

Hi Wim, Can you please mail me (hkhandel atadobedotcom) the java stacktrace and any repro code from CF server that will help me in trying to debug this issue and fix it? Thanks, Hemant