CSS

Monday, May 14, 2012

Death to a Deadlock

One of the systems I have been working on has has had a deadlock that happens intermittently for the last 6 years.  Well today I finally found it.  We have a thread that runs in our application that times how long a web request takes.  If the request takes more then 30 minuets it assumes a deadlock and kills the application.  The thread also send an email to the programmers.  Below is the code I inserted in the monitoring thread code that allowed me to find the deadlock.



public static StringBuffer getDeadTraces(StringBuffer errorStringBuff) {
        StringBuffer deadtraces = new StringBuffer();
	try {
	    long[] ids = ManagementFactory.getThreadMXBean().findDeadlockedThreads();
	    if(null != ids && ids.length > 0) {
	         Map map = Thread.getAllStackTraces();
		 Set set = map.entrySet();
		 for (long id : ids) {
		    for (Entry entry : set) {
			 Thread thread = entry.getKey();
			if (thread.getId() == id) {
				deadtraces.append("\n===================  ");
				deadtraces.append(thread.getName());
				StringBufferUtils.append(deadtraces, "  ===================","\n");
							
				for (StackTraceElement el : entry.getValue()) {
					StringBufferUtils.append(deadtraces, el.toString(),"\n");                    						
				}												
			}
		    }				 
	         }
	    }
	} catch (Exception e) {
		errorStringBuff.append("Error trying to create a snapshot: " + 
		e.getMessage() + org.apache.commons.lang.exception.ExceptionUtils.getFullStackTrace(e));
	}
	return deadtraces;
}


No comments:

Post a Comment