I found following sentences regarding quality in Art of Agile Development while commuting to work this morning.

The best way I know to reduce the cost of writing software is to improve the internal quality of its code and design. I've never seen high quality on a well-managed project fail to repay its investment. It always reduces the cost of development in the short term as well as in the long term. On a successful XP project, there's an amazing feeling - the feeling of being absolutely safe to change absolutely anything without worry.

I wish more people were aware of this and acted accordingly. I have to make sure this happens at least at my present and future jobs.

I had interesting mail exchange with my colleague Gilles today. It was about multithreading in Java and how hard is for people to buy into all this.

I would say that the biggest mistake in Java is that it made multithreading to look easy while it is not. Everybody can spawn new threads easily with copy&paste (in the worst case scenario ;-) ). But consequences of incorrect thread-safeness and ignored memory visibility are usually terrible. And it is even worse since introduction of multi core CPUs.

It was about 4 years ago when I was looking for bug in JBoss that caused that an instance of Entity Bean was not saved every 4-6 hours. It took me a week to track it down to incorrect synchronization - two places, one synchronized at instance other at class level. On other side it was rather relaxing - wait till it fails, go through logs, formulate new hypothesis, modify logging in sources, start and wait till another error occurs. Fine for me, but I am not sure about my employer :-) Unfortunately before I could send a patch it was already fixed, so no glory for me :-( ;-) . This example should show how hard it is to fix such a problem and that it happens also to experts occasionally.

Now back to knowledge about Java multithreading. Recently I published my list of influential books. It did not contain anything about particular technology, but I should add following: Brian Goetz: Java Concurrency in Practice. Everybody programming in Java professionally should know its content by heart. There is no better book about this topic.

I believe it is only fair to downgrade to Junior all Senior Java developers not being able to explain how they use multithreading correctly.

I was struggling with running our unit tests in Idea for some time. Problem was that we have many modules and some of our tests using files try to find them through working directory instead of classpath. So every time I needed to execute test/suite from different module I needed to change working directory of test to point to its own module.

Finally I was able to find great solution (I really do not understand why I did not try it sooner :-) ). Just open Run/Debug dialog, click to Edit Defaults and write $MODULE_DIR$ into Working directory: field.

Continuing reading The Art of Agile Development I have found another gem for me. (Well, there were more, what I mean here something I can relate to my past experience).

Project I had been working a year ago in my former company went very well. We were using Scrum (or what we believed is very good approximation of Scrum in our environment) and I was proud that our team is delivering a lot features with rather stable velocity.

But then reading this book I have found following:

On the other hand, if your velocity is rock solid, try reducing slack by committing to a small extra story next iteration.

Of course, it makes a lot of sense. How could we know that we are using our time effectively if we did not try to finish a bit more? If we tried to commit to a bit more work we could either finish it all or our velocity would stay approximately at the same level. Small fluctuations of velocity is not big problem, but having stable velocity is probably suspicious.

I am currently reading The Art of Agile Development that I've bought at Javapolis last year, but I did not get to it sooner.

I am close to middle of it and I am sure it is the best book I have read about agile development, because it says what, why, how, when not and alternatives (if they exist).

What inspired this post was reading chapter No Bugs. I could not stop thinking about it and I have shown it to all colleagues in my team. It contains something that should cause envy among the most of programmers.

However, XP teams can achieve dramatically lower bug rates. [Van Schooenderwoert]'s team averaged one and a half bugs per month in a very difficult domain. In an independent analysis of a company practicing a variant of XP, QSM Associates reported an average reduction from 2,270 defects to 381 defects [Mah].

and later on

For example, [Van Schooenderwoert] delivered 21 bugs to customers. Working from Capers Jones' data, Van Schooenderwoert says that a "best in class" team building their software would have generated 460 defects, found 95 percent of them, and delivered 23 to their customer. In contrast, Van Schooenderwoert's team generated 51 defects, found 59 percent of them, and delivered 21 to their customer. At 0.22 defects per function point, this result far exceeds Capers Jones' best-in-class expectation of two defects per function point.

It is very hard to imagine something like that even I think projects I've been working for last few years were better than average regarding bug count (I have no data to prove that to you or even me, it is just feeling).

I am big fan of Pixar. Recently I've read article written by its president Ed Catmull.

Harvard Business Review article

I like whole article, but there were a few sentences that I consider really interesting:

If you give a good idea to a mediocre team, they will screw it up; if you give a mediocre idea to a great team, they will either fix it or throw it away and come up with something that works.

it's OK to hire people who are smarter than you are

There were more, but I wanted to avoid quoting the same ideas that were quoted in blog post where I found the article.

Last entry inspired me to think about books I really liked, they taught me a lot and I wish everybody around me have read too :-) Here it is, it is not complete and their order is not meant as ranking.

• Martin Fowler: Refactoring Improving the Design of Existing Code
• Michael Feathers: Working Effectively with Legacy Code
• Kent Beck: Test Driven Development: By Example
• Dave Astels: Test-Driven Development: A Practical Guide
• Eric Evans: Domain-Driven Design
• Martin Fowler: Patterns of Enterprise Application Architecture
• Gerard Meszaros: xUnit Test Patterns: Refactoring Test Code
• Erich Gamma, Richard Helm, Ralph Johnson, John M. Vlissides: Design Patterns: Elements of Reusable Object-Oriented Software
• Alan Shalloway: Design Patterns Explained: A New Perspective on Object-Oriented Design
• Andrew Hunt, David Thomas: The Pragmatic Programmer: From Journeyman to Master
• Tom DeMarco, Timothy Lister Peopleware: Productive Projects and Teams
• Johanna Rothman, Esther Derby: Behind Closed Doors: Secrets of Great Management

I am sure I have missed some important or that I do not remember that some influenced me as much as these did. Only thing I am sure is that this list should be longer.

Following list contains books still in queue (either ready on shelf or still at Amazon :-) ). I have strong feeling they are worth of reading:

• James Shore, Shane Warden: The Art of Agile Developement
• Robert C. Martin: Clean Code: A Handbook of Agile Software Craftsmanship
• (many): The ThoughtWorks Anthology

About 2 years ago I was handing my project over to my successor. He was younger, less experienced so apart of showing him details of project I was also giving him good advice.

I was talking about unit testing, TDD, something about design and that led me to refactoring. At the end of the session I gave him Refactoring book thinking how much I helped him.

To my big surprise he came to me next day handing book back with words "I can click in Eclipse". I was shocked, not able to react. My first thought was probably something that I misjudged him. I realized whole point only later, but then I felt it is too late to return to that for him already closed matter.

The real meaning of those words was "I don't know and I don't know I don't know.". Really - he thought that book talks only about things Eclipse can do easily. But that was huge mistake - it talks about more important things like what are code smells, how to detect them, how to achieve better design with small steps, what to do if simple refactorings fail or they don't lead to nicer result and so on.

I believe this book taught me how to work a bit differently - be less concerned about immediate appeal of code I am currently writing, because I can change it easily when bigger picture is available. That I can Extract Method later when I know what it does and how to name it correctly. I do not worry about every aspect, because I know I can change it easily. And I know what my possibilities are, so I am able to spot opportunity when it occurs. I would say that spotting opportunity when it occurs is the most important part. If all code looks satisfying/passable to me, there is no possibility for improvement. I believe it was the first book that taught me something about aesthetics in code.

Looking back to this "incident" I think I missed good opportunity to help him, but it is quite possible his ego did not allowed me to, so maybe my reaction was OK. Since then I try to be more persuative in similar situations. So if you stumble on me, trying to help you, be tolerant and patient, please :-)

Project I am currently working on has rather sophisticated build system. I believe that huge project as this one is deserves it. It has many advantages and a few disadvantages. Problem is that it is optimized for Eclipse as all developers are using it.

Unfortunatelly, some features like:

• dependencies between modules done through jar files
• regeneration of Eclipse's project files by Ant (needed every time you update jar files)
• dependenent jars copied to target/
• fact that Eclipse does not distinguish between production and test sources

made my life with Idea miserable in the beginning. It used lot of memory, was slow and needed refresh of internal data often (very slow in Idea) and remote debugger was not able to switch to different module. I gave a chance to Eclipse (3 weeks) before returning to Idea with goal to overcome all these problems (plus those happening in Eclipse too like impossibility to refactor between projects) by custom scripts.

I started slowly, then I added more features. Because my goal was to fix problems in Eclipse/Idea integration, fix disadvantages in dependencies as well as to fix management of project I rejected an idea to create plugin as it would need learning huge Plugin SDK and it could not help me to solve all problems. I decided to go through command line scripts (not to mention I never liked to do project management related tasks through IDE). My scripts are done in Ruby.

I am able to clean all projects in directory by subsequent calling of 'ant clean' in all subdirectories:

def clean
    in_dirs_with 'build.xml' do
        system 'ant clean'
    end
end

I was really nicely surprised seeing how nice and readable those simple tasks could be. I added a few more:

def update
    in_dirs_with '.svn' do
        system 'svn up'
    end
end

def eclipse
    in_dirs_with '.classpath' do
        system 'ant eclipse'
    end
end

All magic is hidden in following method:

	def in_dirs_with(guard)
		Dir['*'].each do |dir|
			if File.exist?("#{dir}/#{guard}")
				puts "Processing #{dir}..."		
				Dir.chdir(dir) do
					raise Exception.new($?) unless yield
				end
			end
		end
	end

OK, less annoying things are handled, now I can get to fixing problems with Idea. I am not going to show any code for that, it is bigger, but I will describe what I have done:

• script finds all generated .classpath and replaces jar dependencies with known sources (fixed refactoring and debugging problems)
• script finds all .eml files (Idea keeps here information what .classpath belongs to module and what is missing in .classpath but Idea needs it). Script modifies it by stating which directories are tests and which directories to exclude from project.

With these two scripts my life is wonderful again (as it was before I joined this huge project with build system optimized for Eclipse). Idea responds immediately and I can refactor as no of my colleagues can :-) (well, they could use the one wiring modules together, because it modifies only .classpath, but they would need to install Ruby first).

Apart from that I have created small DSL that helps me to perform merges (merging multiple modules in Idea is rather annoying), but I needed it only twice. Example how merging file might look like follows:

Merger.new.
	from_revision(12305).
	from_branch('http://server/svn/branches/mybranch').
	to_branch('http://server/svn/branches/head').

	add_new('new_module').
	add_new('other_new_module').

	merge('changed_module1').
	merge('changed_module2')
end

If I tried to do it in Idea (similarly in Eclipse) I would need to state revisions and branches for every module that needs to be merged. Small script helped enormously with 10 modules I was merging last time. (Note: add_new is for adding module that was created in mybranch so it cannot be merged directly to head, at least I am not aware of directly working solution).

Doing these things is called yak shaving and considered problematic, but I have to say, my yak is shaved very well now and I guess time invested has been paid off for some time already.