I am slowly getting through Domain Specific Languages by Martin Fowler. I really enjoy it as I expected I will. Apart of the main theme - building DSLs, their parsing and processing - it is full of little gems, making it much more useful than I would imagine before. But I would not expect anything less from Martin Fowler.
I have done a few internal DSLs before, always thinking external DSLs could be more useful for developers "stuck" in Java world, where internal DSLs are rather limited in way they can look like and when they can be processed (compilation). The big plus of this book is in that it shows ANTLR can be useful and is not very complicated if you don't want to build Turing complete language. A point of this exercise is to check how simple is to use it. I was not able to make up a nice example, but this book provided a help also here. I really liked the example showing internal DSL for access control list. It reminded me something I was working years ago, but Martin solved it in much better way. I don't mean a usage of DSL for configuring ACL, but his semantic model and a way it processes security checks - by usage of Specification pattern. He provides several implementations (Java, Ruby) and I've decided to base the format of my external DSL on his internal DSL in Ruby to serve as example in Java. I don't want to repeat his work here, so I will show only text I want to parse, the grammar and how to load it. I will skip creation of semantic model, because it would be ripped of the book directly and I don't want to do that, not to mention it would not be useful for my experiment. I don't want to hone my copy&pasting skills here :-) Instead I will only log information parsed from the file.What I want to parse
As I said he uses following configuration in example, I've just removed ':' forming symbols. The point is to list rules allowing or disallowing access to some resource based on when a person does it and to which department they belong.
allow {
department mf
ends 2008, 10, 18
}
refuse department finance
refuse department audit
allow {
gradeAtLeast director
during 1100, 1500
ends 2008, 5, 1
}
refuse {
department k9
gradeAtLeast director
}
allow department k9
Initial grammar that parses the configuration
grammar Acl;
acl : (allows | refuses)+;
allows : 'allow' what;
refuses : 'refuse' what;
what : condition | '{' condition+ '}';
condition : department | ends | grade_at_least | during;
department : 'department' ID;
ends : 'ends' INT ',' INT ',' INT;
grade_at_least : 'gradeAtLeast' ID;
during : 'during' INT ',' INT;
ID : ('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'0'..'9'|'_')*;
INT : '0'..'9'+;
WS : (' ' | '\t' | '\r' | '\n' ) + {skip();};
ILLEGAL : .;
Token rules ID and INT were generated by ANTLWorks. Token rules WS and ILLEGAL come from the book. ILLEGAL is about parser stopping when no other rule matches. Also the usage of EOF in the top most rule was described in the book. Rest of it is my own creativity. I think it is not very complicated or hard to read for now. It will get more complicated shortly when I will add some clutter.
Initial loader
I would like to write a piece of code that will invoke generated parser and reads the file. Again, this is taken directly from the book, there is not much point to do it differently.
public class AclLoader {
private Reader input;
private AclParser parser;
public AclLoader(Reader input) {
this.input = input;
}
public void load() {
try {
AclLexer lexer = new AclLexer(new ANTLRReaderStream(input));
parser = new AclParser(new CommonTokenStream(lexer));
parser.acl();
} catch (Exception e) {
throw new RuntimeException(e);
}
}
}
AclLexer and AclParser were generated by ANTLR from my grammar. It should parse the file, but it does nothing with it yet.
Let's add some tests.
public class AclLoaderTest {
@Test
public void readsValidFile() throws IOException {
Reader input = new FileReader("resources/acl.txt");
AclLoader loader = new AclLoader(input);
loader.load();
}
@Test(expected = RuntimeException.class)
public void failsForInvalidFile() throws IOException {
Reader input = new FileReader("resources/acl-bad.txt");
AclLoader loader = new AclLoader(input);
loader.load();
}
}
I am bit confused with the 2nd one - it fails, because invalid file is not refused. Hmm, I've read something about errors, in the book. Let's review it...
Error handling
I forgot to add reporting of errors to my parser. After seeing advantages in the book I am going to add superclass for parser (so that I can add custom code to parser without being afraid ANTLR will delete it next time I will generate the parser. It will be useful later when processing output from parser too. This is taken directly from the book, but simplified for my needs.
public abstract class BaseAclParser extends Parser {
private List errors = new ArrayList();
public BaseAclParser(TokenStream input, RecognizerSharedState state) {
super(input, state);
}
public void reportError(RecognitionException e) {
errors.add(e);
}
public boolean hasErrors() {
return !errors.isEmpty();
}
public String getErrorReport() {
StringBuffer result = new StringBuffer();
for (Object e : errors) result.append(e).append("\n");
return result.toString();
}
}
I have to tell ANTLR to use it. I am going to modify beginning of my grammar file:
grammar Acl;
options {superClass = BaseAclParser;}
...
Finally to check for errors at the end of parsing in AclLoader (again taken from the book).
if (parser.hasErrors()) throw new RuntimeException("Loading failed: "+parser.getErrorReport());
Now both tests are passing, but I do not like error message I am getting. I don't want to spend time with cryptic error messages now, so I will ignore it.
Experiments with processing content
My parser seems to be reading correct configuration file, but it ignores everything it learns there. It is time to process that information. Now it will get a bit messy, but I hope I will make it (really, I have not done it myself yet - let's go back to the book to reread particular chapter so I can pretend I am smart again). ANTLR allows to invoke custom code and I am going to try simple thing to see if it works as expected. I modify the grammar like follows:
allows : 'allow' {allowBlock();} what;
That should invoke method allowBlock in BaseAclParser when 'allow' is parsed, but before parsing of conditions starts. Let's add simple implementation of allowBlock:
protected void allowBlock() {
System.out.println("allowBlock");
}
As I said in the beginning I am not going to build semantic model, just use logging instead. If I wanted to do something more serious I would set Context Variable here, or alternatively I could propagate complete information about parsed text from inner rules here, but that would get pretty messy.
I do similar thing for 'refuse' block:
refuses : 'refuse' {refuseBlock();} what;
BaseAclParser:
protected void refuseBlock() {
System.out.println("refuseBlock");
}
Processing conditions
Analogicaly to previous code I modified the grammar to inform parser about parsed content. Any time a rule about condition is successfuly parsed it invokes custom method inBaseAclParser that processes it.
grammar Acl;
options {superClass = BaseAclParser;}
acl : (allows | refuses)+ EOF;
allows : 'allow' {allowBlock();} what;
refuses : 'refuse' {refuseBlock();} what;
what : condition | '{' condition+ '}';
condition : department | ends | grade_at_least | during;
department : 'department' depName=ID
{departmentCondition($depName.text);};
ends : 'ends' year=INT ',' month=INT ',' day=INT
{endCondition($year.text, $month.text, $day.text);};
grade_at_least : 'gradeAtLeast' grade=ID
{gradeAtLeastCondition($grade.text);};
during : 'during' after=INT ',' before=INT
{duringCondition($after.text, $before.text);};
ID : ('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'0'..'9'|'_')*;
INT : '0'..'9'+;
WS : (' ' | '\t' | '\r' | '\n' ) + {skip();};
ILLEGAL : .;
Of course, processing in my case means just logging it:
protected void departmentCondition(String name) {
System.out.println("\tdepartmentCondition: " + name);
}
protected void endCondition(String year, String month, String day) {
System.out.println("\tendCondition: "+year + "-" + month + "-" + day);
}
protected void gradeAtLeastCondition(String grade) {
System.out.println("\tgradeAtLeastCondition: " + grade);
}
protected void duringCondition(String after, String before) {
System.out.println("\tduringCondition: <" + after + ", " + before + ">");
}
What I have got
And as the result I am getting following output:allowBlock departmentCondition: mf endCondition: 2008-10-18 refuseBlock departmentCondition: finance refuseBlock departmentCondition: audit allowBlock gradeAtLeastCondition: director duringCondition: <1100, 1500> endCondition: 2008-5-1 refuseBlock departmentCondition: k9 gradeAtLeastCondition: director allowBlock departmentCondition: k9 allowBlock refuseBlock departmentCondition: k9It looks suspiciously similar to original file :-) but I am happy with outcome. I am going to play with it more, to add building of semantic model and to add more unit tests. It should be easy, because the book contains all code for semantic model, I need just adapt my hook methods. Unfortunately I don't think I can publish it here completely. It is coming from still unpublished book...
I am in the middle or reading Domain Specific Languages by Martin Fowler and I have started to experiment with ANTLR today. After copy&pasting of example and fixing a few problems I wanted to generate a parser from grammar. I wanted to use following Ant snippet taken from Ant documentation (Rough Cuts of the book does not show how to do that):
<target name="generate" depends="prepare">
<antlr target="${dir.antlr.src}/Greetings.g" outputdirectory="${dir.gen}" />
</target>
But regardless of my experiments ANTLR process invoked from Ant always ended with Unable to determine generated class. Googling did not help. I found many questions, but no solution. Finally I settled on a following workaround, based on standalone generation of sources from the grammar:
<target name="generate" depends="prepare">
<java jar="lib/antlr-3.2.jar" fork="true">
<arg value="-o"/>
<arg value="${dir.gen}"/>
<arg value="${dir.antlr.src}/Greetings.g"/>
</java>
</target>
This does the trick. Now back to the example to see what it does.
Long awaited book Domain Specific Languages By Martin Fowler have finally got to Rough Cuts stage at Safari Books Online. I've started to read it immediately. I've just finished narratives part and now I am getting into reference material (I am one of those who read it cover to cover even it is not expected). So far I have enjoyed it and my expectations are still high.
I have done some small DSLs in the past, mostly for using in unit and system tests and I enjoy using fluent interfaces of mocking libraries. I can say I have done also some code generation on project for previous employer. But all of them were quite informal. That's the reason I like this book - it gives me an overall view, it helps to transform my former experiences into coherent bulk of knowledge. It should help me to understand WHY, and WHEN so that I can spot possible applications easier. HOW is also useful, it can save a lot of time when exploring possible solutions. Unfortunately I think majority of programmers will not try to read this book, because its topic is too far from their interest and experiences. But maybe the book will help to spread awareness of DSL usefulness. I guess there are not many programmers these days who are not using some kind of DSL, be it SQL, CSS, mocking library, ...Today I heard interesting thing. Our deadline for committing to plan for next year was moved to later, some time in the middle of June instead of being as planned. "I hope it will help us to get better estimates" we were told (this was my interpretation, the original was probably a bit different).
I was quiet as I would not be able to change that decision in any way. Still I wonder how giving us 2 or 3 weeks more will improve our ability to estimate 12 following months. Why management feels better to guess what 20 people can do in 12 months than estimating only 1-2 months, performing planned tasks and then checking where we got. But that is not the reason for this post. It reminded me important information I have found in Implementing Lean Software Development: From Concept to Cash some time ago.Goldratt believes that the key constraint of projects - he considers the product development to be a project - is created when estimates are regarded as commitments. [...] Since the estimate will be regarded as a commitment, the estimator accommodates by including a large amount of a "safety" in case things go wrong. However, even if things go well, the estimated time will be used up anyway, since estimators don't want to look like they over-estimated.and bit later in the same chapter:
In fact, if half of the activities don't take longer than their estimated time, the system will not achieve the desired improvement.This is that important bit of information - when all tasks are achieved in estimated time we cannot be sure we did not waste time. But when some of them (ideally 50%) take longer than estimated time then one of the possible reasons could be that our estimates are very close to minimal time needed, thus waste is minimized.
I "forced" my brother to buy Programming in Scala in December and I got a chance to quickly read a few chapters. It is really interesting language, strange syntax is well compensated with strong features it has. So I have decided to buy one too.
I have started to read it from scratch recently, doing examples and experimenting as readers are supposed to. While doing my experiments I needed to use scaladoc documentation. When I have found method I was looking for, it was a bit shock for me - How am I supposed to read that? And I know what 'transform' should do." Let's see an example fromMap:
def transform[C](f : (A, B) => C) : Map[A, C]So I did a few experiments until interpreter accepted my input, then a few more and I know how to read it now :-)
var x = Map(1->"A", 2->"B")
x.transform((k,v) => v*2)
//-> Map(1 -> AA, 2 -> BB)
x.transform((k,v) => v.isEmpty)
//-> Map(1 -> false, 2 -> false)
So, now I understand that method transform takes function with parameters A and B (meaning key and value pair) and transforming it to C. Result C is taken and as result of transform new Map A->C is returned. A is not modified, so it is not specified in transform[C] I guess.
I can see there will be a few more surprises along the way, but if solving all of them feels as good as cracking this little problem, then I will have enjoyable experience in learning Scala.
Note: I decided to give Scala a try after my discussions about it with Peter Misak.
I found following sentences regarding quality in Art of Agile Development while commuting to work this morning.
The best way I know to reduce the cost of writing software is to improve the internal quality of its code and design. I've never seen high quality on a well-managed project fail to repay its investment. It always reduces the cost of development in the short term as well as in the long term. On a successful XP project, there's an amazing feeling - the feeling of being absolutely safe to change absolutely anything without worry.I wish more people were aware of this and acted accordingly. I have to make sure this happens at least at my present and future jobs.
Continuing reading The Art of Agile Development I have found another gem for me. (Well, there were more, what I mean here something I can relate to my past experience).
Project I had been working a year ago in my former company went very well. We were using Scrum (or what we believed is very good approximation of Scrum in our environment) and I was proud that our team is delivering a lot features with rather stable velocity. But then reading this book I have found following:On the other hand, if your velocity is rock solid, try reducing slack by committing to a small extra story next iteration.Of course, it makes a lot of sense. How could we know that we are using our time effectively if we did not try to finish a bit more? If we tried to commit to a bit more work we could either finish it all or our velocity would stay approximately at the same level. Small fluctuations of velocity is not big problem, but having stable velocity is probably suspicious.
I am currently reading The Art of Agile Development that I've bought at Javapolis last year, but I did not get to it sooner.
I am close to middle of it and I am sure it is the best book I have read about agile development, because it says what, why, how, when not and alternatives (if they exist). What inspired this post was reading chapter No Bugs. I could not stop thinking about it and I have shown it to all colleagues in my team. It contains something that should cause envy among the most of programmers.However, XP teams can achieve dramatically lower bug rates. [Van Schooenderwoert]'s team averaged one and a half bugs per month in a very difficult domain. In an independent analysis of a company practicing a variant of XP, QSM Associates reported an average reduction from 2,270 defects to 381 defects [Mah].and later on
For example, [Van Schooenderwoert] delivered 21 bugs to customers. Working from Capers Jones' data, Van Schooenderwoert says that a "best in class" team building their software would have generated 460 defects, found 95 percent of them, and delivered 23 to their customer. In contrast, Van Schooenderwoert's team generated 51 defects, found 59 percent of them, and delivered 21 to their customer. At 0.22 defects per function point, this result far exceeds Capers Jones' best-in-class expectation of two defects per function point.It is very hard to imagine something like that even I think projects I've been working for last few years were better than average regarding bug count (I have no data to prove that to you or even me, it is just feeling).
Last entry inspired me to think about books I really liked, they taught me a lot and I wish everybody around me have read too :-) Here it is, it is not complete and their order is not meant as ranking.
- Martin Fowler: Refactoring Improving the Design of Existing Code
- Michael Feathers: Working Effectively with Legacy Code
- Kent Beck: Test Driven Development: By Example
- Dave Astels: Test-Driven Development: A Practical Guide
- Eric Evans: Domain-Driven Design
- Martin Fowler: Patterns of Enterprise Application Architecture
- Gerard Meszaros: xUnit Test Patterns: Refactoring Test Code
- Erich Gamma, Richard Helm, Ralph Johnson, John M. Vlissides: Design Patterns: Elements of Reusable Object-Oriented Software
- Alan Shalloway: Design Patterns Explained: A New Perspective on Object-Oriented Design
- Andrew Hunt, David Thomas: The Pragmatic Programmer: From Journeyman to Master
- Tom DeMarco, Timothy Lister Peopleware: Productive Projects and Teams
- Johanna Rothman, Esther Derby: Behind Closed Doors: Secrets of Great Management
- James Shore, Shane Warden: The Art of Agile Developement
- Robert C. Martin: Clean Code: A Handbook of Agile Software Craftsmanship
- (many): The ThoughtWorks Anthology
RSS feed