Thanks to a former colleague, I received word that my post about my first 100 days of working with Clojure scored Number 3. on the Hacker News.

I just wanted to thank all of you for the interest and I really hope my english is not that worse. Thanks to some attentive readers, I where able to fix some glitches in some of the examples or the spelling.

Due to fortunate circumstances I had some time to setup my blog on a new platform almost exactly a year ago. Now using Nginx with WordPress. Witch performed remarkably well.If your are interested here are some stats:

Starting with about 200 concurrent connections to the my Nging web server, which are 66 times the average over the last year. That’s nice because I saw some setups which would have died due to the load. Thanks to the guy who introduced me into Nginx. We were using it at scoyo and gamigo and this proved that it was a good decision – if I had any doubts left.

As a matter of fact, that worked so well, that it didn’t even had an impact on the CPU load:

Or even the memory usage:

I guess making all the content static using WP SuperCache and the HTTP rewrite module was a good idea.

In politics you have a one hundred days period of grace. In December I started to work with Clojure, so I guess it’s time to have a clojure look.

In the last couple of years I encountered a couple of different languages. At scoyo it was ActionScript and Flex from Adobe. I had the honor to lead a team of great engineers, but with good people you don’t get close enough to the real stuff. At gamigo the new technology was PHP. But for the last thirteen years Java was always a big part of my career…At my new job the new kid on the block is Clojure – a LISP dialect based on Java VM. Clojure was created by Rich Hickey and first appeared in 2007.

When we started to work on our new product I said to my boss and my colleagues: In Java I know how a state of the art application has to look like, but I have no clue how to build a Clojure application. Fortunately a colleague had some Common LISP experience and already a closer look on the stack we should use. At the moment our application uses this tools and libraries. I guess I will come back to some of them later on.

• Leiningen (aka lein) – The build and dependency managmement tool for Clojure.
• Noir – A nice library to build websites/-services in Clojure.
• CongoMongo and mongoDB – The database driver and the database we use.

My first days with Clojure
When you are coming from a language like Java you are used to some things like:
Structure everything – Static typing is your friend, so smack every data you have into a class. I guess most Java Enterprise application use a significant part of their computing time with copying references from one bean to another. When you use Clojure you simply don’t do this and sometimes is still hard for me not to worry about it. But this gives you a whole new level of freedom, you may use e.g. (merge …) to simply combine two of you entities into one larger one.

Lots of ceremony – When you work in Java you create a huge amount of classes. The longer you work with Java the more inner and anonymous classes you tend to use. From these classes ninety percent are stupid bean classes with some attributes plus a setter and a getter per attribute and if you work with experts a hashCode and equals method. The remaining ten (or so) percent of you classes are services or stuff like that, who transform beans in different beans or XML, JSON, String, stuff like that. In clojure you put your structured data in maps – that’s it. Due to the fact that all of closures datatypes are immutable you build some functions who do the same like the services and you are done. But instead of shuffling data from one bean to another you may use (assoc …), (dissoc …), (select-keys …) or (merge …) to efficiently create modified or new entities for the business logic or output generation.

But I can say, you’ll get used to this really fast and so I had programmed my first JSON web service very fast. But as always stuff get a little more complex …

The first hurdle I had to climb where collections. Unlike the good old for (…;…;…) {…} loop in Java the (for …) list comprehension in Clojure is a beast. The first thing to do, is to realize that despite the name these two guys are totally different. Staring with the fact that the Clojure (for …) will be evaluated lazily, so if you want a similar evaluation behavior that the Java version you should try (doseq …).
A standard pattern in Java is to define some nice result variables and then iterate with for through the collection changing your variables. After the loop your result is in the variable ready for further action. But you cannot do this in Clojure because you don’t have variables. You have let bindings which look somewhat similar but they are not mutable like your datatypes.
So the (for …) list comprehension works a little bit different. It takes a collection and returns a collection of same or shorter length. In the function body you can compute the new value of the new collection for returns. You even may skip values with the :when <val> option. A very similar function is the (map …) function which also maps a value to a new value and returns a lazy collection of these new values.

The first pitfalls
Don’t get me wrong, the Clojure collection library is great, but you need some time to get used to it. A nasty side effect was, that shortly after I though: “Yes, I am king of all collections!” you stumble over oddities like the different behavior in the (conj …) function. In Clojure you have these functions (cons …) which adds an element in front of a sequence and (conj …) which append an element at the end of a sequence. At least this is what I though. In fact it does append an element if the sequence is a vector, if it is a list it behaves like (cons …). You can spend some hours before you find this in the doc.

Another nice one – showing the power and danger – was related to Facebook and Mongo DB. We query some data from Facebook and store them into the a Mongo DB. The process only does some restructuring of the data but doesn’t touch the values before we insert them into the database. To optimize the Facebook querying we switched from a hundered calls of the Facebook graph API to three FQL queries. This worked really well. But Facebook has a minor difference in the results. In the graph API the IDs are strings in the FQL result set IDs are numbers. The whole process worked percieved well for a while but the I noticed that the queries on the Mongo DB delivered not enough results. What happened? The Clojure JSON library we used started to deliver numbers instead of strings and we where storing them into the Mongo DB. This worked without changing a line of code, which is nice if you want it. But the queries on the Mongo DB where still looking for strings …

Thinking more, coding less
I guess this says it all. The first weeks of Clojure where unsatisfying. When coded in Java I had lots of code: Classes, XML files, etc. But in Clojure the result of the day were sometimes two functions and a macro.
First that occurred very strange to me but I realized that I had to code less to get the same result. And these two functions and the macro where very lean and elegant and solved problems worth a couple of Java classes. This was the first time I really realized the power of Clojure and the different approach.

Homoiconic or code is data
If you start with Clojure you soon stumble about this whole code is data stuff. This is was homoiconic in the end means, that the syntax of your language can be written in data structures of your language. If you look at:

(for [i (range 1 100)] (println i))

You can read it as list with the first element for as symbol, a vector of further forms as second element and another list as third elements. The term for this is forms.

So doing this all LISPs have an extremely small syntax. At least that is what they are telling you. But I believe that the LISP/Clojure gurus (e.g. Gerald Jay Sussman or Rich Hickey) are cheating us a little bit.

I guess I have to agree that the languages it self has very little syntax, but the language is shifting the complexity into the libraries. If you have a look at the (for …) comprehension. You saw the simple variant a few lines ago but you can also du this:

(for [i (range 1 100) :let [f (format "%03d" i)] :when (odd? i)] (println f))

This nice little feller prints all odd numbers from 1 to 100 formatted with leading zeros. But instead of using (let …) or filtering the range collection I used some not so well documented options of the binding vector of the (for …). So how does it work? First for works internally different because it is a Clojure special form, but there are several macros – yes and I also wrote some – who do the same. If you are interested you may look at the (defpage …) macro, which does also some black magic.
You usually use macros to do this type of things but you also may use functions (to a certain degree). So what there macros is to analyze the vector on the first position of the parameter list. In the for case you would iterate through the list and if you encounter a keyword (the things with : like :let) instead of a symbol you do some extra special stuff.

Usually you use this to define your own domain specific language (DSL) in Clojure. This is what makes the solution of your problems very elegant because you make your problem related language where you can model and solve the problem very easily.

But also the Clojure standard functions are using this heavily and are so hinding a lot of implicit syntax form the user who depends on documentation and examples to discover how this works. For me this was sometimes very frustrating because very helpful options (e.g. :when <val>) are sometimes hard to find.

Java interoperability or it’s safe as long as you are in object land
Clojure lives (at least in the version we use) on the Java VM – there is a .NET variant and also with ClojureScript a variant which runs as Java-Script. So if you are on the JVM you want to use some the great libraries already exist in the Java ecosystem. If you e.g. look at clj-json a very fast JSON library for Clojure. It depends on the Jackson library and provides only a thin layer upon it.

To use Java objects and libraries in Clojure is very simple, the messy part only begins if you want to use primitive types. These are supported but its no fun anymore. A word of warning on working with Java objects. You are leaving the zone of immutability when you work with java objects. If you get yourself a java.util.Date you can mutate the date like you can do it in Java and mess up big time.

A very nice interop feature are (deftype …), (proxy …) or (reify …). You can use this macros to create or extend java Objects. Have a look at this:

(proxy [com.rabbitmq.client.DefaultConsumer] [(channel connection)]
       (handleDelivery [tag envelope prop body] (println body))
       )

This tiny piece of code extends the DefaultConsumer and overrides the handleDelivery method. You can hand over the received object to a channel as a consumer and integrate directly into the Rabbit MQ Java driver.

Some words to the ecosystem
Finally I want to have some word about some nice tools, library and stuff on Clojure ecosystem.

Leiningen – A very popular build tool which also offers dependency resolution. My experience with Leiningen is, that it works out of the box in most cases. Unfortunately it tends to hang once or twice a day, I guess it tries to figure out if it must update some dependencies when it’s gone.

Midje – A very powerful Clojure testing framework which includes facilities to easily mock every function. This makes writing even complex tests very easy.

avout – This one is nice. It integrates tools like Clojure atoms and refs into Zookeeper (or some other coordination engines). It also offers distributed locks. Very easy to use and to build distributed
cluster coordination.

pallet – A library who offers infrastructure management (e.g. for Amazon EC2) in clojure. You also may remotely install stuff or start and stop services. Great for dev-ops, but the documentation could be better and at the moment only Clojure 1.2 is supported. It took me some effort to get it working but when you know how to use it very powerful.

What suc**s?
In this post I told you various thing about the power and beauty of Clojure but where are some things that I don’t like:

No informations on data structures – I already told you that in Java you write a lot of classes to structure you data. On one hand side this is a lot of overhead, but on the other hand side its also a kind of always correct and available documentation. If you have to get into some foreign code that fetches data from a database it’s very hard to discover what is the right data structure and which type are the right ones. This also makes changes very difficult.

Dynamic invocation – Functions are compiled and invoked when whey are invoked. You don’t have any chance of previous checking if the function call is possible or not. This is extremely annoying when you change the signature of a function e.g. you remove a parameter. If you miss a spot where the function is called you get an error. This gets even worse if you use restructuring and variable parameter lists. You can do something like this:

(defn fetch [database & {:keys [:where]}] …)

You can call this function with or without the :where parameter. This looks like this:

(let [all (fetch my-database)
      some (fetch my-database :where {:height {:$gt 42}})]
	…
)

This is every powerful when you are creating flexible functions for DSLs but can cause very nasty side effects when you refactor these functions and change the signature.

No bootstrapping – In the Java world you have dependency injection frameworks like Google Guice or the Spring framework. These frameworks are very helpful when you want to invert the control flow. The services you write do not have to be aware how to find the suitable services and resources. You are centralizing the bootstrapping process in one location (e.g. classes in Guice or XML in Spring).
Clojure does not offer this facilities – you don’t have objects to store state, not even configuration state. The solution we choose is like a typical pattern. One namespace provides functions to access the resources (e.g. database connections, etc.) and there functions can be called from every where.
But this has also the same drawbacks than the classical service locator. You don’t have control about timing and the class is like a spider in the web called from every where.

Clojure 1.2 – The current stable release is Clojure 1.3. Unfortunately some libraries we wanted to use where still based on Clojure 1.2. In general this could work but be had some strange effects on some libraries which took us a while to discover that it was a Cojure 1.2/1.3 problem. A clear strategy how to deal with this isn’t there yet, so all you can do is try and error.

Loose ends
I didn’t talk about stuff like STM (software transactional memory), Agents, Refs or Atoms. Mostly because they worked in the first place for me, or never even tried them. There are also a couple of nice library in the Clojure eco system which make your life as developer very easy. I recommend, that you have closer look yourself.

… and in the end
To be honest, I had a tough couple of days in the last days. We are are in a stage where our prototype application grows into a real world application and we want to establish structures which makes maintenance easier. Unfortunately this is not so easy as we expected.

So my conclusions right now:

• Clojure is fun – I really like it to write two line functions that kick ass.
• Clojure does not need to hide – The eco system is growing fast and has some interesting libraries
• The lack of tooling is sometimes a pain. There are times where clojure.tools.logging/spy or println are not enough. The absence of a debugger is sometimes a real productivity black hole.
• Due to the dynamic nature you’ll have to write a lot of tests to be able to perform refactorings. Only with a lot of tests you can make sure that the code will work afterwards.
• Document your data structures to make sure that other developers have a chance to work in your code. I made some good experiences with a namespace that containes accessor functions. You can validate there functions with unit tests and the other guys in your team can easily see which values the can expect.

Nobody can say right now, how our little adventure with Clojure will end. But for the first release of out product our backend services will be based on Clojure. The time will tell how the maintainability and the performance of the backend will develop.

Right now I am not that confident that the solution will be good enough to be in service for a long time. But maybe Clojure surprises me … again …

It has been not even a full week building REST style interfaces for the Playmaker Studio mobile platform and I am very annoyed of all the curl guides how to access your REST API with curl. So here is my cheat sheet on this topic. Be aware that the I will update this frequently.

Common use cases

GETting data from a URL

This is a very simple use case just use:

curl https://graph.facebook.com/740999649
{"id":"740999649","name":"Niclas Meier"}

curl will automatically use the HTTP method GET.

POSTting updates

In the REST dictionary the HTTP method POST represents the verb ‘update’. So updating data implies a POST request to an URL.

Basic

Let’s have a look at the basics, I added some (revised / half the original) debugging output by supplying the -v option.

curl -v -X POST http://localhost:8080/1/stuff/23f96c6bd4c87910c.json -d 'name=Niclas'

> POST /1/stuff/23f96c6bd4c87910c.json HTTP/1.1
> Host: localhost:8080
> Accept: */*
> Content-Length: 11
> Content-Type: application/x-www-form-urlencoded
>
< HTTP/1.1 200 OK
< Content-Type: text/plain; charset=utf-8
< Content-Length: 25
<
Stuff: 23f96c6bd4c87910c

There are some options involved:

• -v added the request/response header for debugging
• -X specified the request method (POST) to use.
• -d added some data in the request (POST) body.

So the 'name=Niclas' was sent to the server using the request body, but curl does not display the request body using the -v option.

Using a JSON body

So this is nice, when you want to post key-value pairs to your server, but many applications (e.g. CouchDB want to post JSON objects to the server. The initial guess

curl -v -X POST http://localhost:8080/1/stuff/23f96c6bd4c87910c.json -d '{"name":"Niclas"}'

Stuff: 23f96c6bd4c87910c, JSON body:

does not work. That’s where the content type comes in. If you post the JSON whiteout supplying the proper content type curl and/or the server will try to use application/x-www-form-urlencoded^* content type. This is the same content type your browser uses when submitting a form to a web server. Doesn’t sound bad, does it?

But it’s bad because the JSON will be ignored/dropped/whatever and the server is unable to understand the request. To solve the problem use:

curl -H "Content-Type: application/json" -X POST  http://localhost:8080/1/stuff/23f96c6bd4c87910c.json -d '{"name":"Niclas"}'

Stuff: 23f96c6bd4c87910c, JSON body: {"name":"Niclas"}

So these options are involved

• -X specified the request method (POST) to use.
• -H added the header Content-Type: application/json to the request.
• -d defined the data used for the request body ({"name":"Niclas"}).

A JSON file as body

So far so good. But if you have huge JSON requests, you don’t want to type all the JSON into the console. You may use the @ symbol in conjunction with the -d option so curl will read the request body contents from a file. This may look like this.

curl -H "Content-Type: application/json" -X POST  http://localhost:8080/1/stuff/23f96c6bd4c87910c.json -d @some-huge-request.json

Stuff: 23f96c6bd4c87910c, JSON body: {"name":"Niclas", ..., "foo":"Bar"}

A nice trick is, that if you use the -d with @-. curl will read the request body from the standard input. So you can do stuff like this:

cat some-huge-request.json | curl -H "Content-Type: application/json" -X POST  http://localhost:8080/1/stuff/23f96c6bd4c87910c.json -d @-

Stuff: 23f96c6bd4c87910c, JSON body: {"name":"Niclas", ..., "foo":"Bar"}

PUTting stuff

Is much like POST, so if you can POST you can PUT. One nasty little difference is, that you don’t have to supply a Content-Type HTTP header for PUT. curl does not use application/x-www-form-urlencoded as content type for PUT. Lost an hour (or so), because I did some PUT stuff before switching to POST.

Debugging

Detailed information

The key to get almost every information by supplying the -v option to curl.

curl -v https://graph.facebook.com/740999649

* About to connect() to graph.facebook.com port 443 (#0)
*   Trying 66.220.147.38... connected
* Connected to graph.facebook.com (66.220.147.38) port 443 (#0)
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using RC4-MD5
* Server certificate:
* 	 subject: C=US; ST=California; L=Palo Alto; O=Facebook, Inc.; CN=*.facebook.com
* 	 start date: 2010-01-13 00:00:00 GMT
* 	 expire date: 2013-04-11 23:59:59 GMT
* 	 common name: *.facebook.com (matched)
* 	 issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert High Assurance CA-3
* 	 SSL certificate verify ok.
> GET /740999649 HTTP/1.1
> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
> Host: graph.facebook.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Access-Control-Allow-Origin: *
< Cache-Control: private, no-cache, no-store, must-revalidate
< Content-Type: text/javascript; charset=UTF-8
< ETag: "f89bfe40c87e487ae641d427c96746b226a91f3c"
< Expires: Sat, 01 Jan 2000 00:00:00 GMT
< Pragma: no-cache
< X-FB-Rev: 482216
< X-FB-Server: 10.36.50.102
< X-Cnection: close
< Date: Wed, 07 Dec 2011 13:14:19 GMT
< Content-Length: 183
<
* Connection #0 to host graph.facebook.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
{"id":"740999649","name":"Niclas Meier","first_name":"Niclas","last_name":"Meier","link":"http:\/\/www.facebook.com\/people\/Niclas-Meier\/740999649","gender":"male","locale":"de_DE"}

But for may occasions that is simply too much, so check out the -i option.

Header information

Often it’s enough to get only the response headers for debugging. To get only these headers just use the -i option in curl.

curl -i https://graph.facebook.com/740999649
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Cache-Control: private, no-cache, no-store, must-revalidate
Content-Type: text/javascript; charset=UTF-8
ETag: "f89bfe40c87e487ae641d427c96746b226a91f3c"
Expires: Sat, 01 Jan 2000 00:00:00 GMT
Pragma: no-cache
X-FB-Rev: 482216
X-FB-Server: 10.42.67.29
X-Cnection: close
Date: Wed, 07 Dec 2011 13:18:24 GMT
Content-Length: 183

{"id":"740999649","name":"Niclas Meier","first_name":"Niclas","last_name":"Meier","link":"http:\/\/www.facebook.com\/people\/Niclas-Meier\/740999649","gender":"male","locale":"de_DE"}

After been almost 10 years at SinnerSchrader it’s very confusing for me to be on the move again. On the 1^st of December I joined Playmaker Studio after being at gamigo for 18 month.You now may ask why? I don’t want to explain every thought that led to the decision, but I can say I enjoyed my job at gamigo. I learned damn much, especially stuff I never anticipated (I am not saying that I didn’t wanted to learn most of it ). I had a great team making a giant leap in the software development processes, nice colleagues and we brought the company a huge step ahead.

But it was more than about time to get a new direction in working life. At gamigo my job drifted to a pure management position and I was nudging around JIRA tickets and Excel files (okay some of them where Google Docs ) the whole day. But now I joined the Ranks of technicians again <cheer />!

Hopefully I will have some time for reviving this blog. For now I am trying to learn some new stuff – like Clojure, Noir, Heroku or mongoDBfor my new job. For me, this sounds like lots of fun!

After some inspiration the last few weeks, I decided to scrape together some spare time and have a closer look at the Scala programming language. This was kind of overdue because I am trying to have a look at new technologies at least once a year.

One major challenge is always finding an interesting topic which also has an appropriate level of difficulty. For my venture into Scala I choose a small open source project which I initiated two years ago called . This small framework validates an e-mail address and also checks the domain (via the DNS). This project seemed to have the right size and so I decided to build a scala variant called saev.

This turned out to be a very demanding venture which challenges a lot of knowledge and beliefs which I have from my activities in the Java environment.I would be lying if wouldn’t tell you, that I already head some doubts on the proceedings in the Java community. Things like having no update on the JDK for about two years now, including the unfortunate discussions on lambda expressions in Java, are paralyzing the development on the language basis. On the other hand side I have no idea how to judge the acquisition of Sun Microsystems by Oracle.

I first stumbled over Scala when I still was working at scoyo, when my former boss was building a small script which was converting localized text into an XML format. I found the language it self quite interesting but when you are in the middle of a Java team it was not easy to get started with it and had other things occupying me more.

Beginning this year I was willing to give Scala a new chance after having a quick look at PHP and Ruby. Both of PHP and Ruby didn’t convince me so I bought a Scala book and read it. But I must confess after this lecture I still was interested but nothing more. But after attending the Developer Conference Hamburg 2011 I decided to give Scala a real try. So I bought another book especially written by Prof. Dr. Friedrich Esser for guys who want to change form e.g. Java to Scala. I just want to take the take the time to recommend this book “Scala für Umsteiger“. Very well written book which shows you in the first chapter (approx. 100 pages) how to write the stuff you do in Java in the Scala language. An then fasten your seat belts

I am playing around with Scala for about a week right now, but I have to confess that I am somehow impressed and also a little bit scared. Let me explain why:
Fist of all the syntactical power of this “post functional” language is impressive for someone who did Java for over ten years. You can describe many of your problem solution in a very simple way. But when you want/need you can always fall back into Java because Scala uses the Java VM as foundation. On the other hand side I met a couple of programmers which are nice people, but who would be overstrained by this. And to be honest there are always situations where I take both of my books and google to figure out what some code is doing and why it’s working.
Don’t get me wrong I am pretty good programmer and I am using even complex concepts like AOP (aspect oriented programming) or dependency injection for a couple of years now. And I know the can cause lots of trouble and I am not sure if I should like @Transactional or not.

For my personal development I am quite sure, that I will continue to investigate Scala a little further. On the business side I am not sure if Scala will find a place in my development team. I said already ten years ago that you can use a bunch of trained apes to finish you Java project. It will take some time but the apes will figure it out. And that robustness is what you want on enterprise projects. Maybe for innovative companies or startups Scala is the language of choice or for companies which are willing to have polyglot development environment.

It’s sunday and I am fed up, sitting lazy behind my computer and taking a review of the last week. I attended two conferences this week so I am full of new ideas (don’t know if they are good yet) and wan’t to take a closer look on so many topics.

The first conference was the next conference in Berlin which was initiated some years a go by my former employer SinnerSchrader. So the focus of the next conference was mixture of brand, media, entrepreneurship and technology. Due to the fact that I meet a couple of former colleagues – some of them still working for SinnerSchrader.

The second conference was the Developer Conference Hamburg 2011 which had a sole technology focus.I heard couple of very good talks on both conferences. My highlight was the the keynote of Werner Vogels the CTO of Amazon which spoke about the analysis of big dataset which are regularly produced by modern IT infrastructures. If you generate a couple of gigabytes of data every day you should get the information out of them.

Another highlight of both conferences where the Couch DB talks. Both held by Jan Lehnhardt, one of the Couch DB core commiters who works for Couchbase. Jan can talk very fast in his talks, especially when he had some coffee . But there is no bull*** and he definitely knows what he is talking about. Because I attended the next and the devcon I was able to get a 10.000m overview and a 100m close look on the Couch DB. I am very fascinated by this topic, but I have at the moment no idea where to use it.

A dilemma what Couch DB shares with node.js. I heard of node on both conference too. On the next node.js was presented by Isaac Z. Schlueter. The topic was how node.js performs on DIRT (data intensive real time) applications. At devcon node.js was presented by Felix Geisendörfer which gave a more wide range overview on the capabilites of node.js. Unfortunately my current employer does at the moment some kind of “boring” e-commerce stuff, so no real use for node here. Some SinnerSchrader guys had a very innovative idea using node building the The World’s First Digital Foosball Table – nice one!

But the most influencing talks where from Prof. Dr. Friedrich Esser from the HAW Hamburg about Scala. The talk and the workshops where full of snappy remarks on Java programming language. I am doing Java for over ten years now. And I am confident that I am pretty good at Java and it was my favorite language the last ten years. But in the recent time I am feeling bored by Java and it’s very tight restrictions. We’ll see where the journey goes with this Scala stuff.

On the other hand side there were also some talks – which we presented brilliantly – that where about stuff that I am doing for a couple of year now. One of these was the “Abstract monitoring” talk from Dr. Johannes Mainusch from Xing. Very nice talk, I agree 100% but I was also doing this since 2005. Good to see that other developers are cooking with water too

But the last week reminded me of the gap what we have at the moment and what I am supposed to close as soon as possible. But now I am armed with some new ideas …

Some of you may already have noticed, that this blog uses WordPress as blog software. Nothing special about that I guess. As I mentioned before in January/February I moved my website from my old provide to a new one making some significant changes to the blog and it’s infrastructure.

I am in the web business since a couple of years and I already worked with a couple of servers. Like most of the others I worked most with the Apache web server. Which is still the work horse of the internet. But recently the nginx web server which is a lean, mean web serving machine. To get a little more experience with this product I decided to use it for my blog.The nginx web server already matured already 9 years and reached recently (12. April 2011) the version 1.0.0 (congratulations on this by the way ). But there are still some major differences between the Apache and the nginx web servers.For me this started when I wanted to use WordPress which is build upon PHP. PHP itself depends on the common gateway interface (a.k.a. CGI) which is a long time component of the Apache web server. Unfortunately the nginx server does not support CGI so you’ll have cover some more basics.

nginx itself supports CGI using the fast CGI standard. Which does actually a little bit more the plan CGI. To serve fast CGI you may use the nginx fast CGI module, so you still have access to all the scripting languages you like. But be aware that the nginx web server is known for it’s fabulous capabilities serving static content.

After installing nginx, a fast CGI wrapper app to execute the PHP scripts and configuring the fast CGI module, the setup of the WordPress blog was pretty basic. I’m using a standard MySQL database backing the WordPress. The whole system runs on a CentOS linux. The choice of the linux distribution was also inspired by the preferred system I use at my job for almost two years. Some times it’s good to stick to things you know

This was the status on which the whole system ran since February. But as a few – regarding to google analytics – of you may have noticed that the whole system was very slow. This is a direct result of my stinginess, because I don’t want to pay more money for my root server

So today I invested some spare time to look for other optimizations and I would like to say, that I was successful. My first search lead to a WordPress plugin to cache the pre-rendered WordPress pages. I have not this much posting and commenting traffic, so pre-rendering the pages seemed to be a good idea. The plugin of my choice is WP Super Cache which provides a very easy to configure caching of WordPress pages.
But I wanted to go one step further: The WP Super Cache plugin offer the possibility to bypass WordPress and PHP completely by using compressed pre-rendered pages via mod_rewrite. I guess most of you noticed the problem: mod_rewrite is an Apache module, which does not work with the nginx server, but nginx provides a powerful rewriting module too. A very helpful blog post from the WP Super Cache author helped to configure the web server. And now – at least for any anonymous user – the site is fast.

It do no longer take about two seconds to return the the first bytes to the user and the page itself is compressed only approximately five kilobytes large. To speed up the perceived loading time of the page, I am using the Use Google Libraries plugin pass the delivery of standard Java Script libraries to google. So the libraries can participate on sharing between different pages and the long time expiry which is used by google. The plugin WP Minify joins and compresses the JS and CSS used by the theme and other plugins I use, so the browser does download only a few minimized files instead of dozens of CSS and JS files.

I can say, that the experiment was quite successful and after getting familiar with some peculiarities the setup process was quite easy.

I just wanted to say, that I am back – at least a kind of

The last couple of month I was buried under a sh****** of work since I joined my new employer, but on the other hand side I a lot of fun and it is a very challenging job.

The last couple of weeks I used some spare time to move my infrastructure to a new home, including this blog. I will shut down my private blog soon, I guess my live is not exciting enough or may be I am not exhibitionistic enough to write over every breath I take. But now this blog is almost completely migrated and got a shiny new theme too – I hope you like it!

The good news are, that I want to take up my technophile blogging activities. I learned very much in the last couple of month about writing good software and what to do to prevent it.

The subject of my first post of the series “Contemporary Software Development” will be building. How to build your software project is a decision which is made quite early when you start developing.

Unfortunately in a lot of projects this very important topic does not get the appreciation it deserves. On one hand side it’s a very sad story of suffering and endurance. On the other hand side we are talking about a lot of wasted money.In my career as software engineer I experienced some different approaches to the project setup and the build process.

It started with something like find . -name '*.java' | xargs javac, which worked quite well for the first weeks of my professional career. But with a couple of software developer working on a project you want something more standardized.

So we went on with make(*). I guess most of the software engineers will know this build tool which still is a standard tool in the Unix/Linux world. make is a very powerful tools but has its peculiarities also. The make grammar is very delicate when it comes to white spaces. I quess quite a few people sat over a broken make build when a TAB char wasted your build script.

In the Java world Apache Ant was the successor to make in my career. The problem with make and the Java world is, that make is not really platform independent. There are make ports for Microsoft Windows but (at least when I made the jump from make to Ant) it does not work very well. So if you have a heterogeneous environment e.g. you develop on Microsoft Windows and you deploy into a Linux based cloud, you want your build to to be platform independent.

A nice story about platform independence. I did some consulting in a project for the Deutsche Bank Italy and a very large (and expensive) consulting company provided the BEA Weblogic build and setup. I am not 100% sure but I guess it was an Apache Ant based build. One major issue was the application server integration integration. To solve it, the complete project was placed and build in the application server which was committed in to the CVS source control management system. The delivery artifact was – of course – delivered without the application server. But this leads to another wish I have when I am using a build. You want the build to credit the current environment. If you are building for your local development environment you may have different requirements concerning the built artifact as when you are creating an artifact that will be deployed on staging or productive environment.

Back on memory lane: Apache Ant worked pretty well for some years and a couple of projects. But one thing that started to annoy me. Every project was using a slightly build setup so in the daily business there were always searching for artifacts or checking library versions.

Some former colleagues introduced me to Apache Maven. I started with Maven 1 which is a quite interesting mixture of Ant with a framework that offered the possibility to manipulate the Ant build file during the runtime. But Apache Maven had some features which I hat to get used to.
An example is the rigid directory structure defined in a Maven build: Why do I have to follow them? I want my special directory names! But nowadays I love this structure which is equal in every project. You find your files so fast now and even in foreign project. That’s another thing that you want: You want your builds to follow standards, so you can get familiar with the project in very short time. If you are working in the software business a couple of years, may have already realized that you will see very much projects during a professional career and standard projects and processes will make your life much more easy.

Let’s talk about some goodies now which modern build system offer. I will use Apache Maven as example but other systems like rake or buildr offer similar functions.

One thing I love/hate is dependency management. When I started with make or ant, we where committing all libraries into our CVS. Which worked quite well, in most cases you could check our the project, performed the initial build and started to work. But there where lots of minor but very nasty problems. Starting from which library version is really used to incompatibilities of the build tool version.

With the Apache Maven dependency resolution mechanism a new area begun. If you are having one independent project the mechanism is pretty easy to use. You define the libraries you want to use in your pom.xml and maven takes care of the rest. Okay if you want to use libraries which are not in the maven central repository it’s a little more complicated

If you are having a multi module/project environment things will get messy pretty easy, but: They would also if you don’t use the dependency management mechanism. You ask why? When you don’t use this mechanism you’ll normally have a huge monolithic project in your SCM system which will get even more difficult to manage or build. So you want to have a transparent dependency management in your build system. With Ivy for example you’ll have an add-on which offers the possibility to have an dependency management when you are using Ant.

Another goodie is release management. It’s very handy if you start a build to create your release artifact and your tag (or something similar) in the SCM system of your choice. You want your release process embedded in your build system, so you can perform the build when you are creating your release.

Another important topic is testing. You really want your build tool to support your testing facilities. At least when you create your release or deployment artifact you want your build to automatically run your test suite.

I have only one topic left in this post, that is IDE support. Unfortunately most modern IDE like Eclipe, Netbeans or Intellij IDEA offer a IDE specific build. This this very comfortable if you have a small project and what to deal only with your IDE. But you’ll grow really beyond this (my experience). But integration into other build tools is sometimes really poor. So you have to check how you can bridge the gap between your IDE and the build tool. E.g. the maven directory structure troubles some IDE
standard configurations. Fortunatly more and more projects provide solutions for this.

So let’s have a quick resume. This is what you should want in your build system:

• standardized process
• platform indipendence (if your need it)
• credit current environment
• standardized structures
• dependency management
• embeddable release process (including SCM support)
• support of your test tools
• IDE support

I think this list is not too lang and offers some major improvements for your daily work. So if your build process is missing some items from this list, you maybe should think about your build process. And remember: Your development resources are far to valuable to be wasted in an inefficient build process

It has been some time since my last blog post. But I promise to do better. I am planning a series ob blog post in which I am trying to explain how you should want to develop software nowadays – at least in my opinon.

I will try to cover topics like: How to build your system? How to manage you source code and you versions? Or what do do with your deployment data?

But I don’t want you to tell you which frameworks to use in fact I want to discuss what you should want to do not how you are going to do it.

I am excited, it will be a lot of work, but I am excited