Bootstrapping EC2 servers with Puppet

At HEROLABS we are doing sports / gaming apps and one inherent characteristic is, that you have load peaks around the games. I guess having load peaks is characteristic to for most gaming apps. If we take a look at your EMHERO2012 app we had most server traffic during the time the German national Football team played and almost no traffic in between.

What we wanted to do: Scale up our server capacity around periods of increased load and scale down the rest of the time. To achieve this, we chose the Amazon (AWS) cloud as deployment environment and an automated server deployment process. To shed some light on the result already, after some a hard work and steep learning curve, it works ;-) Continue reading

Eventsourcing and CQRS with Clojure and MongoDB

It was about a year ago when I stumbled about the concepts of event sourcing and command query responsibility segregation in a talk from Greg Young at QCon 2011 in London which I watched on InfoQ. This challenged some things I knew for sure regarding how to store and manage data.

Fortunately I joined HEROLABS in December 2011 and due to the fact that we where starting a brand new product, I was able to play around with these concepts. Additionally we chose MongoDB instead of a relational database in our first project as our main database management system. MongoDB is attributed to the NoSQL databases and is in the first place a big document (JSON/BSON) store. Additionally MongoDB offers facilities to index and search the collection of documents.

In conjunction with Clojure your are able to build a simple a quite powerful system to use event sourcing and CQRS. Continue reading

Underestimated Amazon S3

The Amazon Simple Storage Service (S3) is the big data work horse of Amazon and one of the corner stones of their cloud infrastructure. For our recent project we received a considerable amount of game data from our data provider. At the moment these file are stored in the GridFS of our Mongo DB installation, but once the data is processed we don’t really need these files any more, except for development or archiving purposes.

At the moment we are using the Amazon Elastic Compute Cloud (EC2) to run our infrastructure, so it was a logical choice to evaluate the S3 to store the received game data and I have to confess I was really underestimating this service as simple file system. Continue reading

From Promises to Futures

This is the second post on an alternate implementation/approach to Futures and Promises on the Clojure programming language. In my first post I talked a little bit about strength and weaknesses of the the actual implementation of Futures and Promises in Clojure and I defined how my Future and Clojure implementation should work.

At the end of the first post I described how to implement a Promise in Clojure. A Promise in the context of my implementation should be a write handle to e.g. pass later on computed values to other processes.

In this blog post I want to:

  • show you how to get from the Promise implementation to a simple Future realization
  • talk about different execution strategies and why it’s good do have influence on the future execution

So let’s get started … Continue reading

Alternate Futures and Promises for Clojure

At the moment I am experimenting with various techniques to speed up some update queries from our client. One important element of this efforts is making things parallel. E.g. to compute a list update of deltas for the client we need to compare the version of the user on the client and in the database and we need the user data at some other places on the delta computation. So it would be a good idea to initiate the loading of the user data asynchronous right at the beginning of the request processing. While the user data is loaded we can load/compute some other stuff (e.g. we need some data about the competition and the matches). When all the required information is present continue the rest of the the job (e.g. generate some XML or JSON).

Futures implementations like java.util.concurrent.Future, scala.concurrent.Future or clojure.core/future seem to be the weapon of choice for this.

Due to the fact that our backend is written in Clojure I took a closer look at the Clojure Future implementation clojure.core/future. The Clojure implementation is pretty basic and very similar the the underlying Java implementation. This isn’t bad, but rather unsatisfying for a language emphasizing on concurrency like Clojure. Continue reading

Suddenly somewhat famous

Thanks to a former colleague, I received word that my post about my first 100 days of working with Clojure scored Number 3. on the Hacker News.

I just wanted to thank all of you for the interest and I really hope my english is not that worse. Thanks to some attentive readers, I where able to fix some glitches in some of the examples or the spelling.

Due to fortunate circumstances I had some time to setup my blog on a new platform almost exactly a year ago. Now using Nginx with WordPress. Witch performed remarkably well. Continue reading

First hundred days of Clojure

In politics you have a one hundred days period of grace. In December I started to work with Clojure, so I guess it’s time to have a clojure look.

In the last couple of years I encountered a couple of different languages. At scoyo it was ActionScript and Flex from Adobe. I had the honor to lead a team of great engineers, but with good people you don’t get close enough to the real stuff. At gamigo the new technology was PHP. But for the last thirteen years Java was always a big part of my career… Continue reading

Playing around with Scala

After some inspiration the last few weeks, I decided to scrape together some spare time and have a closer look at the Scala programming language. This was kind of overdue because I am trying to have a look at new technologies at least once a year.

One major challenge is always finding an interesting topic which also has an appropriate level of difficulty. For my venture into Scala I choose a small open source project which I initiated two years ago called . This small framework validates an e-mail address and also checks the domain (via the DNS). This project seemed to have the right size and so I decided to build a scala variant called saev.

This turned out to be a very demanding venture which challenges a lot of knowledge and beliefs which I have from my activities in the Java environment. Continue reading