Jekyll2016-12-11T22:11:02+00:00www.niclas-meier.de//byte bohemianTechnophile blogging - sending out my message to information technology related topics
Niclas MeierWelcome to Jekyll!2016-12-10T09:06:50+00:002016-12-10T09:06:50+00:00www.niclas-meier.de/private/2016/12/10/welcome-to-jekyll<p>Look like the Jekyll default post, but it’s a little more I guess. After more than three years I found some time update me digitial infrastructure to a more serverless way.</p>
<p>Two weeks ago Amazon had their <a href="https://reinvent.awsevents.com">re:Invent</a> conference in Las Vegas and I was watching the keynote and some sessions with quite some interest. After launching/anncouning <a href="https://aws.amazon.com/de/lambda/">AWS Lambda</a> in 2014 serverless computing is getting more and more a thing and this years re:Invent made it more and more clear to me. So more about time to get rid of my old vServer and to move forward to a serverless solution …</p>
<p>Okay, let’s be honest: My email account has long moved to a public email provider and is no longer hosted on my own server. As a matter of fact it never was, the sole purpose of Postfix was to provide <a href="https://en.wikipedia.org/wiki/Greylisting">email greylisting</a> for improved SPAM protection. It worked quite well for a while, but my email provider can handle it very well own his own. The secondary reason to build an email facade was, that it was fun and I wanted to try it …</p>
<p>Leaving me and my <a href="https://wordpress.org">Wordpress</a> blog on the vServer. I already looked at some offers like <a href="https://www.blogger.com/">Blogger</a> or <a href="https://www.wordpress.com">Wordpress.com</a>, but I wanted my domain to come along. But I also didn’t want to migrate a MySQL database, web server and Wordpress to some different provider.</p>
<p>Then, last week I tried <a href="https://jekyllrb.com">Jekyll</a> and was convinced. No more need for a database and stuff, just generate the pages. I am using Wikis for quite some year now and <a href="http://github.com">github.com</a>, I already have some stuff there, so free hosting!</p>
<p>The most complex part was migrating the data out of the Wordpress MySQL, seems I misplaced the root password for the instance running the Wordpress :-), but after migration the dump on my local compute the <a href="https://import.jekyllrb.com/docs/wordpress/">jekyll-import</a> work reasonably well and getting all the ugly HTML to <a href="https://en.wikipedia.org/wiki/Markdown">Markdown</a> was an easy job.</p>
<p>So, let’s see how this works out, and … “Welcome to Jekyll!”</p>Niclas MeierLook like the Jekyll default post, but it’s a little more I guess. After more than three years I found some time update me digitial infrastructure to a more serverless way.Bootstrapping EC2 servers with Puppet2013-02-10T00:00:00+00:002013-02-10T00:00:00+00:00www.niclas-meier.de/technology/clojure/amazon%20ec2/amazon%20aws/puppet/2013/02/10/bootstrapping-ec2-servers-with-puppet<p>At HEROLABS we are doing sports / gaming apps and one inherent characteristic is, that you have load peaks around the games. I guess having load peaks is characteristic to for most gaming apps. If we take a look at your EMHERO2012 app we had most server traffic during the time the German national Football team played and almost no traffic in between. What we wanted to do: Scale up our server capacity around periods of increased load and scale down the rest of the time. To achieve this, we chose the Amazon (AWS) cloud as deployment environment and an automated server deployment process. To shed some light on the result already, after some a hard work and steep learning curve, it works ;-)</p>
<p>Okay here is how we did it: The first thing you need is a (metaphorically) unlimited amount of servers to rent, which should be available in a very short time (minutes). This is the job of the EC2 cloud in our scenario. That is the easy part, if you have a credit card. Now you have rented an almost blank server, except of the operating system. We chose Ubuntu (12.04 LTS - Precise Pangolin) as server operating system which is quite well documented and maintained for use in the EC2 cloud. The hard parts are now:</p>
<ul>
<li>automate the installation process of the software you need</li>
<li>add the right configuration files to your server</li>
<li>add the system to your cluster</li>
</ul>
<p>At the moment there are several products/projects for deployment automation / configuration management. The most known are <a href="http://www.opscode.com/chef">Chef</a> and [Puppet]:<a href="https://puppetlabs.com/what-is-puppet">https://puppetlabs.com/what-is-puppet</a> from <a href="https://puppetlabs.com/">Puppetlabs</a>. Due to the fact that we are using <a href="http://clojure.org">Clojure</a> we first started with <a href="http://palletops.com">Pallet</a>, but we changed to Puppet this year and it started to work out well for us.</p>
<p>In general you can divide these systems in two groups. One group is like a SSH shell on steroids: You script what needs to be done on the server and the tools does it in parallel on all the hosts you want. The other group defines the state you want the servers in and the the transition steps you need to do. And every node tries to to get into the state you defined. Chef and Pallet are more fitting into the first group and Puppet into the second.</p>
<p>The first step for me was to define the state and transitions for our backend server installation. To run our backend server we need Java, Leiningen and our Software package (cloned from <a href="https://github.com">GitHub</a>).</p>
<p>A nice about thing on Puppet is, that you don’t need any special programming skills. You define classes in so called manifest files which define the dependencies and actions you want to take. If you understand how Puppet works you get really fast into it. E.g. if you want Java installed you simply say:</p>
<figure class="highlight"><pre><code class="language-puppet" data-lang="puppet"> <span class="n">package</span> <span class="p">{</span><span class="s1">'openjdk-7-jdk'</span><span class="p">:</span> <span class="py">ensure</span> <span class="p">=></span> <span class="n">present</span><span class="p">,</span> <span class="p">}</span> </code></pre></figure>
<p>and Puppet will install the JDK on the next run.</p>
<p>Maybe a quick word how Puppet works. Puppet has two major part: A Puppetmaster, which is a central server with all state information, and a Puppet Agent which runs on each node. The Puppet Agent runs in a fixed interval and compares the state from the Puppetmaster with the local state. If these state are not identical it will try to adjust the local state to match the required state.</p>
<p>To create the Manifests was not very hard. We had some special issues with some SSH keys, GitHub and the local non-privileged user. But this was more about my lack of knowledge in these things. But I learned a lot (e.g. to write <a href="http://upstart.ubuntu.com">Upstart</a> scripts) about creating a server environment. I guess it’s always hard when you know what you should do, but you don’t known how to get there. But let me assure you that I took me only 2-3 days to create the automatic setup of our Clojure backend server (including setup of a Puppetmaster, etc.), starting by near zero.</p>
<p>One major obstacle was how Puppet defines the state of a server. Normally you find a manifest file on the Puppetmaster (e.g. <code class="highlighter-rouge">site.pp</code>) which defines the classes a server should contain. This looks like:</p>
<figure class="highlight"><pre><code class="language-puppet" data-lang="puppet"><span class="c"># /etc/puppetlabs/puppet/manifests/site.pp
</span>
<span class="k">node</span> <span class="s1">'www1.example.com'</span> <span class="p">{</span> <span class="k">include</span> <span class="nc">common</span> <span class="k">include</span> <span class="nc">apache</span> <span class="k">include</span> <span class="nc">squid</span>
<span class="p">}</span> </code></pre></figure>
<p>Nice for a small website, not so good in a cloud, when you don’t really know the names of your server. But Puppet has a great mechanism to solve our problems. It’s called <a href="http://docs.puppetlabs.com/guides/external_nodes.html">External Node Classifiers (ENC)</a>. How does this work? Instead of using the manifest data Puppet called an external program which returns the node classification as YAML. Great and simple solution.</p>
<p>After some googling I stumbled across a nice <a href="http://engineering.gowalla.com/2011/10/21/puppet/">blog post</a> by <a href="https://github.com/lordkev">Kevin Lord</a> on the Gowalla engineering blog, which helped a lot. So I decided to write my own ENC which you can find <a href="https://github.com/niclasmeier/puppet-ec2-enc">here</a>. It’s basically the same solution, but my version takes more than one security group into account and also merges the EC2 tags into the <code class="highlighter-rouge">parameter</code> section of the result. Which comes handy e.g. when you want to activate special services only on one server.</p>
<p>So we are now almost done on the automatic installation part. So when we start Puppet on a new host the Puppetmaster will figure out this is a productive backend server and install all the software we need on the on the server. Unfortunately there a no standard AMIs with Puppet pre-installed and I sure don’t want any standard AMI with our Puppetmaster address too. A first idea was to create an AMI with Puppet and our Puppetmaster pre-installed, but for an initial version we wanted to be able to use standard AMI.</p>
<p>The solution was a so-called <a href="http://alestic.com/2009/06/ec2-user-data-scripts"><code class="highlighter-rouge">user-data</code> script</a>. This is a mechanism that works on the EC2 Ubuntu images. The <code class="highlighter-rouge">user-data</code> script is a script which executed on instance startup and can be passed as a parameter on the Amazon EC2 API call.</p>
<p>With a suitable <code class="highlighter-rouge">user-data</code> script starting new instances looks like this:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"> ec2-run-instances -k my-ec2-key -n 1 --user-data-file <span class="o">[</span>install-puppet.sh]<span class="o">(</span>https://github.com/niclasmeier/puppet-ec2-enc/blob/master/install-puppet.sh<span class="o">)</span> -g backend-productive -t m1.small ami-960f03e2 </code></pre></figure>
<p>Our script does nothing more than adding the Puppetlabs <code class="highlighter-rouge">apt</code> repository, installs the newest version of Puppet and writes a very basic <code class="highlighter-rouge">puppet.conf</code> file. Fortunately Puppet is also managed by Puppet, so the <code class="highlighter-rouge">puppet.conf</code> will be adjusted later. The <code class="highlighter-rouge">-g backend-productive</code> selects the security group of the server and determines what to install.</p>
<p>As a matter of fact we don’t use the command line to start servers. Our admin tool as a button to start servers and uses the Amazon EC2 API directly to spawn new instances if needed.</p>
<p>Having a server installed is great, but worth nothing if you don’t get any traffic on the system. For our new product we use the Amazon <a href="http://aws.amazon.com/en/elasticloadbalancing/">elastic load balancer</a> (ELB). This makes the task somewhat easy.</p>
<p>The solution I chose was to create a <a href="https://github.com/niclasmeier/puppet-ec2-enc/blob/master/register_at_elb.rb">small script</a> that registers the instance to the ELB (using the ELB web service interface).</p>
<p>This script is part of the installation manifest for the backend server and a simple <code class="highlighter-rouge">exec</code>:</p>
<figure class="highlight"><pre><code class="language-puppet" data-lang="puppet"> <span class="n">exec</span> <span class="p">{</span><span class="s1">'register_at_api_elb'</span><span class="p">:</span> <span class="py">command</span> <span class="p">=></span> <span class="s2">"</span><span class="nv">${api::params::scripts_dir}</span><span class="s2">/register_at_elb.rb </span><span class="nv">${ec2_instance_id}</span><span class="s2"> </span><span class="nv">${loadbalancer_name}</span><span class="s2"> && touch /var/run/register_at_api_elb.lock"</span><span class="p">,</span> <span class="py">creates</span> <span class="p">=></span> <span class="s1">'/var/run/register_at_api_elb.lock'</span><span class="p">,</span> <span class="py">logoutput</span> <span class="p">=></span> <span class="n">on_failure</span><span class="p">,</span> <span class="kp">require</span> <span class="p">=></span> <span class="nc">Class</span><span class="p">[</span><span class="s1">'api::install'</span><span class="p">],</span> <span class="p">}</span> </code></pre></figure>
<p>To prevent the multiple registration the script is succeeded by a touch on a lock file. The <code class="highlighter-rouge">exec</code> will only performed by Puppet only if the lock file does not exists.</p>
<p>Some closing words: I love DevOps. Programming on your infrastructure is nice and opens a lot of opportunities. In my former jobs I worked with some great SysAdmins. This on hone hand helps very much, but can be a burden sometimes. Especially when you don’t known enough about the tools you want/have to use.</p>
<p>To give you a better perspective. All our servers use Puppet right now, but some of them (e.g. the databases) are not able to autoscale (yet ;-) ).</p>Niclas MeierAt HEROLABS we are doing sports / gaming apps and one inherent characteristic is, that you have load peaks around the games. I guess having load peaks is characteristic to for most gaming apps. If we take a look at your EMHERO2012 app we had most server traffic during the time the German national Football team played and almost no traffic in between. What we wanted to do: Scale up our server capacity around periods of increased load and scale down the rest of the time. To achieve this, we chose the Amazon (AWS) cloud as deployment environment and an automated server deployment process. To shed some light on the result already, after some a hard work and steep learning curve, it works ;-)Eventsourcing and CQRS with Clojure and MongoDB2013-01-26T00:00:00+00:002013-01-26T00:00:00+00:00www.niclas-meier.de/clojure/mongodb/2013/01/26/eventsourcing-and-cqrs-with-clojure-and-mongodb<p>It was about a year ago when I stumbled about the concepts of <a href="http://martinfowler.com/eaaDev/EventSourcing.html">event sourcing</a> and <a href="http://martinfowler.com/bliki/CQRS.html">command query responsibility segregation</a> in a talk from <a href="http://codebetter.com/gregyoung/">Greg Young</a> at QCon 2011 in London which I watched on <a href="http://www.infoq.com/presentations/Events-Are-Not-Just-for-Notifications">InfoQ</a>. This challenged some things I knew for sure regarding how to store and manage data.</p>
<p><a href="http://www.infoq.com/presentations/Events-Are-Not-Just-for-Notifications"></a></p>
<p><a href="http://www.infoq.com/presentations/Events-Are-Not-Just-for-Notifications">Fortunately I joined</a> <a href="http://herolabs.com">HEROLABS</a> in December 2011 and due to the fact that we where starting a brand new product, I was able to play around with these concepts. Additionally we chose <a href="http://www.mongodb.org/">MongoDB</a> instead of a relational database in our <a href="http://itunes.apple.com/de/app/em-hero-2012/id530513702?l=de&ls=1&mt=8">first project</a> as our main database management system. <a href="http://www.mongodb.org/">MongoDB</a> is attributed to the NoSQL databases and is in the first place a big document (<a href="http://json.org">JSON</a>/<a href="http://bsonspec.org">BSON</a>) store. Additionally MongoDB offers facilities to index and search the collection of documents.</p>
<p><a href="http://bsonspec.org"></a></p>
<p><a href="http://bsonspec.org">In conjunction with</a> <a href="http://clojure.org">Clojure</a> your are able to build a simple a quite powerful system to use event sourcing and CQRS.<a href=""></a><a href=""></a></p>
<h1 id="creating-events">Creating events</h1>
<p>Okay, lets start with the events: How to model events in Clojure? Basically there are two good choices, the first one is the Clojure standard namely maps. The second choice are records with protocols (e.g. <code class="highlighter-rouge">defrecord</code> and`defprotocol) which are a little more turbocharged. So the protocol for our event may look like this:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="nf">defprotocol</span><span class="w"> </span><span class="n">EventProtocol</span><span class="w">
</span><span class="p">(</span><span class="nf">id</span><span class="w"> </span><span class="p">[</span><span class="n">this</span><span class="p">])</span><span class="w">
</span><span class="p">(</span><span class="nf">type</span><span class="w"> </span><span class="p">[</span><span class="n">this</span><span class="p">])</span><span class="w">
</span><span class="p">(</span><span class="nf">created</span><span class="w"> </span><span class="p">[</span><span class="n">this</span><span class="p">])</span><span class="w">
</span><span class="p">(</span><span class="nf">aggregate-id</span><span class="w"> </span><span class="p">[</span><span class="n">this</span><span class="p">])</span><span class="w">
</span><span class="p">(</span><span class="nf">aggregate-version</span><span class="w"> </span><span class="p">[</span><span class="n">this</span><span class="p">]))</span></code></pre></figure>
<p>We have:</p>
<ul>
<li>an ID to identify the event.</li>
<li>an event type (e.g. <code class="highlighter-rouge">:user-created</code>).</li>`</li>
<li><code class="highlighter-rouge">a timestamp when the event was created/occured. Notice that the event does not have a modification timestamp, because we do not modify events.</li></code></li>
<li><code class="highlighter-rouge">an ID of the aggregate where the events is related to.</li></code></li>
<li>`a version identifier which tells us which aggregate version/modification this event represents.</li>
</ul>
<p>Lets have a quick view on the theoretical part. What is the goal? We want to represent the state the application or of an (aggregate) entity to represented as a stream of events, that is was <a href="http://martinfowler.com/eaaDev/EventSourcing.html">event sourcing</a> is all about. The work on event sourcing and CQRS is often closely tied to the <a href="http://en.wikipedia.org/wiki/Domain-Driven_Design">domain driven design (DDD)</a> ideas, so you often find references to <a href="http://en.wikipedia.org/wiki/Domain-Driven_Design">aggregates</a> or [aggregate roots. Aggregates are one of the building blocks of DDD and are similar to entities.</p>
<p>So if you take a look at the defined protocol you will notice, that we have a minimal set of properties defined for a event. The relation to the aggregate is defined by the <code class="highlighter-rouge">aggregate-id</code> and we are able to order the events by the`aggregate-version.</p>
<p>To bring the events in Clojure to life we define a record like this:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">[(</span><span class="nf">defrecord</span><span class="w"> </span><span class="n">Event</span><span class="w"> </span><span class="p">[</span><span class="n">_id</span><span class="w"> </span><span class="n">_aid</span><span class="w"> </span><span class="n">_t</span><span class="w"> </span><span class="n">_c</span><span class="w"> </span><span class="n">_v</span><span class="p">]</span><span class="w">
</span><span class="n">EventProtocol</span><span class="w">
</span><span class="p">(</span><span class="nf">id</span><span class="w"> </span><span class="p">[</span><span class="n">_</span><span class="p">]</span><span class="w"> </span><span class="n">_id</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="nf">type</span><span class="w"> </span><span class="p">[</span><span class="n">_</span><span class="p">]</span><span class="w"> </span><span class="n">_t</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="nf">created</span><span class="w"> </span><span class="p">[</span><span class="n">_</span><span class="p">]</span><span class="w"> </span><span class="n">_c</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="nf">aggregate-id</span><span class="w"> </span><span class="p">[</span><span class="n">_</span><span class="p">]</span><span class="w"> </span><span class="n">_aid</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="nf">aggregate-version</span><span class="w"> </span><span class="p">[</span><span class="n">_</span><span class="p">]</span><span class="w"> </span><span class="n">_v</span><span class="p">))</span></code></pre></figure>
<p>The first question may be: Why these ugly names (like <code class="highlighter-rouge">_id</code>,<code class="highlighter-rouge">_aid</code>, <code class="highlighter-rouge">_t</code>,<code class="highlighter-rouge">_c</code> or <code class="highlighter-rouge">_v</code>)? I already mentioned MongoDB and this relates to my target implementation on MongoDB (I will explain the idea behind this a little later) and to be honest: This source code relates to my blog entry and I am not planning to create a generic event sourcing library for Clojure, so I decided to make my life a little easier here.`````</p>
<p>To create new events we may define a function like this:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">[(</span><span class="k">defn</span><span class="w"> </span><span class="n">new-event</span><span class="w">
</span><span class="p">([</span><span class="n">type</span><span class="w"> </span><span class="n">aggregate-id</span><span class="p">]</span><span class="w"> </span><span class="p">(</span><span class="nf">new-event</span><span class="w"> </span><span class="n">type</span><span class="w"> </span><span class="n">aggregate-id</span><span class="w"> </span><span class="n">nil</span><span class="p">))</span><span class="w">
</span><span class="p">([</span><span class="n">type</span><span class="w"> </span><span class="n">aggregate-id</span><span class="w"> </span><span class="n">payload</span><span class="p">]</span><span class="w">
</span><span class="p">(</span><span class="nf">map->Event</span><span class="w"> </span><span class="p">(</span><span class="nb">assoc</span><span class="w"> </span><span class="n">payload</span><span class="w">
</span><span class="no">:_aid</span><span class="w"> </span><span class="n">aggregate-id</span><span class="w">
</span><span class="no">:_t</span><span class="w"> </span><span class="n">type</span><span class="w">
</span><span class="no">:_c</span><span class="w"> </span><span class="p">(</span><span class="nf">now</span><span class="p">)))))</span></code></pre></figure>
<p>To create a new event like the creation of a new user we may call:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">[(</span><span class="nf">clj-eventsourcing.event/new-event</span><span class="w"> </span><span class="no">:user-created</span><span class="w"> </span><span class="s">"5050480d3004e17b14ecb3fe"</span><span class="w"> </span><span class="p">{</span><span class="no">:user-name</span><span class="w"> </span><span class="s">"niclas"</span><span class="p">})</span></code></pre></figure>
<p>and receive this record:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="o">#</span><span class="n">clj_eventsourcing.event.Event</span><span class="p">{</span><span class="no">:_id</span><span class="w"> </span><span class="n">nil,</span><span class="w"> </span><span class="no">:_aid</span><span class="w"> </span><span class="s">"5050480d3004e17b14ecb3fe"</span><span class="n">,</span><span class="w"> </span><span class="no">:_t</span><span class="w"> </span><span class="no">:user-created,</span><span class="w"> </span><span class="no">:_c</span><span class="w"> </span><span class="o">#</span><span class="n"><datetime</span><span class="w"> </span><span class="mi">2012-09-12</span><span class="n">T08</span><span class="no">:34:36.331Z>,</span><span class="w"> </span><span class="no">:_v</span><span class="w"> </span><span class="n">nil,</span><span class="w"> </span><span class="no">:user-name</span><span class="w"> </span><span class="s">"niclas"</span><span class="p">}</span><span class="w">
</span><span class="n"></datetime></span></code></pre></figure>
<p>You see that the payload <code class="highlighter-rouge"><span class="p">{</span><span class="err">:user-name</span><span class="w"> </span><span class="nt">"niclas"</span><span class="err">}</span></code> is attached to the record even when the field was not defined in the`defrecord but if you check:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">[(</span><span class="nb">instance?</span><span class="w"> </span><span class="n">Event</span><span class="w"> </span><span class="p">(</span><span class="nf">clj-eventsourcing.event/new-event</span><span class="w"> </span><span class="no">:user-created</span><span class="w"> </span><span class="s">"5050480d3004e17b14ecb3fe"</span><span class="w"> </span><span class="p">{</span><span class="no">:user-name</span><span class="w"> </span><span class="s">"niclas"</span><span class="p">}))</span></code></pre></figure>
<p>As you can see, this evaluates to <code class="highlighter-rouge">true</code>, so the instance returned is still anEvent.</p>
<h1 id="storing-events">Storing events</h1>
<p>Creating events is nice, but if the events stay ephemeral they are of no use to your application. To store events we may define this preliminary protocol.</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">[(</span><span class="nf">defprotocol</span><span class="w"> </span><span class="n">EventStore</span><span class="w">
</span><span class="p">(</span><span class="nf">version-from</span><span class="w"> </span><span class="p">[</span><span class="n">this</span><span class="w"> </span><span class="n">aggregate-id</span><span class="p">])</span><span class="w">
</span><span class="p">(</span><span class="nf">store-events-into</span><span class="w"> </span><span class="p">[</span><span class="n">this</span><span class="w"> </span><span class="n">version</span><span class="w"> </span><span class="n">events</span><span class="p">])</span><span class="w">
</span><span class="p">(</span><span class="nf">load-events-from</span><span class="w"> </span><span class="p">[</span><span class="n">this</span><span class="w"> </span><span class="n">aggregate-id</span><span class="p">]))</span></code></pre></figure>
<p>and a very simplistic implementor for MongoDB</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">[(</span><span class="nf">deftype</span><span class="w"> </span><span class="n">MongoStore</span><span class="w"> </span><span class="p">[</span><span class="n">collection</span><span class="p">]</span><span class="w">
</span><span class="n">clj_eventsourcing.store.EventStore</span><span class="w">
</span><span class="p">(</span><span class="nf">version-from</span><span class="w"> </span><span class="p">[</span><span class="n">_</span><span class="w"> </span><span class="n">aggregate-id</span><span class="p">]</span><span class="w"> </span><span class="p">(</span><span class="nf">version</span><span class="w"> </span><span class="n">collection</span><span class="w"> </span><span class="n">aggregate-id</span><span class="p">))</span><span class="w">
</span><span class="p">(</span><span class="nf">store-events-into</span><span class="w"> </span><span class="p">[</span><span class="n">_</span><span class="w"> </span><span class="n">version</span><span class="w"> </span><span class="n">events</span><span class="p">]</span><span class="w"> </span><span class="p">(</span><span class="nf">store</span><span class="w"> </span><span class="n">collection</span><span class="w"> </span><span class="n">version</span><span class="w"> </span><span class="n">events</span><span class="p">))</span><span class="w">
</span><span class="p">(</span><span class="nf">load-events-from</span><span class="w"> </span><span class="p">[</span><span class="n">_</span><span class="w"> </span><span class="n">aggregate-id</span><span class="p">]</span><span class="w"> </span><span class="p">(</span><span class="nb">load</span><span class="w"> </span><span class="n">collection</span><span class="w"> </span><span class="n">aggregate-id</span><span class="p">))</span><span class="w">
</span><span class="n">java.lang.Object</span><span class="w">
</span><span class="p">(</span><span class="nf">toString</span><span class="w"> </span><span class="p">[</span><span class="n">this</span><span class="p">]</span><span class="w"> </span><span class="p">(</span><span class="nb">str</span><span class="w"> </span><span class="s">"<#MongoStore: "</span><span class="w"> </span><span class="n">collection</span><span class="w"> </span><span class="s">">"</span><span class="p">)))</span></code></pre></figure>
<p>which will dispatch the calls to the protocol to dedicated functions merging his internal state (the MongoDB collection to use) with the call parameter.</p>
<p>Let’s have a quick look at the simplest of the three functions which should be <code class="highlighter-rouge">version-from</code>, unfortunately this leads to some basics first.`</p>
<h4 id="versions-and-optimistic-locking">Versions and optimistic locking</h4>
<p>One important part of storing an event is to assign a version to the event to create an ordering. This is very important when you want to project events into a data structure. If you do not know is a the value of a field was first ‘A’ then ‘B’ or vice versa is bad.</p>
<p>How do we now assign versions? Unfortunately this is not an easy task in a distributed environment with only a MongoDB. If you use a SQL database you may use a sequence, <code class="highlighter-rouge">SELECT … FOR UPDATE</code> clause or an auto-increment field, but the MongoDB does not provide a similar mechanism.`</p>
<p>For this example I choose a quite simple strategy. As version I used (positive) integers starting at one. When inserting events I will fetch the current version out of the database and assign new versions to the events by simply increasing the version (by one). Additionally I create a unique index for the events collection containing the aggregate ID (<code class="highlighter-rouge">_aid</code>) and the version (<code class="highlighter-rouge">:_v)</code> field. If e.g. another process is doing the same in parallel we will have overlapping versions and MongoDB will raise an error when the second one tries to insert the event(s).</p>
<p>Using this simple strategy we have an optimistic locking, where no node need to lock down the whole collection (or even worse database) to make sure that no one else updated the data while he was working on them. We can take this even one step further by supplying the version of the aggregate in use back to the <code class="highlighter-rouge">(store-events-into ...)</code> function of the protocol.</p>
<p>You can find a simple realization of the <code class="highlighter-rouge">version-from</code> function on my <a href="https://github.com/niclasmeier/clj-eventsourcing/blob/master/src/clj_eventsourcing/store/mongo.clj">github repository.</a></p>
<h2 id="inserting-into-mongodb">Inserting into MongoDB</h2>
<p>The insertion of the events is now very simple. Provided we have a function that adds the version to the events you will have something like:](<a href="https://github.com/niclasmeier/clj-eventsourcing/blob/master/src/clj_eventsourcing/store/mongo.clj">https://github.com/niclasmeier/clj-eventsourcing/blob/master/src/clj_eventsourcing/store/mongo.clj</a>)</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">[(</span><span class="k">defn</span><span class="w"> </span><span class="n">store</span><span class="w"> </span><span class="p">[</span><span class="n">collection</span><span class="w"> </span><span class="n">version</span><span class="w"> </span><span class="n">events</span><span class="p">]</span><span class="w">
</span><span class="p">(</span><span class="nf">insert-batch</span><span class="w"> </span><span class="n">collection</span><span class="w"> </span><span class="p">(</span><span class="nf">enrich-events</span><span class="w"> </span><span class="n">version</span><span class="w"> </span><span class="n">events</span><span class="p">))</span></code></pre></figure>
<p>The only problem is that you have no safety net on this one. You want to check some constraints like:</p>
<ul>
<li>If the version is only a number, you should make sure that all events refer to the same aggregate</li>
<li>If the version is a map from aggregate ID to a number, you should make sure that you have versions for all aggregates referred by the events.</li>
<li>If an Exception about violating the unique index on the MongoDB collection is thrown, it might be a good idea converting it into something more useful (e.g. a different exception type)</li>
</ul>
<h1 id="loading-events">Loading events</h1>
<p>Loading events on the other hand side is very simple, now we known how the documents should look like:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">[(</span><span class="k">defn</span><span class="w"> </span><span class="nb">load</span><span class="w"> </span><span class="p">[</span><span class="n">collection</span><span class="w"> </span><span class="n">aggregate-id</span><span class="p">]</span><span class="w">
</span><span class="p">(</span><span class="nf">with-collection</span><span class="w"> </span><span class="n">collection</span><span class="w">
</span><span class="p">(</span><span class="nf">where</span><span class="w"> </span><span class="p">{</span><span class="no">:_aid</span><span class="w"> </span><span class="n">aggregate-id</span><span class="p">})</span><span class="w">
</span><span class="p">(</span><span class="nb">sort</span><span class="w"> </span><span class="p">{</span><span class="no">:_v</span><span class="w"> </span><span class="mi">1</span><span class="p">})))</span></code></pre></figure>
<p>Simply fetching all event documents from the MongoDB ordered by the version.</p>
<h1 id="putting-it-together">Putting it together</h1>
<p>Creating, loading and storing events with Clojure and storing them into MongoDB is not that hard and requires very lithe source code. You may use Protocols and Records to get a little bit more structure into this, but the initial event sourcing system used at HEROLABS was relying on Clojure maps and works fine.</p>
<p>Unfortunately we are far from done yet. Working on event streams is harder than fetching data from a business “object” (they will be maps or Records in Clojure). So the next step will be to explore how to project the events into data structures or operations, so that</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">[[{</span><span class="no">:_aid</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="no">:_v</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="no">:_t</span><span class="w"> </span><span class="no">:set-name</span><span class="w"> </span><span class="no">:first-name</span><span class="w"> </span><span class="s">"Niclas"</span><span class="w"> </span><span class="no">:last-name</span><span class="w"> </span><span class="s">"Meier"</span><span class="p">}</span><span class="w">
</span><span class="p">{</span><span class="no">:_aid</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="no">:_v</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="no">:_t</span><span class="w"> </span><span class="no">:add-blog</span><span class="w"> </span><span class="no">:url</span><span class="w"> </span><span class="s">"http://www.niclas-meier.de/"</span><span class="p">}]</span></code></pre></figure>
<p>will get</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">{</span><span class="no">:_aid</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="no">:_v</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="no">:first-name</span><span class="w"> </span><span class="s">"Niclas"</span><span class="w"> </span><span class="no">:last-name</span><span class="w"> </span><span class="s">"Meier"</span><span class="w"> </span><span class="no">:url</span><span class="w"> </span><span class="s">"http://www.niclas-meier.de/"</span><span class="p">}</span></code></pre></figure>
<p>or at least something like this, so stay tuned for part two. In the meantime you may check out some more source code at my <a href="https://github.com/niclasmeier/clj-eventsourcing">github repository</a>, but be aware that this is only an implementation sketch and not fully implemented.</p>Niclas MeierIt was about a year ago when I stumbled about the concepts of event sourcing and command query responsibility segregation in a talk from Greg Young at QCon 2011 in London which I watched on InfoQ. This challenged some things I knew for sure regarding how to store and manage data.Underestimated Amazon S32012-07-10T00:00:00+00:002012-07-10T00:00:00+00:00www.niclas-meier.de/clojure/amazon%20s3/2012/07/10/underestimated-amazon-s3<p>The <a href="http://aws.amazon.com/de/s3/">Amazon Simple Storage Service (S3)</a> is the big data work horse of Amazon and one of the corner stones of their cloud infrastructure. For our recent <a href="http://itunes.apple.com/de/app/em-hero-2012/id530513702?l=de&ls=1&mt=8">project</a> we received a considerable amount of game data from our data provider. At the moment these file are stored in the <a href="http://www.mongodb.org/display/DOCS/GridFS">GridFS</a> of our Mongo DB installation, but once the data is processed we don’t really need these files any more, except for development or archiving purposes.</p>
<p>At the moment we are using the <a href="http://aws.amazon.com/en/ec2/">Amazon Elastic Compute Cloud (EC2)</a> to run our infrastructure, so it was a logical choice to evaluate the <a href="http://aws.amazon.com/en/s3/">S3</a> to store the received game data and I have to confess I was really underestimating this service as simple file system.</p>
<h3 id="reliability">Reliability</h3>
<p>First of all, I wasn’t really aware that S3 has this high reliability of 99.999999999 percent according to the Amazon website. Which is more than enough for our needs. An guaranteed uptime of 99.99 percent is equivalent to max 52.56 minutes of downtime per year, but I am sure that we can’t / don’t want to build a system which offers equal values. You can also lower the cost of you S3 usage by choosing the reduced redundancy service which reduces the reliability of 99.99 percent, which is good enough e.g. for <a href="http://aws.amazon.com/en/elasticmapreduce/">elastic map reduce</a> usage.</p>
<p>To ensure a safe transmission of the data you can also specify a MD5 sum of the content you are storing into S3 to make sure that you have no errors while transferring the data.</p>
<h3 id="expiration">Expiration</h3>
<p>A very nice feature of S3 is that you can set expiration dates or rules for objects. E.g. we receive commentary data from out sports data provider to enrich our content. This content is nice but also automatically generated so we don’t really need to store it. In S3 we have now the possibility to delete these files automatically after 90 days (or something like this).</p>
<h3 id="meta-data">Meta-data</h3>
<p>You may associate meta data as key value pairs to any S3 object you store. These meta data is also transferred as HTTP header with <code class="highlighter-rouge">x-amz-meta-</code> prefix. So there is no need for additional <code class="highlighter-rouge">.md5</code> and <code class="highlighter-rouge">.meta</code> files. We will use this to store some game and data transfer related meta data to the files. The data transfer meta data is a map of several keys and values. As a lithe trick I am storing this additional data as JSON object.</p>
<h3 id="versioning">Versioning</h3>
<p>This is a big one. You can configure the S3 buckets - which are a kind of root file system on S3 - to store the files versioned. So you are able to get any previous version of an object like on <a href="http://git-scm.com/">git</a> or <a href="http://subversion.apache.org/">Subversion</a>.</p>
<p>This is perfect for our needs. We receive for data files like <code class="highlighter-rouge">commentary-3-2011-426847-de.xml</code> which contain all comments for a football (european not american) game. We are receiving a file with this name a couple of times during the game containing the complete data (updates and old data).</p>
<p>Using GridFS and <a href="http://www.mongodb.org">MongoDB</a> we are already keeping multiple versions of the file and it’s great to see, that we are able to store multiple versions in S3 too without auxiliary arrangements.</p>
<h3 id="java-sdk-and-clojure">Java SDK and Clojure</h3>
<p>At the moment we are using the Java SDK from Amazon with a slim Clojure wrapper. This works very well and you can easily wrap the S3 calls into a couple of Clojure functions like this:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">create-folder</span><span class="w"> </span><span class="p">[</span><span class="n">client</span><span class="w"> </span><span class="n">bucket-name</span><span class="w"> </span><span class="nb">key</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="s">"Creates an S3 folder"</span><span class="w">
</span><span class="p">(</span><span class="nb">when</span><span class="w"> </span><span class="p">(</span><span class="nb">and</span><span class="w"> </span><span class="n">client</span><span class="w"> </span><span class="n">bucket-name</span><span class="w"> </span><span class="nb">key</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="n">req</span><span class="w"> </span><span class="p">(</span><span class="nf">PutObjectRequest.</span><span class="w"> </span><span class="n">bucket-name</span><span class="w"> </span><span class="p">(</span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nf">.endsWith</span><span class="w"> </span><span class="nb">key</span><span class="w"> </span><span class="n">folder-extension</span><span class="p">)</span><span class="w"> </span><span class="nb">key</span><span class="w"> </span><span class="p">(</span><span class="nb">str</span><span class="w"> </span><span class="nb">key</span><span class="w"> </span><span class="n">folder-extension</span><span class="p">))</span><span class="w"> </span><span class="n">empty-content</span><span class="w"> </span><span class="n">empty-content-meta</span><span class="p">)</span><span class="w">
</span><span class="n">req</span><span class="w"> </span><span class="p">(</span><span class="k">if</span><span class="w"> </span><span class="n">acl</span><span class="w"> </span><span class="p">(</span><span class="nf">.withCannedAcl</span><span class="w"> </span><span class="n">req</span><span class="w"> </span><span class="n">acl</span><span class="p">)</span><span class="w"> </span><span class="n">req</span><span class="p">)</span><span class="w">
</span><span class="n">req</span><span class="w"> </span><span class="p">(</span><span class="k">if</span><span class="w"> </span><span class="n">listener</span><span class="w"> </span><span class="p">(</span><span class="nf">.withProgressListener</span><span class="w"> </span><span class="n">req</span><span class="w"> </span><span class="n">listener</span><span class="p">)</span><span class="w"> </span><span class="n">req</span><span class="p">)]</span><span class="w">
</span><span class="p">(</span><span class="nf">.putObject</span><span class="w"> </span><span class="n">client</span><span class="w"> </span><span class="n">req</span><span class="p">))))</span></code></pre></figure>
<p>Just create request instance (e.g. <code class="highlighter-rouge">PutObjectRequest</code>) and chain the continuation style <code class="highlighter-rouge">.with</code> Methods. The named arguments are a big help here. A little more work is to convert the result objects of the S3 API into Clojure style datatypes.</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">list-objects</span><span class="w"> </span><span class="p">[</span><span class="n">client</span><span class="w"> </span><span class="n">bucket-name</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="s">"Lists the objects in a bucket."</span><span class="w">
</span><span class="p">(</span><span class="nb">when</span><span class="w"> </span><span class="p">(</span><span class="nb">and</span><span class="w"> </span><span class="n">client</span><span class="w"> </span><span class="n">bucket-name</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="n">r</span><span class="w"> </span><span class="p">(</span><span class="nf">.listObjects</span><span class="w"> </span><span class="n">client</span><span class="w"> </span><span class="p">(</span><span class="nf">ListObjectsRequest.</span><span class="w"> </span><span class="n">bucket-name</span><span class="w"> </span><span class="n">prefix</span><span class="w"> </span><span class="n">marker</span><span class="w"> </span><span class="n">delimiter</span><span class="w"> </span><span class="n">max-keys</span><span class="p">))]</span><span class="w">
</span><span class="p">(</span><span class="nf">letfn</span><span class="w"> </span><span class="p">[(</span><span class="nf">summaries</span><span class="w"> </span><span class="p">[</span><span class="n">s</span><span class="p">]</span><span class="w"> </span><span class="p">(</span><span class="nf">lazy-seq</span><span class="w"> </span><span class="p">(</span><span class="nb">if-let</span><span class="w"> </span><span class="p">[</span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="nb">first</span><span class="w"> </span><span class="n">s</span><span class="p">)]</span><span class="w"> </span><span class="p">(</span><span class="nb">cons</span><span class="w"> </span><span class="p">(</span><span class="nf">coerce-s3-object-summary</span><span class="w"> </span><span class="n">n</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">summaries</span><span class="w"> </span><span class="p">(</span><span class="nb">rest</span><span class="w"> </span><span class="n">s</span><span class="p">)))</span><span class="w"> </span><span class="n">nil</span><span class="p">)))]</span><span class="w">
</span><span class="p">(</span><span class="nb">-</span><span class="o">&</span><span class="n">gt</span><span class="c1">; (transient {})
</span><span class="w"> </span><span class="p">(</span><span class="nf">assoc-if!</span><span class="w"> </span><span class="no">:bucket-name</span><span class="w"> </span><span class="p">(</span><span class="nf">.getBucketName</span><span class="w"> </span><span class="n">r</span><span class="p">))</span><span class="w">
</span><span class="p">(</span><span class="nf">assoc-if!</span><span class="w"> </span><span class="no">:prefix</span><span class="w"> </span><span class="p">(</span><span class="nf">.getPrefix</span><span class="w"> </span><span class="n">r</span><span class="p">))</span><span class="w">
</span><span class="p">(</span><span class="nf">assoc-if!</span><span class="w"> </span><span class="no">:marker</span><span class="w"> </span><span class="p">(</span><span class="nf">.getMarker</span><span class="w"> </span><span class="n">r</span><span class="p">))</span><span class="w">
</span><span class="p">(</span><span class="nf">assoc-if!</span><span class="w"> </span><span class="no">:max-keys</span><span class="w"> </span><span class="p">(</span><span class="nf">.getMaxKeys</span><span class="w"> </span><span class="n">r</span><span class="p">))</span><span class="w">
</span><span class="p">(</span><span class="nf">assoc-if!</span><span class="w"> </span><span class="no">:delimiter</span><span class="w"> </span><span class="p">(</span><span class="nf">.getDelimiter</span><span class="w"> </span><span class="n">r</span><span class="p">))</span><span class="w">
</span><span class="p">(</span><span class="nf">assoc-if!</span><span class="w"> </span><span class="no">:truncated</span><span class="w"> </span><span class="p">(</span><span class="nf">.isTruncated</span><span class="w"> </span><span class="n">r</span><span class="p">))</span><span class="w">
</span><span class="p">(</span><span class="nf">assoc-if!</span><span class="w"> </span><span class="no">:common-prefixes</span><span class="w"> </span><span class="p">(</span><span class="nb">when-let</span><span class="w"> </span><span class="p">[</span><span class="n">p</span><span class="w"> </span><span class="p">(</span><span class="nf">.getCommonPrefixes</span><span class="w"> </span><span class="n">r</span><span class="p">)]</span><span class="w"> </span><span class="p">(</span><span class="nb">when-not</span><span class="w"> </span><span class="p">(</span><span class="nf">empty?</span><span class="w"> </span><span class="n">p</span><span class="p">)</span><span class="w"> </span><span class="n">p</span><span class="p">)))</span><span class="w">
</span><span class="p">(</span><span class="nf">assoc-if!</span><span class="w"> </span><span class="no">:next-marker</span><span class="w"> </span><span class="p">(</span><span class="nf">.getNextMarker</span><span class="w"> </span><span class="n">r</span><span class="p">))</span><span class="w">
</span><span class="p">(</span><span class="nf">assoc-if!</span><span class="o">&</span><span class="n">nbsp</span><span class="c1">;:object-summaries (when-let [s (.getObjectSummaries r)] (summaries s)))
</span><span class="w"> </span><span class="n">persistent!</span><span class="p">)))))</span></code></pre></figure>
<p>But most of it is just repetitive work, but in the your Clojure code this looks really nice.</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="nf">list-objcts</span><span class="w"> </span><span class="n">c</span><span class="w"> </span><span class="s">"my-fancy-bucket"</span><span class="w"> </span><span class="no">:prefix</span><span class="w"> </span><span class="s">"foo"</span><span class="p">)</span></code></pre></figure>
<h3 id="conculsions">Conculsions</h3>
<p>I didn’t talk about some matters of course like access control, multiple APIs based on REST or SOAP available in Java, .NET or Ruby. Or other nice functions like uploading data as anonymous user on pre-signed URLs. But from my perspective this package is more than enough to keep a close look on the <a href="http://aws.amazon.com/de/s3/">Simple Storage Service</a>. I am pretty sure that the <a href="https://developers.google.com/drive/">equivalent Google Service</a> has similar capabilities, so both services should be considered if you want to store some of your data in the cloud.</p>
<p>Another important question, many of the european/german users will ask: “Where will my data be stored?” - Amazon will guarantee that, if you use the EU-Ireland zone the data will remain in the EU. This sounds great, even from the data protection point of view, but remember the fact that the european Amazon (Amazon EU S.à.r.l) is a owned to hundred percent by the US Amazon. So if the Department of Homeland Security want’s a DVD with all of you data, I am pretty confident that they will get it.</p>
<p>In the end, I am more than ever convinced that S3 is a very useful tool and I am exited to continue to work with it.</p>Niclas MeierThe Amazon Simple Storage Service (S3) is the big data work horse of Amazon and one of the corner stones of their cloud infrastructure. For our recent project we received a considerable amount of game data from our data provider. At the moment these file are stored in the GridFS of our Mongo DB installation, but once the data is processed we don’t really need these files any more, except for development or archiving purposes.From Promises to Futures2012-06-10T00:00:00+00:002012-06-10T00:00:00+00:00www.niclas-meier.de/java/clojure/2012/06/10/from-promises-to-futures<p>This is the second post on an alternate implementation/approach to Futures and Promises on the Clojure programming language. In my <a href="/2012/06/alternate-futures-and-promises-for-clojure/">first post</a> I talked a little bit about strength and weaknesses of the the actual implementation of Futures and Promises in Clojure and I defined how my Future and Clojure implementation should work.</p>
<p>At the end of the <a href="/2012/06/alternate-futures-and-promises-for-clojure/">first post</a> I described how to implement a Promise in Clojure. A Promise in the context of my implementation should be a write handle to e.g. pass later on computed values to other processes.</p>
<p>In this blog post I want to:</p>
<ul>
<li>show you how to get from the Promise implementation to a simple Future realization</li>
<li>talk about different execution strategies and why it’s good do have influence on the future execution</li>
</ul>
<p>So let’s get started …</p>
<p>Why it’s good to have the control about the execution of your futures? With futures you are delegating the execution of your code to another thread (if we stick to the JVM) and getting the results back in your thread. So where is this other thread coming from?</p>
<p>This is very important question, because on the one hand creating a thread is a quite costly operation and on the other hand having very many threads is also not very good for your performance on the JVM, due to the heady scheduling required. E.g. if you have a quad core server you may execute for things in parallel, if you have 400 thread in you JVM the virtual machine schedules the execution of these 400 threads, wanting to to execute something, on these four cores. This can add a very huge amount of internal processing load to your server. Additionally every Thread has to allocate some memory for its call stack (see also <a href="http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html">-XX:ThreadStackSize=512</a>) which also increases the memory consumption.</p>
<p>So it’s not very clever to create a new Thread every time you want to execute a future and its also not a good idea to have hundreds of threads hanging around and waiting for work. Prior to Java 5 you had to create and manage threads in pools manually. But nowadays the <a href="http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/Executor.html"><code class="highlighter-rouge">java.util.concurrent.Executor</code></a> and <a href="http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/ExecutorService.html"><code class="highlighter-rouge">java.util.concurrent.ExecutorService</code></a> are the weapons of choice. You can obtain instances from <a href="http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/Executors.html"><code class="highlighter-rouge">java.util.concurrent.Executors</code></a> and if you take a look at the <code class="highlighter-rouge">java.util.concurrent.Executors</code> you will find several different methods to create <code class="highlighter-rouge">ExecutorService</code>s with different execution strategies. If you have a look at Java 7 you’ll find event more implementations like the <a href="http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ForkJoinPool.html"><code class="highlighter-rouge">java.util.concurrent.ForkJoinPool</code></a>. Why do need all this different implementations? Unfortunately there is no solution that fits for any problem domain. Let’s have two examples:</p>
<p>In the first scenario let’s image that you have a legacy service which is optimized for batch processing and has really limited online query capabilities. So if you are connecting to this system via futures you maybe want to make sure that you cannot flood the service accidentially. So you would use a e.g. a fixed thread pool using <a href="http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Executors.html#newFixedThreadPool(int)"><code class="highlighter-rouge">newFixedThreadPool</code></a> to create a pool with a fixed number of threads, so if you are flooding the system the waiting will occur on your thread pool and not messing up your legacy system.</p>
<p>Another scenario would be e.g. a service quering Facebook for user data. Assumed that Facebook has unlimited scalability ;-) there is no sense in limiting the access to Facebook on your side but you have to consider that some request may take very long due to the internet. So maybe <a href="http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Executors.html#newCachedThreadPool()"><code class="highlighter-rouge">newCachedThreadPool</code></a> would be a good choice or you would consider the <a href="http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ForkJoinPool.html"><code class="highlighter-rouge">ForkJoinPool</code></a> which optimizes the throughput with a very small number of threads. How the choice of the right strategy can influence the throughput of your system was measured and documented by the <a href="http://akka.io/">Akka</a> team in a nice <a href="http://letitcrash.com/post/17607272336/scalability-of-fork-join-pool">blog post</a>. When I am not totally wrong the results of the Akka guys also influenced <a href="http://cs.oswego.edu/pipermail/concurrency-interest/2012-January/008987.html">some updates on the <code class="highlighter-rouge">ForkJoinPool</code></a> which are announced for Java 8.</p>
<p>So to summarize this: It’s a good idea to be in control of your execution strategy. If you have a look at the simple solution Future implementation Clojure even I would say, it meets 90% of the requirements even with no big performance penalty. And if you want to have a special solution you can easily create your special <code class="highlighter-rouge">java.util.concurrent.Executor</code> and submit your code there. If you remember the your function is a runnable a <code class="highlighter-rouge">(.submit executor (fn [] (do-some-heavy-loading this and that)))</code> should do the job.</p>
<p>Enough on the theory stuff, so how to we implement the alternate Futures now in the <a href="https://github.com/niclasmeier/clj-future">demo project</a>?</p>
<p>Lets think about what we need for implementing the Future:</p>
<ul>
<li>Another thread that will do the work for us</li>
<li>A reference to get the computed value later</li>
</ul>
<p>We talked about the thread already and I will demonstrate how you may use it in a moment, so lets take a look at this reference thingy first. The reference need to be readable from my current thread and I need to get the reference in the very moment the I call the Future to execute my code. The thread which performs the execution must be able to write the result value of my computation to this thing. In the <a href="/2012/06/alternate-futures-and-promises-for-clojure/">previous post</a> I showed how to implement a promise. This is a read handle to some value that will be available sometime and it is writable (once) to the current thread. So vóila there we are, we can use the promise to pass the value from the worker thread back to the original thread.</p>
<p>So how does this look like in Clojure code: Let’s define a default executor at the beginning. An executor will be a simple function that will execute a passed function on another thread</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="k">def</span><span class="w"> </span><span class="o">^</span><span class="no">:private</span><span class="w"> </span><span class="n">default-executor*</span><span class="w"> </span><span class="p">(</span><span class="nf">atom</span><span class="w"> </span><span class="n">nil</span><span class="p">))</span><span class="w">
</span><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">default-executor</span><span class="w"> </span><span class="p">[]</span><span class="w"> </span><span class="p">(</span><span class="nb">or</span><span class="w"> </span><span class="err">@</span><span class="n">default-executor*</span><span class="w">
</span><span class="p">(</span><span class="nf">swap!</span><span class="w"> </span><span class="n">default-executor*</span><span class="w"> </span><span class="p">(</span><span class="k">fn</span><span class="w"> </span><span class="p">[</span><span class="n">_</span><span class="p">]</span><span class="w"> </span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="n">s</span><span class="w"> </span><span class="p">(</span><span class="nf">Executors/newCachedThreadPool</span><span class="p">)]</span><span class="w">
</span><span class="p">(</span><span class="k">fn</span><span class="w"> </span><span class="p">[</span><span class="o">^</span><span class="n">Runnable</span><span class="w"> </span><span class="n">r</span><span class="p">]</span><span class="w"> </span><span class="p">(</span><span class="nf">.submit</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">r</span><span class="p">)))))))</span></code></pre></figure>
<p>Very simple construction. The <code class="highlighter-rouge">(default-executor …)</code> function will return the default executor function and the private var will store an atom the contains a previous initialized value. In this example I simple choose a cached thread pool, which is the same that Clojure uses internally for its futures.</p>
<p>Next step is to create an execution function like <code class="highlighter-rouge">(future-call …)</code>:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">execute-future</span><span class="w"> </span><span class="p">[</span><span class="no">:as</span><span class="w"> </span><span class="n">params</span><span class="p">}]</span><span class="w">
</span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="n">p</span><span class="w"> </span><span class="p">(</span><span class="nf">promise</span><span class="w"> </span><span class="no">:on-complete</span><span class="w"> </span><span class="n">on-complete</span><span class="w"> </span><span class="no">:on-success</span><span class="w"> </span><span class="n">on-success</span><span class="w"> </span><span class="no">:on-failure</span><span class="w"> </span><span class="n">on-failure</span><span class="p">)</span><span class="w">
</span><span class="n">runnable</span><span class="w"> </span><span class="p">(</span><span class="k">fn</span><span class="w"> </span><span class="p">[]</span><span class="w"> </span><span class="p">(</span><span class="nf">try</span><span class="w">
</span><span class="p">(</span><span class="nf">try-success</span><span class="w"> </span><span class="n">p</span><span class="w"> </span><span class="p">(</span><span class="nf">body</span><span class="p">))</span><span class="w">
</span><span class="p">(</span><span class="nf">catch</span><span class="w"> </span><span class="n">java.lang.Exception</span><span class="w"> </span><span class="n">e</span><span class="w">
</span><span class="p">(</span><span class="nf">try-failure</span><span class="w"> </span><span class="n">p</span><span class="w"> </span><span class="n">e</span><span class="p">))))]</span><span class="w">
</span><span class="p">(</span><span class="nf">executor</span><span class="w"> </span><span class="n">runnable</span><span class="p">)</span><span class="w">
</span><span class="n">p</span><span class="p">))</span></code></pre></figure>
<p>Jepp, that is it ;-) So how does it work? You may pass a body function that will be executed asynchronously and the trinity of<code class="highlighter-rouge"> :on-complete</code>,<code class="highlighter-rouge"> :on-success</code> and<code class="highlighter-rouge"> :on-failure</code> hanlders. Additionally you may pass an executor. If you don’t pass an executor the previously defined default will be used.</p>
<p>The first thing that happens is, that we create a Promise instance and then wrap the body function to a runnable Function which will be passed to the executor function. The runnable function is nothing more like a simple <code class="highlighter-rouge">try-catch</code> expression which will trigger the <code class="highlighter-rouge">try-success</code> or the <code class="highlighter-rouge">try-failure</code> functions of the promise. The only things left is to call the executor function with the fresh runnable function and return the promise to the caller.</p>
<p>Nice, isn’t it? Almost, because your code will look like:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="nf">execute-future</span><span class="w">
</span><span class="no">:body</span><span class="w"> </span><span class="p">(</span><span class="k">fn</span><span class="w"> </span><span class="p">[]</span><span class="w"> </span><span class="p">(</span><span class="nf">java.lang.Thread/sleep</span><span class="w"> </span><span class="mi">1000</span><span class="p">)</span><span class="w"> </span><span class="mi">42</span><span class="p">)</span><span class="w">
</span><span class="no">:on-success</span><span class="w"> </span><span class="p">(</span><span class="k">fn</span><span class="w"> </span><span class="p">[</span><span class="n">v</span><span class="p">]</span><span class="w"> </span><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="s">"Received:"</span><span class="w"> </span><span class="n">v</span><span class="p">))</span><span class="w">
</span><span class="no">:on-failure</span><span class="w"> </span><span class="p">(</span><span class="k">fn</span><span class="w"> </span><span class="p">[</span><span class="n">e</span><span class="p">]</span><span class="w"> </span><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="s">"Exception:"</span><span class="w"> </span><span class="n">e</span><span class="p">)))</span></code></pre></figure>
<p>Kind of clumsy, so let’s enhance it. But Clojure has the possibility to use macros to make it good looking.</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="k">defmacro</span><span class="w"> </span><span class="n">future</span><span class="w"> </span><span class="p">[</span><span class="n">body</span><span class="w"> </span><span class="o">&</span><span class="w"> </span><span class="n">callbacks</span><span class="p">]</span><span class="w">
</span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="n">params</span><span class="w"> </span><span class="p">(</span><span class="nb">reduce</span><span class="w"> </span><span class="p">(</span><span class="k">fn</span><span class="w"> </span><span class="p">[</span><span class="n">l</span><span class="w"> </span><span class="n">e</span><span class="p">]</span><span class="w"> </span><span class="p">(</span><span class="nb">concat</span><span class="w"> </span><span class="n">l</span><span class="w"> </span><span class="n">e</span><span class="p">))</span><span class="w"> </span><span class="p">[</span><span class="no">:body</span><span class="w"> </span><span class="o">`</span><span class="p">(</span><span class="k">fn</span><span class="w"> </span><span class="p">[]</span><span class="w"> </span><span class="o">~</span><span class="n">body</span><span class="p">)]</span><span class="w">
</span><span class="p">(</span><span class="nb">select-keys</span><span class="w">
</span><span class="p">(</span><span class="nb">reduce</span><span class="w"> </span><span class="p">(</span><span class="k">fn</span><span class="w"> </span><span class="p">[</span><span class="n">m</span><span class="w"> </span><span class="n">form</span><span class="p">]</span><span class="w"> </span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[[</span><span class="n">k</span><span class="w"> </span><span class="o">&</span><span class="w"> </span><span class="n">r</span><span class="p">]</span><span class="w"> </span><span class="n">form</span><span class="w">
</span><span class="k">fn</span><span class="w"> </span><span class="p">(</span><span class="nb">cons</span><span class="w"> </span><span class="ss">'fn</span><span class="w"> </span><span class="n">r</span><span class="p">)]</span><span class="w"> </span><span class="p">(</span><span class="nb">assoc</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="p">(</span><span class="nb">keyword</span><span class="w"> </span><span class="n">k</span><span class="p">)</span><span class="w"> </span><span class="k">fn</span><span class="p">)))</span><span class="w"> </span><span class="n">nil</span><span class="w"> </span><span class="n">callbacks</span><span class="p">)</span><span class="w"> </span><span class="p">[</span><span class="no">:on-complete</span><span class="err"> </span><span class="no">:on-success</span><span class="err"> </span><span class="no">:on-failure</span><span class="w"> </span><span class="p">]))]</span><span class="w">
</span><span class="o">`</span><span class="p">(</span><span class="nf">execute-future</span><span class="w"> </span><span class="o">~@</span><span class="n">params</span><span class="p">)))</span></code></pre></figure>
<p>This is small but a little bit nasty, so I simply tell you how beautiful the code will look instead of talking about the macro.</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="nf">future</span><span class="w"> </span><span class="p">(</span><span class="nf">do</span><span class="w"> </span><span class="p">(</span><span class="nf">java.lang.Thread/sleep</span><span class="p">)</span><span class="w"> </span><span class="mi">42</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="nf">on-success</span><span class="w"> </span><span class="p">[</span><span class="n">v</span><span class="p">]</span><span class="w"> </span><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="s">"Received:"</span><span class="w"> </span><span class="n">v</span><span class="p">))</span><span class="w">
</span><span class="p">(</span><span class="nf">on-failure</span><span class="w"> </span><span class="p">[</span><span class="n">e</span><span class="p">]</span><span class="w"> </span><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="s">"Exception:"</span><span class="w"> </span><span class="n">e</span><span class="p">)))</span></code></pre></figure>
<p>So that’s more elegant - at least from my point of view - and I meet the goal that I set to myself. Looking exactly like my idea in the <a href="/2012/06/alternate-futures-and-promises-for-clojure/">previous post</a>.</p>
<p>But how do I influence the execution when using the macro. That’s the part I swept under the rug a little bit. In the <a href="https://github.com/niclasmeier/clj-future">demo project</a> you will find a macro for something like <code class="highlighter-rouge">(with-executor my-special-fork-join-executor …)</code> which will use <code class="highlighter-rouge">(binding ...)</code> to override the default for a certain code block.</p>
<h3 id="so-whats-next">So what’s next?</h3>
<p>The next open issue would be to create composeable futures to be able to do stuff like</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="n">f</span><span class="w"> </span><span class="p">(</span><span class="k">for</span><span class="w"> </span><span class="p">[</span><span class="n">order-data</span><span class="w"> </span><span class="p">(</span><span class="nf">future</span><span class="w"> </span><span class="p">(</span><span class="nf">fetch-order</span><span class="w"> </span><span class="n">oid</span><span class="p">))</span><span class="w">
</span><span class="n">customer-data</span><span class="w"> </span><span class="p">(</span><span class="nf">future</span><span class="w"> </span><span class="p">(</span><span class="nf">fetch-customer</span><span class="w"> </span><span class="n">cid</span><span class="p">))]</span><span class="w"> </span><span class="p">(</span><span class="nf">merge-data</span><span class="w"> </span><span class="n">order-data</span><span class="w"> </span><span class="n">customer-data</span><span class="p">))]</span><span class="w">
</span><span class="p">(</span><span class="nf">on-success</span><span class="w"> </span><span class="n">f</span><span class="w"> </span><span class="p">[</span><span class="n">data</span><span class="p">]</span><span class="w"> </span><span class="p">(</span><span class="nf">send-email</span><span class="w"> </span><span class="n">data</span><span class="p">)))</span></code></pre></figure>
<p>That would be the next challenging thing and I am not sure if I will achieve this. If you keep a look at the demo project you will see my progress and at the moment you will see, that I haven’t started yet.</p>
<p>I will try to keep you informed, so stay tuned …</p>Niclas MeierThis is the second post on an alternate implementation/approach to Futures and Promises on the Clojure programming language. In my first post I talked a little bit about strength and weaknesses of the the actual implementation of Futures and Promises in Clojure and I defined how my Future and Clojure implementation should work.Alternate Futures and Promises for Clojure2012-06-01T00:00:00+00:002012-06-01T00:00:00+00:00www.niclas-meier.de/java/scala/clojure/2012/06/01/alternate-futures-and-promises-for-clojure<p>At the moment I am experimenting with various techniques to speed up some update queries from our <a href="http://www.emhero2012.com">client</a>. One important element of this efforts is making things parallel. E.g. to compute a list update of deltas for the client we need to compare the version of the user on the client and in the database and we need the user data at some other places on the delta computation. So it would be a good idea to initiate the loading of the user data asynchronous right at the beginning of the request processing. While the user data is loaded we can load/compute some other stuff (e.g. we need some data about the <a href="http://de.uefa.com/uefaeuro/index.html">competition</a> and the matches). When all the required information is present continue the rest of the the job (e.g. generate some XML or JSON).</p>
<p>Futures implementations like <a href="http://docs.oracle.com/javase/6/docs/api/java/util/concurrent/Future.html"><code class="highlighter-rouge">java.util.concurrent.Future</code></a>, <a href="http://www.scala-lang.org/archives/downloads/distrib/files/nightly/docs/library/scala/concurrent/Future.html"><code class="highlighter-rouge">scala.concurrent.Future</code></a> or <a href="http://clojure.github.com/clojure/clojure.core-api.html#clojure.core/future"><code class="highlighter-rouge">clojure.core/future</code></a> seem to be the weapon of choice for this.</p>
<p>Due to the fact that our backend is written in <a href="http://clojure.org/">Clojure</a> I took a closer look at the Clojure Future implementation <code class="highlighter-rouge">clojure.core/future</code>. The Clojure implementation is pretty basic and very similar the the underlying Java implementation. This isn’t bad, but rather unsatisfying for a language emphasizing on concurrency like Clojure.<a id="more"></a><a id="more-156"></a>The Clojure Future API consists of these functions in the <code class="highlighter-rouge">clojure.core</code> namespace:</p>
<ul>
<li><code class="highlighter-rouge">(future …)</code> - Creates a future instance and executes the provided code.</li>
<li><code class="highlighter-rouge">(future-call …)</code>- Creates a future instance and executes a provided function. This function is used by the <code class="highlighter-rouge">(future …)</code> marco.</li>
<li><code class="highlighter-rouge">(future-cancel …)</code> - Cancels the further, if possible.</li>
<li><code class="highlighter-rouge">(future-cancelled? …)</code> - Returns <code class="highlighter-rouge">true</code> if future f is cancelled</li>
<li><code class="highlighter-rouge">(promise …)</code> - Returns a promise object that can be read with <code class="highlighter-rouge">deref</code>/<code class="highlighter-rouge">@</code>, and set, once only.</li>
<li><code class="highlighter-rouge">(deliver …)</code> - Delivers the supplied value to the promise, releasing any pending derefs. A subsequent call to deliver on a promise will throw an exception.</li>
</ul>
<p>This is a small but sufficient API. The <code class="highlighter-rouge">promise</code>/<code class="highlighter-rouge">deliver</code> functions are relatively new and having a <em>“Alpha - subject to change”</em> note.</p>
<p>So how does this work:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="n">f</span><span class="w"> </span><span class="p">(</span><span class="nf">future</span><span class="w"> </span><span class="p">(</span><span class="nf">do</span><span class="w"> </span><span class="p">(</span><span class="nf">java.lang.Thread/sleep</span><span class="w"> </span><span class="mi">42</span><span class="p">))]</span><span class="w">
</span><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="s">"value:"</span><span class="w"> </span><span class="err">@</span><span class="n">f</span><span class="p">)</span><span class="w">
</span><span class="p">)</span></code></pre></figure>
<p>You simply wrap the code you want the be execute in parallel in the <code class="highlighter-rouge">(future …)</code> macro and you get a future datatype which reifies <code class="highlighter-rouge">clojure.lang.IDeref</code>, <code class="highlighter-rouge">clojure.lang.IBlockingDeref</code>, <code class="highlighter-rouge">clojure.lang.IPending</code> and <code class="highlighter-rouge">java.util.concurrent.Future</code>. This has some pretty nice effects. The future can be read with <code class="highlighter-rouge">@</code>/<code class="highlighter-rouge">deref</code> and you may use it like any other Java future.</p>
<p>Not so nice is that you have the <code class="highlighter-rouge">(future-cancel …)</code> function or the <code class="highlighter-rouge">cancel()</code> method in the <code class="highlighter-rouge">java.util.concurrent.Future</code> interface. So what is so bad about this? First: This is a mutating function on the reference, so if you pass your future running a important job to someone else, he may simply cancel this job and you can’t prevent this. Second: What happens when you cancel the job? When your long running job has a database update at the end, will the update be executed? Will truncations be rolled back? Connections closed?</p>
<p>That you don’t get noticed when the future processing is finished is somewhat sad too. Due to the fact the functions are first class citizens and closures are all around, it would be great if you could provide e.g. a function as callback when the future was successfully finished. The Scala scala.concurrent.Future implementation offers method like <code class="highlighter-rouge">onComplete(…)</code>, <code class="highlighter-rouge">onSuccess(…)</code> or <code class="highlighter-rouge">onFailure(…)</code> where you can supply these kind of handlers. What is also great on Scala Futures is that they offer methods like <code class="highlighter-rouge">flatMap</code>, etc. which are used for the Scala <code class="highlighter-rouge">for</code> comprehension. This enables you to do stuff like this:</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">for</span> <span class="o">(</span><span class="n">a</span> <span class="k"><-</span> <span class="n">someExpensiveComputation</span><span class="o">()</span>
<span class="n">b</span> <span class="k"><-</span> <span class="n">hugeLoadFromDatabase</span><span class="o">())</span>
<span class="k">yield</span> <span class="n">merge</span> <span class="o">(</span><span class="n">a</span><span class="o">,</span> <span class="n">b</span><span class="o">)</span></code></pre></figure>
<p>This would execute in parallel the <code class="highlighter-rouge">someExpensiveComputation</code> and <code class="highlighter-rouge">hugeLoadFromDatabase</code> and both results are available in the list comprehension.</p>
<p>The last flaw I see is, that you have no control about the execution. Clojure (1.3) uses the same <code class="highlighter-rouge">Executor</code> used for <code class="highlighter-rouge">(send-off …)</code> to execute the future but you cant’t influence this and you have no control on the executor. In Java there is lot’s of boilerplate, because you’ll have to create an <code class="highlighter-rouge">Executor</code> and submit a <code class="highlighter-rouge">Runnable</code> manually, therefore you have the full control. Scala had also this flaw hiding the execution of a Future but with the rework of Futures and Promises in Scala (<a href="http://docs.scala-lang.org/sips/pending/futures-promises.html">SIP-14</a> an <code class="highlighter-rouge">ExecutionContext</code> was introduced, which solves this issue.</p>
<p>So let’s explore how we can build a Future / Promise library in Clojure where that does not have these issues.</p>
<hr />
<h3 id="the-goal">The goal</h3>
<p>My goal would be to write something like this</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="nf">future</span><span class="w"> </span><span class="p">(</span><span class="nf">load-data-from-somewhere</span><span class="w"> </span><span class="n">ctx</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="nf">on-success</span><span class="w"> </span><span class="p">[</span><span class="n">v</span><span class="p">]</span><span class="w"> </span><span class="p">(</span><span class="nf">initiate-process</span><span class="w"> </span><span class="n">v</span><span class="p">))</span><span class="w">
</span><span class="p">(</span><span class="nf">on-failure</span><span class="w"> </span><span class="p">[</span><span class="n">cause</span><span class="p">]</span><span class="w"> </span><span class="p">(</span><span class="nf">warn</span><span class="w"> </span><span class="n">cause</span><span class="w"> </span><span class="p">(</span><span class="nf">format</span><span class="w"> </span><span class="s">"Unable to load data, retrying in %s seconds"</span><span class="w"> </span><span class="n">retry-threshold</span><span class="p">))))</span></code></pre></figure>
<p>it would be also nice to do something like</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="n">f</span><span class="w"> </span><span class="p">(</span><span class="k">for</span><span class="w"> </span><span class="p">[</span><span class="n">order-data</span><span class="w"> </span><span class="p">(</span><span class="nf">future</span><span class="w"> </span><span class="p">(</span><span class="nf">fetch-order</span><span class="w"> </span><span class="n">oid</span><span class="p">))</span><span class="w">
</span><span class="n">customer-data</span><span class="w"> </span><span class="p">(</span><span class="nf">future</span><span class="w"> </span><span class="p">(</span><span class="nf">fetch-customer</span><span class="w"> </span><span class="n">cid</span><span class="p">))]</span><span class="w"> </span><span class="p">(</span><span class="nf">merge-data</span><span class="w"> </span><span class="n">order-data</span><span class="w"> </span><span class="n">customer-data</span><span class="p">))]</span><span class="w">
</span><span class="p">(</span><span class="nf">on-success</span><span class="w"> </span><span class="n">f</span><span class="w"> </span><span class="p">[</span><span class="n">data</span><span class="p">]</span><span class="w"> </span><span class="p">(</span><span class="nf">send-email</span><span class="w"> </span><span class="n">data</span><span class="p">)))</span></code></pre></figure>
<p>This would return a future that aggregates the result of both fetch requests. You would be able to register for the <code class="highlighter-rouge">:on-success</code> event and perform an operation if wanted.</p>
<p>Lets see how far we get …</p>
<h3 id="definition-of-terms">Definition of Terms</h3>
<p>First - as some kind of Homework - I want to define what a Future and a Promise is for me (and yes this is heavily inspired by other sources - e.g. some talks by <a href="https://twitter.com/#!/viktorklang">Viktor Klang</a> or the <a href="http://docs.scala-lang.org/sips/pending/futures-promises.html">SIP-14</a>).</p>
<ul>
<li><em>Future</em> - A Future is a read handle to a value that may be come available some times later. A Future may be read many times.</li>
<li><em>Promise</em> - A Promise is a write handle for a value that will be available in the later process. You may pass the promise out to someone else to provice a reference to a value that you don’t have yet. A promise may be written only one.</li>
</ul>
<p>A Future and a Promise - at least in this example - will have this three states:
* <code class="highlighter-rouge">:pending</code> - The Future/Promise has not yet received a value.
* <code class="highlighter-rouge">:success</code> - The Promise/Future has received the a value and the previous computation has to considered successful.
* <code class="highlighter-rouge">:failure</code> - The Promise/Future has received the a value and the previous computation failed e.g. threw an Exception.</p>
<h3 id="promise---the-basics">Promise - the basics</h3>
<p>So let’s look at some code and start with some <a href="https://github.com/niclasmeier/clj-future">code</a>. You will find a somewhat working <a href="https://github.com/niclasmeier/clj-future">demo project on github.com</a></p>
<p>First I want to define some artifacts needed for a Promise.</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">promise</span><span class="w"> </span><span class="p">[]</span><span class="w"> </span><span class="err">…</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">status</span><span class="w"> </span><span class="p">[</span><span class="n">promise</span><span class="p">]</span><span class="w"> </span><span class="err">…</span><span class="p">)</span></code></pre></figure>
<p>I guess these are the first very functions needed for a promise. The first one <code class="highlighter-rouge">(promise …)</code> is a basic factory function for the handle and the <code class="highlighter-rouge">(status promise)</code> returns the status. The promise will be a <code class="highlighter-rouge">ref</code> data type, so the value may be obtained by using <code class="highlighter-rouge">deref</code>.To set the value we will define functions like:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">success</span><span class="w"> </span><span class="p">[</span><span class="n">promise</span><span class="w"> </span><span class="n">value</span><span class="p">]</span><span class="w"> </span><span class="err">…</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">failure</span><span class="w"> </span><span class="p">[</span><span class="n">promise</span><span class="w"> </span><span class="n">cause</span><span class="p">]</span><span class="w"> </span><span class="err">…</span><span class="p">)</span></code></pre></figure>
<p>These two function will replace the <code class="highlighter-rouge">(declare …)</code> function of the Clojure original. The <code class="highlighter-rouge">(success …)</code> function will set the Promise value to the passed value and the state of the promise will change to <code class="highlighter-rouge">:success</code>. The <code class="highlighter-rouge">(failure …)</code> function will indicate that the computation that gave out the Promise failed, the value of the Promise is reflects the cause for the failure and the state of the promise will change to <code class="highlighter-rouge">:failed</code>. Both functions will throw an exception if someone tries to write to the Promise a second time.</p>
<p>How do we implement the handle. I defined a Clojure protocol to have an interface:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="nf">defprotocol</span><span class="w"> </span><span class="n">Promise</span><span class="w">
</span><span class="p">(</span><span class="nf">future*</span><span class="w"> </span><span class="p">[</span><span class="n">promise</span><span class="p">])</span><span class="w">
</span><span class="p">(</span><span class="nf">try-success*</span><span class="w"> </span><span class="p">[</span><span class="n">promise</span><span class="w"> </span><span class="n">x</span><span class="p">])</span><span class="w">
</span><span class="p">(</span><span class="nf">try-failure*</span><span class="w"> </span><span class="p">[</span><span class="n">promise</span><span class="w"> </span><span class="n">x</span><span class="p">])</span><span class="w">
</span><span class="p">(</span><span class="nf">status*</span><span class="w"> </span><span class="p">[</span><span class="n">promise</span><span class="p">])</span><span class="w">
</span><span class="p">)</span></code></pre></figure>
<p>Pretty simple I guess so here is how it works:</p>
<ul>
<li><code class="highlighter-rouge">(future* [promise])</code> - This function will create a future for this promise. So someone create a promise and pass it to an asynchronous process while passing out a future to another process that will receive the value when the value was written to the promise. In fact in my variant it simply returns itself.</li>
<li><code class="highlighter-rouge">(try-success* [promise x])</code> - This function try to write a value to the promise and switch to the <code class="highlighter-rouge">:success</code> state</li>
<li><code class="highlighter-rouge">(try-failure* [promise x])</code> - This function try to write a value to the promise and switch to the <code class="highlighter-rouge">:success</code> state</li>
<li><code class="highlighter-rouge">(status* [promise])</code> - This function returns the current state of the promise.</li>
</ul>
<p>Why the <code class="highlighter-rouge">*</code> at the end of the function name. I use this convention to indicate some private/delegate functions. E.g. the <code class="highlighter-rouge">(success …)</code> function will simply delegate to the <code class="highlighter-rouge">(try-success* [promise x] …)</code>. The main difference is that both <code class="highlighter-rouge">(try-…)</code> method returns a boolean value indicating a successful operation while <code class="highlighter-rouge">(success …)</code> returns the supply value and will throw an exception if an error occurs.</p>
<p>The next step is to create an instance:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="k">defn-</span><span class="w"> </span><span class="n">notify</span><span class="w"> </span><span class="p">[</span><span class="n">x</span><span class="w"> </span><span class="o">&</span><span class="w"> </span><span class="n">listeners</span><span class="p">]</span><span class="w">
</span><span class="p">(</span><span class="nb">doseq</span><span class="w"> </span><span class="p">[</span><span class="n">listener</span><span class="w"> </span><span class="n">listeners</span><span class="w"> </span><span class="no">:when</span><span class="w"> </span><span class="n">listener</span><span class="p">]</span><span class="w"> </span><span class="p">(</span><span class="nf">listener</span><span class="w"> </span><span class="n">x</span><span class="p">)))</span><span class="w">
</span><span class="p">(</span><span class="k">defn-</span><span class="w"> </span><span class="n">try-complete-promise</span><span class="w"> </span><span class="p">[</span><span class="o">^</span><span class="n">java.util.concurrent.CountDownLatch</span><span class="w"> </span><span class="n">d</span><span class="w"> </span><span class="n">v</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="n">complete</span><span class="w"> </span><span class="n">variable</span><span class="p">]</span><span class="w">
</span><span class="p">(</span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nb">pos?</span><span class="w"> </span><span class="p">(</span><span class="nf">.getCount</span><span class="w"> </span><span class="n">d</span><span class="p">))</span><span class="w">
</span><span class="p">(</span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nf">compare-and-set!</span><span class="w"> </span><span class="n">v</span><span class="w"> </span><span class="n">nil</span><span class="w"> </span><span class="p">[</span><span class="n">s</span><span class="w"> </span><span class="n">x</span><span class="p">])</span><span class="w">
</span><span class="p">(</span><span class="nf">do</span><span class="w">
</span><span class="p">(</span><span class="nf">.countDown</span><span class="w"> </span><span class="n">d</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="nb">apply</span><span class="w"> </span><span class="n">notify</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="p">(</span><span class="nb">concat</span><span class="w"> </span><span class="err">@</span><span class="n">complete</span><span class="w"> </span><span class="err">@</span><span class="n">variable</span><span class="p">))</span><span class="w">
</span><span class="n">true</span><span class="p">)</span><span class="w">
</span><span class="n">false</span><span class="p">)))</span><span class="w">
</span><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">promise</span><span class="w"> </span><span class="p">[]</span><span class="w">
</span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="n">d</span><span class="w"> </span><span class="p">(</span><span class="nf">java.util.concurrent.CountDownLatch.</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w">
</span><span class="n">v</span><span class="w"> </span><span class="p">(</span><span class="nf">atom</span><span class="w"> </span><span class="n">nil</span><span class="p">)</span><span class="w">
</span><span class="n">on-complete</span><span class="w"> </span><span class="p">(</span><span class="nf">create-list</span><span class="w"> </span><span class="n">on-complete</span><span class="p">)</span><span class="w">
</span><span class="n">on-success</span><span class="w"> </span><span class="p">(</span><span class="nf">create-list</span><span class="w"> </span><span class="n">on-success</span><span class="p">)</span><span class="w">
</span><span class="n">on-failure</span><span class="w"> </span><span class="p">(</span><span class="nf">create-list</span><span class="w"> </span><span class="n">on-failure</span><span class="p">)]</span><span class="w">
</span><span class="p">(</span><span class="nf">reify</span><span class="w">
</span><span class="n">Promise</span><span class="w">
</span><span class="p">(</span><span class="nf">future*</span><span class="w"> </span><span class="p">[</span><span class="n">this</span><span class="p">]</span><span class="w"> </span><span class="n">this</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="nf">try-success*</span><span class="w"> </span><span class="p">[</span><span class="n">_</span><span class="w"> </span><span class="n">x</span><span class="p">]</span><span class="w"> </span><span class="p">(</span><span class="nf">try-complete-promise</span><span class="w"> </span><span class="n">d</span><span class="w"> </span><span class="n">v</span><span class="w"> </span><span class="no">:success</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="n">on-complete</span><span class="w"> </span><span class="n">on-success</span><span class="p">))</span><span class="w">
</span><span class="p">(</span><span class="nf">try-failure*</span><span class="w"> </span><span class="p">[</span><span class="n">_</span><span class="w"> </span><span class="n">x</span><span class="p">]</span><span class="w"> </span><span class="p">(</span><span class="nf">try-complete-promise</span><span class="w"> </span><span class="n">d</span><span class="w"> </span><span class="n">v</span><span class="w"> </span><span class="no">:failure</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="n">on-complete</span><span class="w"> </span><span class="n">on-failure</span><span class="p">))</span><span class="w">
</span><span class="p">(</span><span class="nf">status*</span><span class="w"> </span><span class="p">[</span><span class="n">this</span><span class="p">]</span><span class="w"> </span><span class="p">(</span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nf">.isRealized</span><span class="w"> </span><span class="n">this</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nb">first</span><span class="w"> </span><span class="err">@</span><span class="n">v</span><span class="p">)</span><span class="w"> </span><span class="no">:pending</span><span class="w"> </span><span class="p">))</span><span class="w">
</span><span class="n">clojure.lang.IDeref</span><span class="w">
</span><span class="p">(</span><span class="nb">deref</span><span class="w"> </span><span class="p">[</span><span class="n">_</span><span class="p">]</span><span class="w"> </span><span class="p">(</span><span class="nf">.await</span><span class="w"> </span><span class="n">d</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nb">second</span><span class="w"> </span><span class="err">@</span><span class="n">v</span><span class="p">))</span><span class="w">
</span><span class="n">clojure.lang.IBlockingDeref</span><span class="w">
</span><span class="p">(</span><span class="nb">deref</span><span class="w">
</span><span class="p">[</span><span class="n">_</span><span class="w"> </span><span class="n">timeout-ms</span><span class="w"> </span><span class="n">timeout-val</span><span class="p">]</span><span class="w">
</span><span class="p">(</span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nf">.await</span><span class="w"> </span><span class="n">d</span><span class="w"> </span><span class="n">timeout-ms</span><span class="w"> </span><span class="n">java.util.concurrent.TimeUnit/MILLISECONDS</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="nb">second</span><span class="w"> </span><span class="err">@</span><span class="n">v</span><span class="p">)</span><span class="w">
</span><span class="n">timeout-val</span><span class="p">))</span><span class="w">
</span><span class="n">clojure.lang.IPending</span><span class="w">
</span><span class="p">(</span><span class="nf">isRealized</span><span class="w"> </span><span class="p">[</span><span class="n">this</span><span class="p">]</span><span class="w">
</span><span class="p">(</span><span class="nb">zero?</span><span class="w"> </span><span class="p">(</span><span class="nf">.getCount</span><span class="w"> </span><span class="n">d</span><span class="p">)))</span><span class="w">
</span><span class="p">)))</span></code></pre></figure>
<p>The previously announced factory function simply uses <code class="highlighter-rouge">reify</code> to create a new instance. The state is held as closure by the defined <code class="highlighter-rouge">let</code> bindings. Primarily important are <code class="highlighter-rouge">d</code> - a <code class="highlighter-rouge">CountDownLatch</code> to prevent multiple writes and handle synchronization - and <code class="highlighter-rouge">v</code> an atom holding the value. The value of <code class="highlighter-rouge">v</code> will be an array containing the status (the first element) and the supplied value (the second element).</p>
<p>To read the value of the Promise the reified instance also implements <code class="highlighter-rouge">clojure.lang.IDeref</code> and <code class="highlighter-rouge">clojure.lang.IBlockingDeref</code>. This offers the possibility to read the promise by using <code class="highlighter-rouge">deref</code> or the <code class="highlighter-rouge">@</code> reader macro. This works exactly like the original Clojure promise and to be honest the source is also very similar … ;-)</p>
<p>Additionally there is the possibility to pass some listener functions which will be invoked with the value (or cause) when the promise is written. This will be used to the Future realization later on.</p>
<p>If you are still curious have a look at the <a href="2012/06/from-promises-to-futures/">next post</a>.</p>Niclas MeierAt the moment I am experimenting with various techniques to speed up some update queries from our client. One important element of this efforts is making things parallel. E.g. to compute a list update of deltas for the client we need to compare the version of the user on the client and in the database and we need the user data at some other places on the delta computation. So it would be a good idea to initiate the loading of the user data asynchronous right at the beginning of the request processing. While the user data is loaded we can load/compute some other stuff (e.g. we need some data about the competition and the matches). When all the required information is present continue the rest of the the job (e.g. generate some XML or JSON).Suddenly somewhat famous2012-04-17T00:00:00+00:002012-04-17T00:00:00+00:00www.niclas-meier.de/privat/technology/2012/04/17/suddenly-somewhat-famous<p>Thanks to a former colleague, I received word that my post about my first <a href="/2012/04/first-hundred-days-of-clojure/">100 days of working with Clojure</a> scored Number 3. on the <a href="http://news.ycombinator.com/">Hacker News</a>.</p>
<p>I just wanted to thank all of you for the interest and I really hope my english is not that worse. Thanks to some attentive readers, I where able to fix some glitches in some of the examples or the spelling.</p>
<p>Due to fortunate circumstances I had some time to setup <a href="/2011/04/nginxing-around-with-wordpress-2/">my blog on a new platform</a> almost exactly a year ago. Now using <a href="http://wiki.nginx.org">Nginx</a> with <a href="http://wordpress.org/">Wordpress</a>. Witch performed remarkably well.<a id="more"></a><a id="more-152"></a>If your are interested here are some stats:</p>
<p><img src="/assets/localhost-nginx_status-day.png" alt="Localhost nginx status day" /></p>
<p>Starting with about 200 concurrent connections to the my Nging web server, which are 66 times the average over the last year. That’s nice because I saw some setups which would have died due to the load. Thanks to <a href="https://www.xing.com/profile/Lukas_Loesche2">the guy who introduced me into Nginx</a>. We were using it at <a href="www.scoyo.com">scoyo</a> and <a href="http://www.gamigo.com">gamigo</a> and this proved that it was a good decision - if I had any doubts left.</p>
<p>As a matter of fact, that worked so well, that it didn’t even had an impact on the CPU load: <img src="/assets/localhost-load-day.png" alt="Localhost load day" /> <img src="/assets/localhost-load-month1.png" alt="Localhost load month" /></p>
<p>Or even the memory usage:</p>
<p><img src="/assets/localhost-memory-day.png" alt="Localhost memory day" /></p>
<p>I guess making all the content static using <a href="http://wordpress.org/extend/plugins/wp-super-cache/">WP SuperCache</a> and the <a href="http://wiki.nginx.org/HttpRewriteModule">HTTP rewrite module</a> was a good idea.</p>Niclas MeierThanks to a former colleague, I received word that my post about my first 100 days of working with Clojure scored Number 3. on the Hacker News.First hundred days of Clojure2012-04-13T00:00:00+00:002012-04-13T00:00:00+00:00www.niclas-meier.de/clojure/2012/04/13/first-hundred-days-of-clojure<p>In politics you have a one hundred days period of grace. In December I started to work with <a href="http://www.clojure.org">Clojure</a>, so I guess it’s time to have a clojure look.</p>
<p>In the last couple of years I encountered a couple of different languages. At <a href="http://www.scoyo.de">scoyo</a> it was <a href="http://www.adobe.com/devnet/actionscript.html">ActionScript</a> and <a href="http://flex.org/">Flex</a> from Adobe. I had the honor to lead a team of great engineers, but with good people you don’t get close enough to the real stuff. At <a href="http://www.gamigo.de">gamigo</a> the new technology was <a href="http://www.php.net">PHP</a>. But for the last thirteen years <a href="http://www.java.com/">Java</a> was always a big part of my career… At my new <a href="http://www.herolabs.com">job</a> the new kid on the block is <a href="http://www.clojure.org">Clojure</a> - a LISP dialect based on Java VM. Clojure was created by <a href="http://twitter.com/#!/richhickey">Rich Hickey</a> and first appeared in 2007.</p>
<p>When we started to work on our new product I said to my boss and my colleagues: In Java I know how a state of the art application has to look like, but I have no clue how to build a Clojure application. Fortunately a colleague had some Common LISP experience and already a closer look on the stack we should use. At the moment our application uses this tools and libraries. I guess I will come back to some of them later on.</p>
<ul>
<li><a href="https://github.com/technomancy/leiningen">Leiningen</a> (aka <a href="https://github.com/technomancy/leiningen">lein</a>) - The build and dependency managmement tool for Clojure.</li>
<li><a href="http://webnoir.org/">Noir - A nice library to build websites/-services in Clojure.</a></li>
<li><a href="https://github.com/aboekhoff/congomongo">CongoMongo</a> and <a href="http://www.mongodb.org/">mongoDB</a> - The database driver and the database we use.</li>
</ul>
<p><strong>My first days with Clojure</strong> When you are coming from a language like Java you are used to some things like: <em>Structure everything</em> - Static typing is your friend, so smack every data you have into a class. I guess most Java Enterprise application use a significant part of their computing time with copying references from one bean to another. When you use Clojure you simply don’t do this and sometimes is still hard for me not to worry about it. But this gives you a whole new level of freedom, you may use e.g. <code class="highlighter-rouge">(merge …)</code> to simply combine two of you entities into one larger one.</p>
<p><em>Lots of ceremony</em> - When you work in Java you create a huge amount of classes. The longer you work with Java the more inner and anonymous classes you tend to use. From these classes ninety percent are stupid bean classes with some attributes plus a setter and a getter per attribute and if you work with experts a <code class="highlighter-rouge">hashCode</code> and <code class="highlighter-rouge">equals</code> method. The remaining ten (or so) percent of you classes are services or stuff like that, who transform beans in different beans or XML, JSON, String, stuff like that. In clojure you put your structured data in maps - that’s it. Due to the fact that all of closures datatypes are immutable you build some functions who do the same like the services and you are done. But instead of shuffling data from one bean to another you may use <code class="highlighter-rouge">(assoc …)</code>, <code class="highlighter-rouge">(dissoc …)</code>, <code class="highlighter-rouge">(select-keys …)</code> or <code class="highlighter-rouge">(merge …)</code> to efficiently create modified or new entities for the business logic or output generation.</p>
<p>But I can say, you’ll get used to this really fast and so I had programmed my first JSON web service very fast. But as always stuff get a little more complex …</p>
<p>The first hurdle I had to climb where collections. Unlike the good old <code class="highlighter-rouge">for (…;…;…) {…}</code> loop in Java the <code class="highlighter-rouge">(for …)</code> list comprehension in Clojure is a beast. The first thing to do, is to realize that despite the name these two guys are totally different. Staring with the fact that the Clojure <code class="highlighter-rouge">(for …)</code> will be evaluated lazily, so if you want a similar evaluation behavior that the Java version you should try <code class="highlighter-rouge">(doseq …)</code>. A standard pattern in Java is to define some nice result variables and then iterate with <code class="highlighter-rouge">for</code> through the collection changing your variables. After the loop your result is in the variable ready for further action. But you cannot do this in Clojure because you don’t have variables. You have <code class="highlighter-rouge">let</code> bindings which look somewhat similar but they are not mutable like your datatypes. So the <code class="highlighter-rouge">(for …)</code> list comprehension works a little bit different. It takes a collection and returns a collection of same or shorter length. In the function body you can compute the new value of the new collection <code class="highlighter-rouge">for</code> returns. You even may skip values with the <code class="highlighter-rouge">:when <val></val></code>option. A very similar function is the <code class="highlighter-rouge">(map …)</code> function which also maps a value to a new value and returns a lazy collection of these new values.</p>
<p><strong>The first pitfalls</strong> Don’t get me wrong, the Clojure collection library is great, but you need some time to get used to it. A nasty side effect was, that shortly after I though: “Yes, I am king of all collections!” you stumble over oddities like the different behavior in the <code class="highlighter-rouge">(conj …)</code> function. In Clojure you have these functions <code class="highlighter-rouge">(cons …)</code> which adds an element in front of a sequence and <code class="highlighter-rouge">(conj …)</code> which append an element at the end of a sequence. At least this is what I though. In fact it does append an element if the sequence is a vector, if it is a list it behaves like <code class="highlighter-rouge">(cons …)</code>. You can spend some hours before you find this in the doc.</p>
<p>Another nice one - showing the power and danger - was related to Facebook and Mongo DB. We query some data from Facebook and store them into the a Mongo DB. The process only does some restructuring of the data but doesn’t touch the values before we insert them into the database. To optimize the Facebook querying we switched from a hundered calls of the Facebook <a href="http://developers.facebook.com/docs/reference/api/">graph API</a> to three <a href="http://developers.facebook.com/docs/reference/fql/">FQL</a> queries. This worked really well. But Facebook has a minor difference in the results. In the graph API the IDs are strings in the FQL result set IDs are numbers. The whole process worked percieved well for a while but the I noticed that the queries on the Mongo DB delivered not enough results. What happened? The <a href="https://github.com/mmcgrana/clj-json">Clojure JSON library</a> we used started to deliver numbers instead of strings and we where storing them into the Mongo DB. This worked without changing a line of code, which is nice if you want it. But the queries on the Mongo DB where still looking for strings …</p>
<p><strong>Thinking more, coding less</strong> I guess this says it all. The first weeks of Clojure where unsatisfying. When coded in Java I had lots of code: Classes, XML files, etc. But in Clojure the result of the day were sometimes two functions and a macro. First that occurred very strange to me but I realized that I had to code less to get the same result. And these two functions and the macro where very lean and elegant and solved problems worth a couple of Java classes. This was the first time I really realized the power of Clojure and the different approach.</p>
<p><strong>Homoiconic or code is data</strong> If you start with Clojure you soon stumble about this whole code is data stuff. This is was homoiconic in the end means, that the syntax of your language can be written in data structures of your language. If you look at:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="k">for</span><span class="w"> </span><span class="p">[</span><span class="n">i</span><span class="w"> </span><span class="p">(</span><span class="nb">range</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="mi">100</span><span class="p">)]</span><span class="w"> </span><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="n">i</span><span class="p">))</span></code></pre></figure>
<p>You can read it as list with the first element <code class="highlighter-rouge">for</code> as symbol, a vector of further forms as second element and another list as third elements. The term for this is forms.</p>
<p>So doing this all LISPs have an extremely small syntax. At least that is what they are telling you. But I believe that the LISP/Clojure gurus (e.g. <a href="http://groups.csail.mit.edu/mac/users/gjs/">Gerald Jay Sussman</a> or <a href="http://twitter.com/#!/richhickey">Rich Hickey</a>) are cheating us a little bit.</p>
<p>I guess I have to agree that the languages it self has very little syntax, but the language is shifting the complexity into the libraries. If you have a look at the <code class="highlighter-rouge">(for …)</code> comprehension. You saw the simple variant a few lines ago but you can also du this:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="k">for</span><span class="w"> </span><span class="p">[</span><span class="n">i</span><span class="w"> </span><span class="p">(</span><span class="nb">range</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="mi">100</span><span class="p">)</span><span class="w"> </span><span class="no">:let</span><span class="w"> </span><span class="p">[</span><span class="n">f</span><span class="w"> </span><span class="p">(</span><span class="nf">format</span><span class="w"> </span><span class="s">"%03d"</span><span class="w"> </span><span class="n">i</span><span class="p">)]</span><span class="w"> </span><span class="no">:when</span><span class="w"> </span><span class="p">(</span><span class="nf">odd?</span><span class="w"> </span><span class="n">i</span><span class="p">)]</span><span class="w"> </span><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="n">f</span><span class="p">))</span></code></pre></figure>
<p>This nice little feller prints all odd numbers from 1 to 100 formatted with leading zeros. But instead of using <code class="highlighter-rouge">(let …)</code> or filtering the range collection I used some not so well documented options of the binding vector of the <code class="highlighter-rouge">(for …)</code>. So how does it work? First <code class="highlighter-rouge">for</code> works internally different because it is a Clojure special form, but there are several macros - yes and I also wrote some - who do the same. If you are interested you may look at the <code class="highlighter-rouge">(defpage …)</code> macro, which does also some black magic. You usually use macros to do this type of things but you also may use functions (to a certain degree). So what there macros is to analyze the vector on the first position of the parameter list. In the <code class="highlighter-rouge">for</code> case you would iterate through the list and if you encounter a keyword (the things with <code class="highlighter-rouge">:</code> like <code class="highlighter-rouge">:let</code>) instead of a symbol you do some extra special stuff.</p>
<p>Usually you use this to define your own <a href="http://en.wikipedia.org/wiki/Domain-specific_language">domain specific language (DSL)</a> in Clojure. This is what makes the solution of your problems very elegant because you make your problem related language where you can model and solve the problem very easily.</p>
<p>But also the Clojure standard functions are using this heavily and are so hinding a lot of implicit syntax form the user who depends on documentation and examples to discover how this works. For me this was sometimes very frustrating because very helpful options (e.g. <code class="highlighter-rouge">:when <val></val></code>) are sometimes hard to find.</p>
<p><strong>Java interoperability or it’s safe as long as you are in object land</strong> Clojure lives (at least in the version we use) on the Java VM - there is a .NET variant and also with <a href="https://github.com/clojure/clojurescript">ClojureScript</a> a variant which runs as Java-Script. So if you are on the JVM you want to use some the great libraries already exist in the Java ecosystem. If you e.g. look at <a href="https://github.com/mmcgrana/clj-json">clj-json</a> a very fast JSON library for Clojure. It depends on the <a href="http://jackson.codehaus.org/">Jackson</a> library and provides only a thin layer upon it.</p>
<p>To use Java objects and libraries in Clojure is very simple, the messy part only begins if you want to use primitive types. These are supported but its no fun anymore. A word of warning on working with Java objects. You are leaving the zone of immutability when you work with java objects. If you get yourself a <code class="highlighter-rouge">java.util.Date</code> you can mutate the date like you can do it in Java and mess up big time.</p>
<p>A very nice interop feature are <code class="highlighter-rouge">(deftype …)</code>, <code class="highlighter-rouge">(proxy …)</code> or <code class="highlighter-rouge">(reify …)</code>. You can use this macros to create or extend java Objects. Have a look at this:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="nb">proxy</span><span class="w"> </span><span class="p">[</span><span class="n">com.rabbitmq.client.DefaultConsumer</span><span class="p">]</span><span class="w"> </span><span class="p">[(</span><span class="nf">channel</span><span class="w"> </span><span class="n">connection</span><span class="p">)]</span><span class="w">
</span><span class="p">(</span><span class="nf">handleDelivery</span><span class="w"> </span><span class="p">[</span><span class="n">tag</span><span class="w"> </span><span class="n">envelope</span><span class="w"> </span><span class="n">prop</span><span class="w"> </span><span class="n">body</span><span class="p">]</span><span class="w"> </span><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="n">body</span><span class="p">))</span><span class="w">
</span><span class="p">)</span></code></pre></figure>
<p>This tiny piece of code extends the <code class="highlighter-rouge">DefaultConsumer</code> and overrides the <code class="highlighter-rouge">handleDelivery</code> method. You can hand over the received object to a channel as a consumer and integrate directly into the Rabbit MQ Java driver.</p>
<p><strong>Some words to the ecosystem</strong> Finally I want to have some word about some nice tools, library and stuff on Clojure ecosystem.</p>
<p><em>Leiningen</em> - A very popular build tool which also offers dependency resolution. My experience with Leiningen is, that it works out of the box in most cases. Unfortunately it tends to hang once or twice a day, I guess it tries to figure out if it must update some dependencies when it’s gone.</p>
<p><em><a href="https://github.com/marick/Midje">Midje</a></em> - A very powerful Clojure testing framework which includes facilities to easily mock every function. This makes writing even complex tests very easy.</p>
<p><em><a href="http://avout.io/">avout</a></em> - This one is nice. It integrates tools like Clojure atoms and refs into Zookeeper (or some other coordination engines). It also offers distributed locks. Very easy to use and to build distributed cluster coordination.</p>
<p><em><a href="http://palletops.com/">pallet</a></em> - A library who offers infrastructure management (e.g. for Amazon EC2) in clojure. You also may remotely install stuff or start and stop services. Great for dev-ops, but the documentation could be better and at the moment only Clojure 1.2 is supported. It took me some effort to get it working but when you know how to use it very powerful.</p>
<p><strong>What suc</strong>s?** In this post I told you various thing about the power and beauty of Clojure but where are some things that I don’t like:</p>
<p><em>No informations on data structures</em> - I already told you that in Java you write a lot of classes to structure you data. On one hand side this is a lot of overhead, but on the other hand side its also a kind of always correct and available documentation. If you have to get into some foreign code that fetches data from a database it’s very hard to discover what is the right data structure and which type are the right ones. This also makes changes very difficult.</p>
<p><em>Dynamic invocation</em> - Functions are compiled and invoked when whey are invoked. You don’t have any chance of previous checking if the function call is possible or not. This is extremely annoying when you change the signature of a function e.g. you remove a parameter. If you miss a spot where the function is called you get an error. This gets even worse if you use restructuring and variable parameter lists. You can do something like this:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">fetch</span><span class="w"> </span><span class="p">[</span><span class="n">database</span><span class="w"> </span><span class="o">&</span><span class="w"> </span><span class="p">{</span><span class="no">:keys</span><span class="w"> </span><span class="p">[</span><span class="no">:where</span><span class="p">]}]</span><span class="w"> </span><span class="err">…</span><span class="p">)</span></code></pre></figure>
<p>You can call this function with or without the <code class="highlighter-rouge">:where</code> parameter. This looks like this:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="n">all</span><span class="w"> </span><span class="p">(</span><span class="nf">fetch</span><span class="w"> </span><span class="n">my-database</span><span class="p">)</span><span class="w">
</span><span class="nb">some</span><span class="w"> </span><span class="p">(</span><span class="nf">fetch</span><span class="w"> </span><span class="n">my-database</span><span class="w"> </span><span class="no">:where</span><span class="w"> </span><span class="p">{</span><span class="no">:height</span><span class="w"> </span><span class="p">{</span><span class="no">:$gt</span><span class="w"> </span><span class="mi">42</span><span class="p">}})]</span><span class="w">
</span><span class="err">…</span><span class="w">
</span><span class="p">)</span></code></pre></figure>
<p>This is every powerful when you are creating flexible functions for DSLs but can cause very nasty side effects when you refactor these functions and change the signature.</p>
<p><em>No bootstrapping</em> - In the Java world you have dependency injection frameworks like <a href="http://code.google.com/p/google-guice/">Google Guice</a> or the <a href="http://www.springsource.org/">Spring framework</a>. These frameworks are very helpful when you want to invert the control flow. The services you write do not have to be aware how to find the suitable services and resources. You are centralizing the bootstrapping process in one location (e.g. classes in Guice or XML in Spring). Clojure does not offer this facilities - you don’t have objects to store state, not even configuration state. The solution we choose is like a typical <a href="http://java.sun.com/blueprints/corej2eepatterns/Patterns/ServiceLocator.html">pattern</a>. One namespace provides functions to access the resources (e.g. database connections, etc.) and there functions can be called from every where. But this has also the same drawbacks than the classical service locator. You don’t have control about timing and the class is like a spider in the web called from every where.</p>
<p><em>Clojure 1.2</em> - The current stable release is Clojure 1.3. Unfortunately some libraries we wanted to use where still based on Clojure 1.2. In general this could work but be had some strange effects on some libraries which took us a while to discover that it was a Cojure 1.2/1.3 problem. A clear strategy how to deal with this isn’t there yet, so all you can do is try and error.</p>
<p><strong>Loose ends</strong> I didn’t talk about stuff like STM (software transactional memory), Agents, Refs or Atoms. Mostly because they worked in the first place for me, or never even tried them. There are also a couple of nice library in the Clojure eco system which make your life as developer very easy. I recommend, that you have closer look yourself.</p>
<p><strong>… and in the end</strong> To be honest, I had a tough couple of days in the last days. We are are in a stage where our prototype application grows into a real world application and we want to establish structures which makes maintenance easier. Unfortunately this is not so easy as we expected.</p>
<p>So my conclusions right now:</p>
<ul>
<li>Clojure is fun - I really like it to write two line functions that kick ass.</li>
<li>Clojure does not need to hide - The eco system is growing fast and has some interesting libraries</li>
<li>The lack of tooling is sometimes a pain. There are times where <code class="highlighter-rouge">clojure.tools.logging/spy</code> or <code class="highlighter-rouge">println</code> are not enough. The absence of a debugger is sometimes a real productivity black hole.</li>
<li>Due to the dynamic nature you’ll have to write a lot of tests to be able to perform refactorings. Only with a lot of tests you can make sure that the code will work afterwards.</li>
<li>Document your data structures to make sure that other developers have a chance to work in your code. I made some good experiences with a namespace that containes accessor functions. You can validate there functions with unit tests and the other guys in your team can easily see which values the can expect.</li>
</ul>
<p>Nobody can say right now, how our little adventure with Clojure will end. But for the first release of out product our backend services will be based on Clojure. The time will tell how the maintainability and the performance of the backend will develop.</p>
<p>Right now I am not that confident that the solution will be good enough to be in service for a long time. But maybe Clojure surprises me … again …</p>Niclas MeierIn politics you have a one hundred days period of grace. In December I started to work with Clojure, so I guess it’s time to have a clojure look.The curl cheat sheet to myself2011-12-07T00:00:00+00:002011-12-07T00:00:00+00:00www.niclas-meier.de/technology/software%20development/rest/2011/12/07/the-curl-cheat-sheet-to-myself<p>It has been not even a full week building REST style interfaces for the <a href="http://www.playmakerstudio.com/">Playmaker Studio</a> mobile platform and I am very annoyed of all the <code class="highlighter-rouge">curl</code> guides how to access your REST API with <code class="highlighter-rouge">curl</code>. So here is my cheat sheet on this topic. Be aware that the I will update this frequently.</p>
<h2 id="common-use-cases">Common use cases</h2>
<h3 id="getting-data-from-a-url">GETting data from a URL</h3>
<p>This is a very simple use case just use:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">curl https://graph.facebook.com/740999649
<span class="o">{</span><span class="s2">"id"</span>:<span class="s2">"740999649"</span>,<span class="s2">"name"</span>:<span class="s2">"Niclas Meier"</span><span class="o">}</span></code></pre></figure>
<p><code class="highlighter-rouge">curl</code> will automatically use the HTTP method <code class="highlighter-rouge">GET</code>.</p>
<h3 id="postting-updates">POSTting updates</h3>
<p>In the REST dictionary the HTTP method <code class="highlighter-rouge">POST</code> represents the verb ‘update’. So updating data implies a <code class="highlighter-rouge">POST</code> request to an URL.</p>
<h4 id="basic">Basic</h4>
<p>Let’s have a look at the basics, I added some (revised / half the original) debugging output by supplying the <code class="highlighter-rouge">-v</code> option.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">curl -v -X POST http://localhost:8080/1/stuff/23f96c6bd4c87910c.json -d <span class="s1">'name=Niclas'</span>
<span class="gp">> </span>POST /1/stuff/23f96c6bd4c87910c.json HTTP/1.1
<span class="gp">> </span>Host: localhost:8080
<span class="gp">> </span>Accept: <span class="k">*</span>/<span class="k">*</span>
<span class="gp">> </span>Content-Length: 11
<span class="gp">> </span>Content-Type: application/x-www-form-urlencoded
>
< HTTP/1.1 200 OK
< Content-Type: text/plain; <span class="nv">charset</span><span class="o">=</span>utf-8
< Content-Length: 25
<
Stuff: 23f96c6bd4c87910c</code></pre></figure>
<p>There are some options involved:</p>
<ul>
<li><code class="highlighter-rouge">-v</code> added the request/response header for debugging</li>
<li><code class="highlighter-rouge">-X</code> specified the request method (<code class="highlighter-rouge">POST</code>) to use.</li>
<li><code class="highlighter-rouge">-d</code> added some data in the request (<code class="highlighter-rouge">POST</code>) body.</li>
</ul>
<p>So the <code class="highlighter-rouge">'name=Niclas'</code> was sent to the server using the request body, but <code class="highlighter-rouge">curl</code> does not display the request body using the <code class="highlighter-rouge">-v</code> option.</p>
<h4 id="using-a-json-body">Using a JSON body</h4>
<p>So this is nice, when you want to post key-value pairs to your server, but many applications (e.g. <a href="http://couchdb.apache.org/">CouchDB</a> want to post JSON objects to the server. The initial guess</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">curl -v -X POST http://localhost:8080/1/stuff/23f96c6bd4c87910c.json -d <span class="s1">'{"name":"Niclas"}'</span>
Stuff: 23f96c6bd4c87910c, JSON body:</code></pre></figure>
<p>does not work. That’s where the content type comes in. If you post the JSON whiteout supplying the proper content type <code class="highlighter-rouge">curl</code> and/or the server will try to use <code class="highlighter-rouge">application/x-www-form-urlencoded</code><a href="http://www.w3.org/TR/html4/interact/forms.html"><sup>*</sup></a> content type. This is the same content type your browser uses when submitting a form to a web server. Doesn’t sound bad, does it?</p>
<p>But it’s bad because the JSON will be ignored/dropped/whatever and the server is unable to understand the request. To solve the problem use:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">curl -H <span class="s2">"Content-Type: application/json"</span> -X POST http://localhost:8080/1/stuff/23f96c6bd4c87910c.json -d <span class="s1">'{"name":"Niclas"}'</span>
Stuff: 23f96c6bd4c87910c, JSON body: <span class="o">{</span><span class="s2">"name"</span>:<span class="s2">"Niclas"</span><span class="o">}</span></code></pre></figure>
<p>So these options are involved</p>
<ul>
<li><code class="highlighter-rouge">-X</code> specified the request method (<code class="highlighter-rouge">POST</code>) to use.</li>
<li><code class="highlighter-rouge">-H</code> added the header <code class="highlighter-rouge">Content-Type: application/json</code> to the request.</li>
<li><code class="highlighter-rouge">-d</code> defined the data used for the request body (<code class="highlighter-rouge"><span class="p">{</span><span class="nt">"name"</span><span class="p">:</span><span class="s2">"Niclas"</span><span class="p">}</span></code>).</li>
</ul>
<h4 id="a-json-file-as-body">A JSON file as body</h4>
<p>So far so good. But if you have huge JSON requests, you don’t want to type all the JSON into the console. You may use the <code class="highlighter-rouge">@</code> symbol in conjunction with the <code class="highlighter-rouge">-d</code> option so <code class="highlighter-rouge">curl</code> will read the request body contents from a file. This may look like this.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">curl -H <span class="s2">"Content-Type: application/json"</span> -X POST http://localhost:8080/1/stuff/23f96c6bd4c87910c.json -d @some-huge-request.json
Stuff: 23f96c6bd4c87910c, JSON body: <span class="o">{</span><span class="s2">"name"</span>:<span class="s2">"Niclas"</span>, ..., <span class="s2">"foo"</span>:<span class="s2">"Bar"</span><span class="o">}</span></code></pre></figure>
<p>A nice trick is, that if you use the <code class="highlighter-rouge">-d</code> with <code class="highlighter-rouge">@-</code>. <code class="highlighter-rouge">curl</code> will read the request body from the standard input. So you can do stuff like this:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">cat some-huge-request.json | curl -H <span class="s2">"Content-Type: application/json"</span> -X POST http://localhost:8080/1/stuff/23f96c6bd4c87910c.json -d @-
Stuff: 23f96c6bd4c87910c, JSON body: <span class="o">{</span><span class="s2">"name"</span>:<span class="s2">"Niclas"</span>, ..., <span class="s2">"foo"</span>:<span class="s2">"Bar"</span><span class="o">}</span></code></pre></figure>
<h3 id="putting-stuff">PUTting stuff</h3>
<p>Is much like <code class="highlighter-rouge">POST</code>, so if you can <code class="highlighter-rouge">POST</code> you can <code class="highlighter-rouge">PUT</code>. One nasty little difference is, that you don’t have to supply a <code class="highlighter-rouge">Content-Type</code> HTTP header for <code class="highlighter-rouge">PUT</code>. <code class="highlighter-rouge">curl</code> does not use <code class="highlighter-rouge">application/x-www-form-urlencoded</code> as content type for <code class="highlighter-rouge">PUT</code>. Lost an hour (or so), because I did some <code class="highlighter-rouge">PUT</code> stuff before switching to <code class="highlighter-rouge">POST</code>.</p>
<h2 id="debugging">Debugging</h2>
<h4 id="detailed-information">Detailed information</h4>
<p>The key to get almost every information by supplying the <code class="highlighter-rouge">-v</code> option to curl.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">curl -v https://graph.facebook.com/740999649
<span class="k">*</span> About to connect<span class="o">()</span> to graph.facebook.com port 443 <span class="o">(</span><span class="c">#0)</span>
<span class="k">*</span> Trying 66.220.147.38... connected
<span class="k">*</span> Connected to graph.facebook.com <span class="o">(</span>66.220.147.38<span class="o">)</span> port 443 <span class="o">(</span><span class="c">#0)</span>
<span class="k">*</span> SSLv3, TLS handshake, Client hello <span class="o">(</span>1<span class="o">)</span>:
<span class="k">*</span> SSLv3, TLS handshake, Server hello <span class="o">(</span>2<span class="o">)</span>:
<span class="k">*</span> SSLv3, TLS handshake, CERT <span class="o">(</span>11<span class="o">)</span>:
<span class="k">*</span> SSLv3, TLS handshake, Server finished <span class="o">(</span>14<span class="o">)</span>:
<span class="k">*</span> SSLv3, TLS handshake, Client key exchange <span class="o">(</span>16<span class="o">)</span>:
<span class="k">*</span> SSLv3, TLS change cipher, Client hello <span class="o">(</span>1<span class="o">)</span>:
<span class="k">*</span> SSLv3, TLS handshake, Finished <span class="o">(</span>20<span class="o">)</span>:
<span class="k">*</span> SSLv3, TLS change cipher, Client hello <span class="o">(</span>1<span class="o">)</span>:
<span class="k">*</span> SSLv3, TLS handshake, Finished <span class="o">(</span>20<span class="o">)</span>:
<span class="k">*</span> SSL connection using RC4-MD5
<span class="k">*</span> Server certificate:
<span class="k">*</span> subject: <span class="nv">C</span><span class="o">=</span>US; <span class="nv">ST</span><span class="o">=</span>California; <span class="nv">L</span><span class="o">=</span>Palo Alto; <span class="nv">O</span><span class="o">=</span>Facebook, Inc.; <span class="nv">CN</span><span class="o">=</span><span class="k">*</span>.facebook.com
<span class="k">*</span> start date: 2010-01-13 00:00:00 GMT
<span class="k">*</span> expire date: 2013-04-11 23:59:59 GMT
<span class="k">*</span> common name: <span class="k">*</span>.facebook.com <span class="o">(</span>matched<span class="o">)</span>
<span class="k">*</span> issuer: <span class="nv">C</span><span class="o">=</span>US; <span class="nv">O</span><span class="o">=</span>DigiCert Inc; <span class="nv">OU</span><span class="o">=</span>www.digicert.com; <span class="nv">CN</span><span class="o">=</span>DigiCert High Assurance CA-3
<span class="k">*</span> SSL certificate verify ok.
<span class="gp">> </span>GET /740999649 HTTP/1.1
<span class="gp">> </span>User-Agent: curl/7.21.4 <span class="o">(</span>universal-apple-darwin11.0<span class="o">)</span> libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
<span class="gp">> </span>Host: graph.facebook.com
<span class="gp">> </span>Accept: <span class="k">*</span>/<span class="k">*</span>
>
< HTTP/1.1 200 OK
< Access-Control-Allow-Origin: <span class="k">*</span>
< Cache-Control: private, no-cache, no-store, must-revalidate
< Content-Type: text/javascript; <span class="nv">charset</span><span class="o">=</span>UTF-8
< ETag: <span class="s2">"f89bfe40c87e487ae641d427c96746b226a91f3c"</span>
< Expires: Sat, 01 Jan 2000 00:00:00 GMT
< Pragma: no-cache
< X-FB-Rev: 482216
< X-FB-Server: 10.36.50.102
< X-Cnection: close
< Date: Wed, 07 Dec 2011 13:14:19 GMT
< Content-Length: 183
<
<span class="k">*</span> Connection <span class="c">#0 to host graph.facebook.com left intact</span>
<span class="k">*</span> Closing connection <span class="c">#0</span>
<span class="k">*</span> SSLv3, TLS alert, Client hello <span class="o">(</span>1<span class="o">)</span>:
<span class="o">{</span><span class="s2">"id"</span>:<span class="s2">"740999649"</span>,<span class="s2">"name"</span>:<span class="s2">"Niclas Meier"</span>,<span class="s2">"first_name"</span>:<span class="s2">"Niclas"</span>,<span class="s2">"last_name"</span>:<span class="s2">"Meier"</span>,<span class="s2">"link"</span>:<span class="s2">"http:</span><span class="se">\/\/</span><span class="s2">www.facebook.com</span><span class="se">\/</span><span class="s2">people</span><span class="se">\/</span><span class="s2">Niclas-Meier</span><span class="se">\/</span><span class="s2">740999649"</span>,<span class="s2">"gender"</span>:<span class="s2">"male"</span>,<span class="s2">"locale"</span>:<span class="s2">"de_DE"</span><span class="o">}</span></code></pre></figure>
<p>But for may occasions that is simply too much, so check out the <code class="highlighter-rouge">-i</code> option.</p>
<h4 id="header-information">Header information</h4>
<p>Often it’s enough to get only the response headers for debugging. To get only these headers just use the <code class="highlighter-rouge">-i</code> option in <code class="highlighter-rouge">curl</code>.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">curl -i https://graph.facebook.com/740999649
HTTP/1.1 200 OK
Access-Control-Allow-Origin: <span class="k">*</span>
Cache-Control: private, no-cache, no-store, must-revalidate
Content-Type: text/javascript; <span class="nv">charset</span><span class="o">=</span>UTF-8
ETag: <span class="s2">"f89bfe40c87e487ae641d427c96746b226a91f3c"</span>
Expires: Sat, 01 Jan 2000 00:00:00 GMT
Pragma: no-cache
X-FB-Rev: 482216
X-FB-Server: 10.42.67.29
X-Cnection: close
Date: Wed, 07 Dec 2011 13:18:24 GMT
Content-Length: 183
<span class="o">{</span><span class="s2">"id"</span>:<span class="s2">"740999649"</span>,<span class="s2">"name"</span>:<span class="s2">"Niclas Meier"</span>,<span class="s2">"first_name"</span>:<span class="s2">"Niclas"</span>,<span class="s2">"last_name"</span>:<span class="s2">"Meier"</span>,<span class="s2">"link"</span>:<span class="s2">"http:</span><span class="se">\/\/</span><span class="s2">www.facebook.com</span><span class="se">\/</span><span class="s2">people</span><span class="se">\/</span><span class="s2">Niclas-Meier</span><span class="se">\/</span><span class="s2">740999649"</span>,<span class="s2">"gender"</span>:<span class="s2">"male"</span>,<span class="s2">"locale"</span>:<span class="s2">"de_DE"</span><span class="o">}</span></code></pre></figure>Niclas MeierIt has been not even a full week building REST style interfaces for the Playmaker Studio mobile platform and I am very annoyed of all the curl guides how to access your REST API with curl. So here is my cheat sheet on this topic. Be aware that the I will update this frequently.Making a new move - again2011-12-07T00:00:00+00:002011-12-07T00:00:00+00:00www.niclas-meier.de/allgemein/privat/2011/12/07/making-a-new-move-again<p>After been almost 10 years at <a href="http://www.sinnerschrader.de/">SinnerSchrader</a> it’s very confusing for me to be on the move again. On the 1<sup>st</sup> of December I joined <a href="http://www.playmakerstudio.com/">Playmaker Studio</a> after being at <a href="http://www.gamigo.de">gamigo</a> for 18 month. You now may ask why? I don’t want to explain every thought that led to the decision, but I can say I enjoyed my job at gamigo. I learned damn much, especially stuff I never anticipated (I am not saying that I didn’t wanted to learn most of it ;-)). I had a great team making a giant leap in the software development processes, nice colleagues and we brought the company a huge step ahead.</p>
<p>But it was more than about time to get a new direction in working life. At gamigo my job drifted to a pure management position and I was nudging around <a href="http://www.atlassian.com/de/software/jira/overview">JIRA tickets</a> and <a href="http://office.microsoft.com/de-de/excel/">Excel files</a> (okay some of them where <a href="https://docs.google.com/">Google Docs</a> ;-)) the whole day. But now I joined the Ranks of technicians again !</p>
<p>Hopefully I will have some time for reviving this blog. For now I am trying to learn some new stuff - like <a href="http://clojure.org/">Clojure</a>, <a href="http://webnoir.org/">Noir</a>, <a href="http://www.heroku.com/">Heroku</a> or <a href="http://www.mongodb.org/">MongoDB</a> for my new job. For me, this sounds like lots of fun!</p>Niclas MeierAfter been almost 10 years at SinnerSchrader it’s very confusing for me to be on the move again. On the 1st of December I joined Playmaker Studio after being at gamigo for 18 month. You now may ask why? I don’t want to explain every thought that led to the decision, but I can say I enjoyed my job at gamigo. I learned damn much, especially stuff I never anticipated (I am not saying that I didn’t wanted to learn most of it ;-)). I had a great team making a giant leap in the software development processes, nice colleagues and we brought the company a huge step ahead.