I have been playing around with LINQ the last couple of weeks. Really nice stuff to work with, it changes the way you work with sets of data and the process of transforming them into other sets of data. But I don't want to focus on the utility of LINQ here, but on the underlying paradigm shift that we might encounter and that I have been thinking about lately: drawing the analogy between application tiers and a supply chain, and giving the power back to the interface tier.

Typically it is considered good practice to split up your application in multiple tiers. This will enhance your reusability, flexibility and scalability. All good qualities to strive for obviously. But it comes at a cost: all tiers combined will be larger in code size and more complex to enable strict seperation of concerns. For example, we all have created applications in which in the eventhandler of a button you directly prepared a SQL statement query string and sended it directly to the database server at hand. Not a good practice in many ways, but very straightforward and effective. Now consider you introduce a method where the SQL preparation and subsequent database fetching takes place. This method will be more flexible and can be used from multiple locations (maybe two or three other button click eventhandlers) in your application code. The multi purpose versatility of the method is reflected in the method itself by adding some if or switch statements. A bit more complex, but no real problem here.

We take the next step and really seperate the interface and the data access layer. So we setup a seperate project which only concerns itself with fetching and updating the data in the database. It is unaware of the front-end and therefore creates many abstractions that will be used by the front-end. This might lead to some redundancy. For example, it might check data that is already checked in the front-end, or it might make calls to the database to retrieve data that was already retrieved before, but of which the data layer was unaware. This might be a slightly bigger problem, the performance hit here to re-fetch data from the database can be considerable. But, CPU and memory is cheap, so you don't really need to worry about it these days.

The applications grows bigger and bigger, the decision is made to implement some kind of middleware, or a business layer at the boundary between your interface tier and your data tier. This middle tier must be quite flexible, because it abstracts away both the data retrieval and the interface functionality. Since there is little knowledge of this available, the middle tier can do nothing else than just work at the level of the lowest common denominator: whole business entities, perform all validations, all authentication, authorization, etc. Good practice in itself because it is the only way you can reliably say that your business logic and integrity will be executed. But again you might face the problem of even more redundancy of functionality in the interface tier, the business logic tier and your data access tier.

But is this really necessary?! The data access layers and business layers are often designed with this question in mind: 'which information do we supply?'. But should the emphasis instead not be on the question 'what information does the interface want?'. Do we push the data? Or do we explicitly pull the data?

For example, in our original application we decided to fetch the name field of an entity (e.g. a user) using in-line sql code. We ask for the name, and that is what we get. Nothing more, nothing less. Now consider the 3-tier application. We ask for the name, but there is no such method in the middleware so we retrieve the whole entity instead. The middleware decides to check whether we have the proper authorization to fetch the entity. And then whether we have the authoriziation to read the name field. The data access layer is unaware of these validity checks and, just to make sure, performs this validity again. The middle tier is also aware of the whole entity hierarchy so for convenience it implicitly figures out the exact (derived) type of the entity, even thought the name field we are interested in is only specified in the root entity. Maybe a bit exagerated, but I hope you get my point: there is a lot of unnecessary overhead involved. Surely we needed all this functionality when we wanted to retrieve the whole entity, using polymorphic fetches, etc. But we do not need it in this case. We just want the name of the entity. Ofcourse proper validation and authorization is needed, but only the kind that is directly related to the data we want. We don't want any data we don't need, and surely no functionality acting upon that unrequired data.

Now consider the three tiers working together as a typical supply chain seen in the retail industry. Within the field of operations research, the effect of unnecessary provided resources compared to the original demand is well-known as the bullwhip effect. This effect occurs when every manufacturer in the supply chain adds a 'safety' margin to the original amount of requested products. This accumulates up the chain and ends up in supply orders that overestimate the original demand. So even though the end customer maybe only have requested ten products, by the time the order reaches the raw materials manufacturer the order has grown to create a potential of twenty products. You can clearly see the problem here in the waste of resources due to the mismatch between supply and actual demand. The solution in this field of research has been the focus on demand-driven supply, or to put it different, the original demand by the customer must stay intact as long as possible upstream in the supply chain. This makes sure the supply is most likely to be equal to the actual demand.

I see analogies in the software systems we are building. But we may even have a bigger problem. Whereas in the retail industry the focus has been on the customer instead of the supplier for a while now, the software industry still considers the 'customer' (in this case the interface tier or the actual user) as just an executor of supplied services and information. Not as the creator of demand of specific services and information. Who is in charge of your information retrieval?!  Is is truly the business layer that can decide which information to supply? Does the interface tier merely makes convenient use of all the entities the middle ware provides? Or is it the interface tier that specifies exactly which data it is interested in? I believe it should be the latter.

There might be some improvements to be made on this topic. Yes, it is bad practice to directly access the database from your button click eventhandler. But this does not mean it is equally wrong to specify what you want in a query format and hand it over the other tiers. E.g. if I would write this in my eventhandler, would it be really wrong?

SELECT Name FROM User where User.Id=123

Is this not exactly what it needs?! How can it be more terse and self-explaning? Still, most programmers would shrug at just the sight of seeing code like this in interface code. But what if I could just hand this query over the middle tier for further processing. The middle tier would then add some business logic and validation logic in this query definition and hand it over to the data tier. This tier would in turn add some database specific filters to the query and execute it. The resultant data (a single string representing the name of the entity) would be returned to the middle tier and back to the interface tier. The amount of overhead is minimal, since the original request of information stays intact all the way up to the data tier. There is nothing more and nothing less we need.

To come back to the technology I started with, I believe LINQ is a path to enable a strategy like this. The deferred execution makes is possible to adjust the query definition in all the tiers before it is being executed. Thereby making it possible to inject the query with validation logic and authorization logic. Or to inspect the query just before executing it, instead of making assumptions about the final use of the supplied data.

Empower your interface, it knows best, but make sure your data and business layers are your law-enforcement troops.

At Google they have created an API to look up a social graph. I found it particularly interesting to come across this, since I had been thinking about the same concept for some while. I even wrote something about it a while ago. The driving idea behind it is that we have many fragmented online identities. But, ofcourse, there is only one real identity, that is the body you're carrying around all the time with you. It makes sense to somehow connect these online identities and identify the only single real identity. Now with the immense information base available at google it should be possible to create some smart page indexing and extract these information. But this is not what the social graph does: it makes use of the XFN standard to extract high-quality and consistent information. An example of an XFN compatible tag would be:

<a href="http://tanya.example.org" rel="friend met colleague">...

In this example you do not only specify the url to which the hyperlink points, but also the relation. It drives the forming of a semantic web, where not only pages and resources are linked together, but also their relations are described. Very nice stuff, but at the same time also vulnerable for abuse and privacy infringement. Use it wisely.

Zonder twijfel is de grootste mysteries van deze tijd de mens zelf. Of eerder, het bewustzijn. We zijn bewust, zeker, maar niemand weet hoe. Het is verleidelijk om in te beelden dat er in ons hoofd een klein mannetje zit die alles ziet en regelt (een zgn. Homunculus), maar daarmee verplaats je het probleem alleen maar. Dusver weten we niet welk gedeelte van onze hersenen zich bezighoudt met zelfbewustzijn, laat staan dat we weten hoe ons bewustzijn fysiek in elkaar zit. Misschien komen we er wel nooit precies achter, maar ik heb er hoop in dat we achter de onderliggende principes kunnen komen. Een fascinerende tak van sport in de wetenschap is de bestudering van complexe systemen en emergence. Hierin worden vragen bestudeerd als 'hoe kunnen simpele insecten gezamenlijk grootse dingen doen (mierennesten, groeperende vogels)?' tot 'hoe werkt een wereldwijde economie?'. Het idee is dat de som van simpele acties kan leiden tot complex en zelfs intelligent gedrag dat boven het individu uitstijgt. Zo werkt een mier enkel reactief op basis van geursporen van andere mieren, maar gezamenlijk hebben mieren een complexe samenleving en zijn ze samen in staat grote vijandigheden (dieren, veranderingen in de omgeving) te lijf te gaan.

In dit perspectief kan ook ons bewustzijn worden gezien als een 'emergence' van onderliggende simpele acties. Op een laag fysisch niveau is ons brein niks meer dan een enorme verzameling van neuronen die hun directe buren continu activeren of deactiveren. Zeer simpel en lokaal gedrag dus. Net als een mier. Toch zorgt de samenwerking van miljarden en miljarden van deze neuronen tot intelligent gedrag; een wezen dat kan communiceren, creatief is en zelfs zelfbewust is: een behoorlijke prestatie!

Waar wil ik nu heen met dit verhaal? Blijf nog even bij me, want ik ga nu een behoorlijke gedachtensprong maken. Als het brein een netwerk is van simpele neuronen die actief zijn, en het internet is een netwerk van computers en mensen die actief zijn met elkaar. En een enorme combinatie van al deze activiteiten vormt intelligent gedrag.. is het internet dan ook intelligent? Kan het internet ook creatief zijn? Of nog belangrijker, is het internet ook bewust? Wellicht is het dat al, maar zijn wij, net als de neuron en de mier, niet in staat om dit te begrijpen. We staan immers lager in de hierarchie dan het internet zelf. Het internet zien als een levend organisme is trouwens minder vreemd dan je op het eerste gezicht zou zeggen, door het als een metafoor te gebruiken wordt het duidelijker. Een hype op het internet kan je zien als de huidige (bewuste) gedachte van het internet. De grootste firewall in China is als een ziekte. Online nieuwsberichten kan je zien als de ogen en oren van het internet. Er zijn veel paralellen te trekken tussen de mens en het internet. En als het internet een beetje menselijk is, en we hebben maar één internet, wat zal het dan eenzaam zijn ..

Recently I decided to investigate more in real smart clients, partly because I am just fed up with all these DHTML / AJAX workarounds. Despite how wonderfull the supporting libraries are these days, it is still in no ways comparable to a decent desktop application. Fortunately I am developing an online application in which we can get away by demanding minimal system requirements at the user's site. Such as the use of Windows XP / Vista and the use of IE7. In this fortunate position I am investigating the use of WPF applications to create a superb interactive interface. I started out with XBAP which stands for XAML Browser Application and in short this means that XBAP applications can run inside webpages (for example within an IFRAME) and you can almost use  the full WPF feature set. Unlike Silverlight, which at the moment only provides a small subset of WPF and they are still busy implementing the first primitive controls. There is a catch however, the clients needs to work on a modern Windows platform and using IE6 or IE7 and must have the .NET framework 3.0 or higher installed. In my case this was no showstopper, so I continued investigating XBAP and try to create some test applications.

What turned out to be the showstopper was the security sandbox in which XBAP applications run. By default your XBAP applications work in Partial Trust, which is severy limited. I had problems setting cookies, accessing webresources (even from the same host?) and was not able to perform any .NET Remoting due to security restrictions, although I was able to use some WCF webservices. Nevertheless, it took me some time to figure all things out and I noticed there is very, very little information about XBAP on the internet. Most of which are not positive at all. It turned out that XBAP with it's sandbox is just too restricted to be meaningful as a smart client platform.

Just as I was about to give up, I turned my eye to developing regular WPF (desktop) applications and using Click-Once for online deployment. ClickOnce supports automatic downloads, automatic update and no local install. In short it takes care of all the hasle of software deployment, which is one of the main reasons web applications are so popular. After the initial download of your application the user is prompted with a screen to accept the installation and from that point on your application can run freely with Full-Trust and it is the smart client you really want it to be.

.. and that is ofcourse the reason XBAP has been around for a year and a half (as far as blogs and forum posts go) but is still not really used. Even though Microsoft markets XBAP as some kind of smart client toolset, it just isn't. For a real smart client you need out of that partial trust and use a regular WPF application with click-once deployment. And for some cool graphics with no real functionality you might as well turn to Adobe Flash (widely supported) or even Microsoft Silverlight (small download and available on multiple platforms).

The coming of the asp.net MVC framework had me thinking about the nature of markup and specifically the difference between the way ASP.NET Webforms handles this and the new ASP.NET MVC framework. I have not really used it yet, but when you come across MVC examples there is no doubt that the feel and look reminds you of the classic ASP style of writing markup: a lot of intermixing between markup and code and logic handling inside templates.  Values are inserted with asp.net tags instead of in the code behind. Compared to the use of webform controls this might seem like more cumbersome work. But when I look at the efforts I actually spent at writing the HTML markup it turns out to be only a small part in the total project. For the interface functionality, a lot of my efforts go into making sure the layout and functionality of the website comes across multiple browsers in a uniform manner. I sketched a diagram which will make it more clear:

So actually I like it that the MVC gives you more control over the  generated markup, since it will hopefully decrease my efforts to develop cross-browser compatibility (sigh) and create more lean-and-mean pages. There are already many helper functions available that can conveniently generate often-used html markup, such as internal links (action links) and textboxes. I am sure that we will see more 3rd party controls and utilities that allow you to generate more complex HTML (such as a treeview), but with more fine-grained control over the generated output and still hiding/covering the hard parts such as ensuring cross-browser compatibility.

In your search for achieving excellence in creating software you might have come across one of these statements that 'code is an art' or something in that fashion. Although these statements are very appealing, I believe it not to be true. I will explain why. First let's start with the reasons why a computer programmer (or coder) is considered an artist and it's product (code) art. Writing code is labour intensive and various attempts have been made to automise the coding process through the use of high-level visual languages or application-generator tools. They all fail. Why is this? Because writing code is a complex and, most importantly, creative process. The same reasoning explains why there are still no good pop-song-generating software or fancy automatic movie generating applications. Maybe this will come one day, but this area is still in its very early infancy. Still, the coder can be considered an artist because he or she is creating a product through an undeterministic and creative process where the exact outcome is not clear upfront. Just like musicians or novelists. So, if writing code can be compared to creating music, writing a novel or painting ... then why can the code not be considered an art? Music certainly is art, and so is a well-written novel. The important, but subtle, difference is this:

The value of art is determined by the art itself. Whereas the value of code is not determined by the code itself.

To restate: a painting is perceived beautiful based on the painting itself. Music is valued via the music itself. In contrast, a software product is not valued on the code itself, but on what it does. No user really cares whether you have an elegent recursive call, or a three-tier architecture. Software should work. We should never forget this. Surely, your colleagues may be impressed by the sheer elegance of your code, but your users certainly won't be! The goal of a musician and its audience are aligned: to create beautiful music. The goal of a coder and its audience is not, there is a clear mismatch and that makes it more difficult to create good software. Most programmers out there, if not all, enjoy writing beautiful and elegant code. Which is a good thing because elegant code tends to have other convenient properties such as easy maintenance, reusability, self-documenting, etc. But let's not forget that this is not the primary goal our users have; just create software that works.