Tigraine

Daniel Hoelbling-Inzko talks about programming

The technology behind kaernten.at

During the last few months I have been the technical architect behind the development of a new platform for tourism in Carinthia. While I have already discussed some of the inner workings of the project during that time, I thought it might be interesting to let you in on a deeper look at the application architecture behind the site: www.kaernten.at.

Quick facts:

The site is built using ASP.NET MVC (since Beta 3) being hosted by IIS6 on a MSSQL 2005 database.

The whole thing is glued together by Castle Windsor and DynamicProxy2 while tests are all run through xUnit and Rhino.Mocks. The error reporting is done through ELMAH.

General Architecture

There are three layers to the application:

image

The Web tier was the actual ASP.NET MVC Site and this was the part I was least involved in. The Controllers use the Services layer to retrieve relevant information and then passes it on to the View.

I knew from previous projects that the web tier is the first place that is going to get torn apart by weird customer requests and other very ui-driven changes, so the general goal of the services layer wasn’t only to present data to the web, but to provide services to manipulate that data for presentation. So the Services layer is mostly  about filtering/sorting/querying data from the database, since there is no common logic to the madness of presentation requirements.

The database tier is actually split across two tiers, the Sql tier and the Proxy tier. (I’ll cover the Proxy in a future post in more detail.) Most notable the whole database retrieval is done through Linq to  Sql.

Also everything in the system is dependency injected, there is not one call to new() anywhere in the system, except for the construction of data objects (that in my design aren’t allowed to have services on them).

Query logic

The whole site is based on the concept of When youre visiting? What are you interested in? And Where would that be?

So, there is not a typical tree of categories with articles to browse through, but rather is the whole content of the site the result of all articles that match your criteria. Therefore every article in the system has a list of tags it’s connected to (when  and what both are just tags).

Now, one of the main challenges for all of the team (involving the customer) was to really define that query logic, and we changed it about 10 times during development, and I’m sure it will change even more in the future.

So, although it might have been possible to do all querying/sorting/filtering with Sql, the Sql code became utterly unreadable and sometimes involved n subselects.
I gave up on the Sql pretty early on since it was just too un-maintainable and felt too restraining.
My solution to this was to load all articles into memory and filter/sort them from there. This then enabled me to easily implement queries like this one:
All articles that are tagged a,b,c AND at least one of d,e,f (or more formally: (a && b && c) && (d || e || f))

And that’s about it for the general picture right now. I’m planning on posting more on the technology behind www.kaernten.at over the next couple of days.

Filed under net, job

kaernten.at – Caching and Lazy loading

In my last post on the technology behind www.kaernten.at, I concluded with the fairly bold statement that all objects are kept in memory to ease querying.

That’s not really true, but also not really false.
When the article list is loaded initially, only a slim index of all articles is retrieved from the database. All information relevant to querying and sorting (id, tags, publication date, visibility) gets fetched at once. I used Castle DynamicProxy2 to create proxy objects that pose a the real thing to lazy load the article data on access. So although there may be 10.000 articles currently in memory, only those that have really been viewed by a user have been fully loaded, while all query criteria are present in memory.

But lazy loading wouldn’t work if the list of proxy objects wouldn’t get cached somewhere, and that’s what I wanted to go over in this post. All articles retrieved (in their proxied form) come from the generic IRepository<T> interface that only has one method: GetAll(). I wanted to avoid having the caching be tied into the Repository at all costs since it would make my life hell when testing the repository itself. So I decided that the best course of action would be to create a decorator that took care of the caching, separating caching from the Repository implementation (and it’s also really easy if there is only one method to implement):

image

Switching caching on or off is now as simple as removing/adding one line to the Windsor configuration. And the caching code is shared among all Repository implementations, for any type of IArticleRepository<T>.
If it needs to get cleared I can simply ask the Windsor container to retrieve all ICache implementing classes (they are singletons) and call ICache.Clear() on them.

Now, the most interesting thing here is that the cache is asynchronous. If it gets cleared, a background worker process gets spawned on the PopulateCache() method, generating the updated data from the system, while subsequent requests to the Repository (while it’s being populated) still return the outdated list of data. Once the background worker is done populating, it simply swaps out the references and the old data gets collected by the garbage collector.
This result in some pretty sweet response times, since except for the very first startup of the system, every request is served directly from memory, without any data fetching at all.

This part was also crucial to the whole thing since fetching the data is a rather long running and cpu heavy task, taking up to 60 seconds while testing it against 10.000 articles (with a debugger attached on my laptop).
Initially I tried to speed the querying and object building up to allow for a synchronous cache refill, but although I did some optimizations (like not using some LinQ methods) the attempt to reduce the execution time to something acceptable was futile.

Right now the cache is implemented by just a simple List<T> that stores everything the repository returned, but if need arises it can easily be changed to use memcached or some other more sophisticated caching method.

Filed under net, programmierung, job

The joy of pair programming

Software design is hard. Not so much because it’s so hard to come up with, but because it takes a very long time to really hit the sweet spot where you really feel it’s good.

Doing this on your own is almost impossible, because you constantly have to switch off your personality and challenge the assumptions you just made when writing something.
I ask myself all the time “Is this module really right here? Should I break this up into smaller modules?” .. And frankly, I am the wrong person to answer that question since I made the mistake in the first place.
So I challenge myself all the time into the mindset of another person (be it a user, tester etc..) and try to forget what I was just thinking for a minute to decide if I’m still on track or not.

It’s like driving alone in a car through unknown terrain, you have to stop all the time and pull out the map to see if you’re still driving in the right direction or not.

And honestly, it sucks big time. Stopping is always bad. Dropping out of your “zone” and doing a complete context switch hurts the flow and I feel mentally exhausted very fast, and nothing seems to get done in the long run.

So, at my current project I decided that I can’t go on on my own. I tried multiple times to get to a good design through lots and lots of whiteboard, spiking and experimenting. But I never quite nailed it, I always felt like 20% away from the real thing, but with a burned bridge in front of me.

So, when I called my employer last Sunday I asked one thing “Do I get some budget to bring in a second pair of eyes to work on this particular problem?”. And the awesome answer was “Just do it and spare me the details.”.

Next day I called Harald Logar and he agreed to stop by and go through the code with me for a day. I gave him a very brief heads up on what I was working on (mind you, he’s a complete outsider to the project) and what problem I’m trying to solve.

When he came in next day I explained the vision, and showed him some tests I prepared before that should demonstrate the “desired” behavior of the system. After that little introduction, we were already implementing like crazy.

It was amazing! Although I was doing most of the typing, Harald was constantly there to challenge my assumptions, answer my questions and throw in his own ideas when necessary.

But what was really an game changer for me that day was that we were not only good, we went faster than I had ever done in the past. We rewrote the complete data access logic of a rather complex system in less then 2 hours (complex means dynamic proxies, caching and some non-trivial retrieval stuff).

We then spent the rest of the day optimizing the system (performance is very critical for the project) and I think we both learned a great deal about the inner workings of .NET collections and how they work performance wise. (We also implemented a very cool cache solution that clears the cache on a background worker thread to avoid downtime)

So, needless to say I’d do this again any time. Harald was a joy to work with, and I think by the end of the day we were both very proud of what we accomplished.

Filed under net, programmierung, job

Feeling like Bruce Wayne

IMG_1861

Although I don’t dress up with a fancy black suit and hunt criminals by night, holiday season means living a double life for me.
Most of my friends study in Vienna or Graz and only visit during holidays, so whenever they are all in town I have to maintain a student life during the night while still working during the day. That usually means getting up at 9am while staying up till 5am.

Believe me, when 6 people don’t accept no for an answer and want to play poker at your place, you’d better restock on coffee and make sure you eat light so you don’t ruin your stomach completely. (I start regretting having built my own poker table)

And still, it’s holidays so trying to maintain a 8 hour per day pace is almost impossible. Besides your family obligations at various christmas celebrations and the usual shopping madness there isn’t really enough space to get focussed on something long enough to actually finish it in a good way.

So today I came back to office and started filling out the holes in my application I left during the holidays. Working my way from //TODO: statement to the next, revisiting the old code I noticed one thing: Code quality doesn’t matter.

When I write code I can’t forget it afterwards. I mean, I suck at remembering syntax or class names (man I love google for bringing back my memories over and over again), but if I feel like a solution isn’t elegant enough or a module should be restructured to make more sense I’ll think about it whenever not occupied and eventually come up with something better.
What would have taken me hours to get right the first time, was fixed in a matter of minutes after I had time to think about it. So this leads to the interesting conclusion that nothing I’ll write today will actually matter next week, as long as I constantly rethink and rework my code I’ll end up with high-quality code no matter how bad it was when I first wrote it.

So the most important thing to consider when writing code the first time is not let implementation details “leak” out to other parts of your system, so reworking one part of the system won’t affect the other. Also writing tests for one-line methods may seem dumb and repetitive, but once you start juggling around stuff while a release date is coming at you at alarming speed, those one-line tests will assure that your app won’t blow up once deployed.

Unfortunate news...

I've been very happy with my project's progress over the last week, so I was totally shocked to hear that a similar project just launched in Germany and appears to be pretty competitive.

It's called pizza.de and is doing pretty much the same thing we were hoping to do. Providing a platform for online food delivery for a whole country.
So, the first design meeting today started with something along the lines of "Why didn't I know".
The research on competitors we did is now almost 3 or 4 months old, and apparently pizza.de launched 3 months ago, slipping through.

They are providing a pretty good platform for online food delivery and service, are using some AJAX to spice up the user experience while providing great branding support for corporations (like PizzaHut, Subway etc... Everything looks pretty solid and it's pretty clear they are working on a far bigger budget than my team currently.

So I had some doubts if we'd carry on our project pizza.at, mainly because pizza.de is almost exactly what we thought we'd have to deliver and we were hoping to be able to expand to Germany in the future.

So, because we can't compete with pizza.de in Germany, we brought up another issue: i18n. Austria is far too small to be the only place where we want pizza.at to be available, so after the initial Austria launch we will be looking into expanding to the east and south.

Now, because of our tight schedule (that just got tighter) hopes are high we'll be launching at the end of march so we'll be first to hit the Austrian market with our site.

Filed under job

I&apos;m frightening!

Today I really scared the hell out of my co-workers.
Although I don't officially work for pixelpoint I am currently working at their office, and so my fellow co-workers had to witness one of my scarce "I jump up and dance through the room in ecstasy because it works" moments.
No need to say that they where completely shocked and came running with water and tranquilizers :).

Why I am a full of joy currently is because my first prototype of my online food-ordering application works!
I was fighting the whole day with validation and data-binding issues on ASP.NET and finally got it to work.

I'll blog on my experiences with LinQ2SQL and ASP.NET in the future. For now I'll just say that I am very very happy with LinQ2SQL and I am absolutely sure that it saved me almost 3 weeks of development.

The whole application is completely functional after 7 days of development (still lacking a GUI)! I mean, obviously you need to be a programmer to order a pizza through the exposed model, but shopping cart, order checkout and product view functionality is already there.

I will spend next week working on a admin application and meeting with the design guys to get the front-end out.

Filed under net, programmierung, job

Strange LinQ to SQL oddity

I'm still in the process of doing some research on the tools I'm going to use over the next weeks to build the Pizza.at web app.
So today I tried some LinQ to SQL so I can get used to it a bit. And while doing so I started out with reading through ScottGu's LinQ to SQL series.

I had already some parts of the database schema in place and so I imported it to the ORM, giving me the following model:

linq1

So, this all worked perfectly and I immediately started doing some select statements (while learning how to use and write Lambda expressions - I'll blog about this later).

linq2

And yippie! It worked, although I was very confused that this statement would return a InvalidOperationException in case the Lambda didn't return something. Although I like the fact that it returns an exception, I think the exception given here is too generic. They could have done better in giving the exception a stronger name.

But, after getting over the exception name, everything went smooth from there. Except for me not seeing one thing:
The C and D in CRUD!

ScottGu showed off in his series that doing a simple Categories.Add and an afterwards call to dbContext.SubmitChanges() should do.
In my version of Visual Studio and LinQ I simply don't find this method. All lists I get from the ORM are IQueryable and don't incorporate any Add and Delete Methods.

I'm getting a bit confused, I must be missing something at this point, there must be something I have overlooked. Any thoughts?

Filed under net, programmierung, job

SQL Server 2005 GUI complexity

You know what?
I really love SQL Server 2005, just because it got rid of the Enterprise Manager stuff and introduced SQL Server Management Studio.

What's bugging me now is that although the Management Studio is great to work with most of the time, there are some tiny little things I really hate whenever I create new databases!

sqldb

Whenever I want a field to be a primary key id field I need to go to declare it as primary key, and then define the identity specification.

And there is the design flaw. You can't just double click Identity Specification to change it from No to Yes, you actually have to first click the + to expand the section in the property grid before I can set the Yes/No value.
I see the added complexity in some points where the increment or seed aren't appropriate, but to be honest, most of our ids start at 1 and increment by 1. So in 99,9% of the cases you don't need the extra complexity.

Filed under programmierung, job

A busy day

At first I didn't want to post this. I'm completely exhausted after 8 hours of doing training at pixelpoint and some computer administration stuff for a friend afterwards.

But, because I talked like 3 hours about the Web 2.0 phenomenon and kept repeating "You should blog" I forced myself into blogging about it.

Although I didn't plan to stay for so long I actually enjoyed the day. I was doing a course on .NET programming in VB.NET and people were very interested and doing very well. Afterwards we had an open discussion (after I showed off some Web 2.0 stuff) about Web 2.0 and how it can be used for various things in the carinthian context.

The Web 2.0 actually got out of hand after we covered WordPress and FeedBurner in detail and we spent like an hours discussing the pro's and con's of WordPress as a CMS system for smaller web sites.

The programming session was far more enjoyable for myself. I think they actually did learn something from the whole thing, although they are still hesitating to write code themselves (I had to force them into writing a small sample application). But I think they now understand the .NET syntax much better and will be more efficient on doing ASP.NET web sites in the future.

Filed under job

New job!

Yes, I'm going back to work.

I accepted to work on a new ASP.NET web project as CTO and developer for my previous employer pixelpoint.

So expect to hear new stuff about ASP.NET and AJAX.NET here on my blog during the next weeks.

The project is called Pizza.at and will be a online food delivery service.
I had been working on a previous version that didn't make it to a release while I was working for pixelpoint. Due to time constraints and poor planning the project failed and I was approached some weeks ago to try to get it done during these holidays.

I accepted the deal, mainly because I was offered complete freedom on development (so I can abandon the old codebase).
So, this time around I'm the one to blame if the project goes wrong, because I'm the guy doing all the planning.
I have chosen an agile development process and am currently working on the requirement analysis and some basic design stuff.
Because I want this project to be done fast I decided not to use any fancy new stuff like MVC or Silverlight. The whole project will be done on a SQL 2000 database with normal ASP.NET. Regardless, I'll be using C# 3.5 and some LinQ queries will surely make it into the application.

The whole thing will start later this week after I did another training course at pixelpoint for their employees.

Filed under programmierung, job

My Photography business

Projects

dynamic css for .NET

Archives

more