Watching Data Evolve

Is it possible to say "I told you so" to yourself.

When I launched my new restaurant rating site about 2 and a half months ago, I made a decision. I decided to allow users to participate in the site without creating an account or logging in. This was calculated; The site will only come alive and be successful if users participate, so I wanted them to be able to interact without the hurdle of having to create an account. To give them value while asking for little in return. The same reason people can rate a restaurant or menu item without having to write about it.

Of course, this openness has an interesting side effect. I'm finding that some folks are rating restaurants with 10s across the board (the restaurant owners perhaps?). Now don't get me wrong, I don't mind. I'm just glad people are beginning to participate. Of course, this isn't exactly an unbiased view of an establishment's fare.

Eventually, I will implement an IMDB style mechanism so the ratings are only displayed when multiple ratings are collected. For now, thought...it's a free-for-all. It's actually a good play by the owners. A top-ten rated dish for a city appears on the home page giving a little extra exposure.

What do you think? Will people pass the site by if the ratings are obviously skewed? Should I do something to stop this at the risk of collecting less data? What would you do?

How to Start a Website For $2853.80

How much moola does it take to get a website off the ground? Of course, that depends on a number of factors: the skills and experience of the person or people involved, the overall goal of the site, the estimated traffic, the marketing strategy/budget, etc. To get FoodFinder.net off the ground, I've amassed a whopping $2853.80 in one time expenses and $480 annually in recurring expenses. Here's the breakdown:

• Domain Names: $1849.80: I probably went a bit overboard here due wholly as a result of alcohol. There are actually 5 domain names pointed to the website right now. www.foodfinder.net ($1,300 - 6 years), www.foodsearch.us.com ($149.95 - 5 years), www.foodfinder.us.com ($149.95 - 5 years), www.lunchfinder.us.com ($149.95 - 5 years), www.lunchfinder.net ($99.95 - 5 years)


• Hosting/Database: I use discountasp.net which I love. I host my jeph.tv site there as well. They start at $10/month for ASP.Net hosting and you can add on from there (I think I'm paying about $40/month with add-ons, plus about $100 worth of various set up options). They do get you for some services, but for an experienced user, I highly recommend them.


• Images: In the early stages of design, I thought I was going to need a vast library of images. As such, I signed up for a subscription at www.123rf.com. The quality is great and for $199, I get a 90 day subscription which allows me 5 downloads per day. I've got a slew of images that I will probably never use, but that's OK.


• Logo: I am no designer; this will become clear as you use the site. The logo was purchased in a contest at www.99designs.com. I loved this process. Basically, you post a description of what you're looking for and what you're willing to pay and designers from around the world submit designs. I saw some really cool options before deciding; there are some very talented folks in that community.

That's it; the remaining work is a lot of blood, sweat, and tears. I force myself to work an hour per day and that usually turns into 4 or 5. I started in April and would say I average 4 hours per day, 7 days per week. That's roughly 476 hours...ugh.

So that's all it takes. Now the path I took isn't viable (or recommended) for most people. In fact, for a non-experienced user, I would recommend creating a blog with links to static pages. Take this site for example (this actually came up recently). For eCommerce, I would probably leverage a subscription package like Ebay stores. There are a ton of free or low-cost options available that allow users with limited technical skill to develop an internet presence

What's your idea?

The Birth of a Website

Wow, it's been a long time since I've posted on this blog. I find myself working on a really interesting development project these days though, so I feel compelled to talk about it. Recently, I was inspired by the story of Plenty of Fish and how Markus has found a degree of financial success creating a site that offers great value and is completely supported by advertising. "I could do that", I cried. And so it begins...

FoodFinder.net is a website that allows users to search and rate restaurants by the menu item. The idea came from a conversation we have pretty much every day at work.

"What should we have for lunch?"
"I don't know...where can we find a decent burger around here?"
"How about Chilis"
"I don't feel like that how about..."

You get the idea. Plus, as a big fan of the food shows, I'm always wanting to try stuff I see on TV. What's the current protocol for finding a muffuletta sandwich or croque monsieur. So there you have it.

I realize this is a technical blog, and will make sure to talk about the technical challenges and architecture involved in the site. I also am amazed that despite being the same technology platform (ASP.Net with a MS SQL Server 2005 DB), the code is night and day different than what I do in my day job.

That's it for now. Next, I'll run down the steps and vendors I used to get this thing off the ground. Remember, when checking it out that is a work in progress, but will one day be great ;).

Waiting...

I spent 8 years in the restaurant industry. Waiting tables, bartending, cooking, chef, and front of the house management…I did it all. That said, I confidently make this statement--I learned more about efficient programming in the restaurant business then through any other education or experience I’ve had.

This may sound crazy, but hear me out. There are two ways to be successful waiting tables: having a great personality, or having exceptional time-management skills. Well, anybody who knows me will confirm that there’s no way I could support myself with personality, but I did learn how to get maximum results out of minimal effort. Give me a minute to lay some groundwork.

For those who have never worked in the restaurant industry, consider this. You’ve got a station of tables where you serve your customers. To service them effectively, you will need to go to several other areas. You’ll pick up (or make) your drinks at the bar, get your food (and possibly add garnishes) at the kitchen, and pickup other supplies and enter check-items at the wait station. Now, bear in mind, most of the work you do as a waiter, bartender, or cook takes place over the course of about 1.5 hours during lunch and 4 hours over dinner. The key to success is being able to meet the needs of your customers efficiently. So how does this translate to programming?

I'll tackle this question via analogy; consider the following:

• Station of tables equates to in-process memory


• Bar equates to the file-system


• Wait-station equates to the database


• Kitchen equates to web services/http-based resources

Efficiency in food service comes through consolidation. Let’s say I have five tables. If I take a drink order from one table, get the drink from the bar and then deliver, and my next action is to take a drink order from another table, I’ve wasted a lot of time. That trip to the bar is expensive. Now, if I can anticipate that both tables need drinks, I can take both orders and deliver them with one trip to the bar. I now have more time for servicing my other tables which translates to either more upselling, or faster turnaround. Both things lead to higher overall tips.

So let’s look at the programming parallels starting with the database (wait-station). If you take a food order from one table, can you take another food or drink order from another table? If so, this would save a trip, and considerable time. What if a table ask for a straw--another trip to the wait-station, unless you are carrying straws in your apron (caching). One could go on-and-on with these types of comparisons, all supporting the same theme: consolidate as many steps as you can.

This is a very common-sense concept, but you'd be surpised how often it's overlooked. From now on when you're typing (or better yet, planning) your code, see if there are opportunities to save a trip to the bar.

To Normalize or not to Normalize?

Normalization is an important concept in database design, but is it always practical? This is a question I’ve been hit with head-on while working on a recent project at work. Now, there’s more at play here than simply normalization. Our overall site design executes an excessive amount of business logic in the database tier, which makes me cringe. Here’s why.

• Databases are almost always the least fault-tolerant layers in an application


• Databases are almost always harder to debug and troubleshoot than other layers in an application.


• Databases are written to read and write from disk efficiently, and apply set operations on groups of data. Branching logic, string manipulation, and the like are convenience tools only.


• Enterprise databases are really expensive and much harder to scale horizontally.

These ideas have been beaten into my head over the past 4 years or so, and they make a lot of sense (or I'm just brainwashed).

The next point is a matter of maintainability. Let's say you've got a stored procedure that's about 150 lines long that contains some conditional logic and is joining across 7 or 8 tables in multiple branches and in various sub-selects. Now, you have to add a whole new set of functionality, but to do so, you have to integrate data filtering that exists across 9 or 10 new tables. Now lets add another real-world variable to the equation...turnover. What happens to the poor bastard that inherits this beast? To me, this is far more costly than any performance loss that is incurred by introducing complex joins and decision logic.

Now don't get me wrong--I'm not anti-normalization. In fact, it should probably be the first thing considered in any data modeling. But normalization-for-the-sake-of-normalization can be a slippery slope in terms of an application's agility and maintainability. Embrace best practices, but question them as well. There are people out there who make a lot of money selling enterprise database software so be careful about whose Kool-Aid you drink.

Rolling Your Own AJAX (Part 3)

I'm looking through the Ajax library source code and noticed something that made me laugh. Three of the five objects were written using JSON and the others were constructed the more traditional way (creating functions to represent the object and defining members using the "this" keyword). Normally, I'm not this inconsistant but hadn't used JSON before and wanted to develop some proficiency. The reason being that deserializing JSON into a native JavaScript object is considerably faster than creating and working with an XML document (although there are security implications). That said, I think any reusable framework needs to support should probably support both, as well as simple string responses.

One of the challenges I ran into when coding this model was getting the message data to be accessible to the context object. The problem was this--the message contains a pointer to a callback function along with a reference to the request object. What I wanted to do was pass the entire parent message object to the callback. This would allow all of the request and response details, along with the properties of the message to be available when synchronized back to the main execution thread. But there's a problem.

The issue came with the "this" keyword. The AjaxMessage object contains a method called "process" which obtains a reference to a request object. This is temporarily stored in a property and the message object calls the "send" method of the request. When the onreadystatechange fires, the context becomes the request object; the reference to the parent message object is lost.

This was actually a simple fix for Firefox. The open-endedness of JavaScript objects would allow a property to be added to the request object with a reference back to the parent AjaxMessage. IE however is a different animal. The HTTPRequest is not exposed as a native JS object, but rather as an ActiveX control which plays by its own rules. This is actually related to the inability to cover a select box with a div object without the use of programming trickery (I prefer the iFrame shim myself). This problem gave me fits for over a week. So what was the solution?

It actually turned out to be pretty simple and logical. I was able to create a local variable that contained a reference to the AjaxMessage within said message.
When the "this" keyword picked up the HTTPRequest reference, I was able to get back to the message by passing the local reference as an arguement to the callback. At this point you may be thinking the same thing I was...with all this circular referencing going on, have we created a memory leak? The short answer is not that I've seen yet, but I'll go into further detail in a future blog.

Anyway, I've been using the Ajax library for a while now and have had great results. It's small (about 400 lines of code), reliable, handles a variety of page types and load, and very easy to plug in and use. If anybody is interested, you can download it here and use it at will. I'd love to hear about any bugs, feature requests, or general feedback.

Rolling Your Own AJAX (Part 2)

I left off talking about some of the perceived UI benefits of using AJAX. What I'd like to cover now is how I modeled the AJAX client library for this project. I did some research into some of the existing solutions, and there is some good stuff out there. Dojo is a very robust client-side library that goes way beyond simple AJAX functionality, and there a couple of .Net frameworks available, too (Ajax.Net and Atlas come to mind...I'm sure there are many more). For the project at hand, I decided to construct a framework from scratch--a direction I almost always lean toward.

Here is the object model I came up with, and the general rolls each would play:

• AjaxMessage: This object encapsulates a single callback to the server, and contains the request details, response details, and handling details (a reference to the callback function). It also contains the actual HTTP request (which I will most likely decouple when I refactor this puppy).


• MessageBuffer: This object is essentially a traffic controller, and determines how messages will be handled in the context of the page. When going through possible AJAX use-cases, I felt like we needed a lot of control here. In the project at hand, the user had the ability to request information faster than it could be processed. The MessageBuffer object can be set up to send messages immediately or in batches, and can filter duplicate messages.


• MessageCache: This is a repository for "cachable" messages. If a message is deterministic (will return the same results with the same inputs), it can be cached and the return data accessed later without going back to the server. This object manages cache settings including capacity, threshold, and expiretype (messages can be set to expire after a given amount of time, or via a least-frequently-used algorithm.)


• CacheObject: This is simply an AjaxMessage object with cache meta-data.


• Request Pool: OK, this was something I coded before had fully researched it. The idea was to create a pool of HTTP Request objects when the AjaxFramework is initialized to avoid the overhead of object creation for each request. Great idea in theory, right? The problem is the HTTP 1.1 specification says that only two request objects can be open by a browser simultaneously. IE adhere to this by default (although it can be changed in the registry) while Firefox has a default of eight. Oh well, if this changes in a future release, I'll be ready ;)

Here's what the model looks like for you UML folks out there:



Next, I'll get into some of the implementation details and specific features of the framework.

Rolling Your Own AJAX (Part 1)

As AJAX has grown as a programming paradigm for the web, there has been an emergence of platforms to support development. I was lucky enough to work for a company that got into the web 2.0-game fairly early and was tasked with designing and creating a client-side framework along with some new form controls (the code I’m going through here was written about a year ago). Today, I’ll talk a little about some of the benefits and challenges of using AJAX in web user-interface design.

This was one of those rare projects where we were given adequate time to do a good job, so I spent a lot of time thinking about what some of the challenges were going to be, both for this and future implementations. AJAX creates an illusion of immediacy to the user by bypassing many of the redundant processes of a typical postback. Think about the lifecycle (we’ll do a simplified version) of an HTTP page request on the web.

1. The user makes a request by clicking a link or submitting a form


2. The browser makes a request to the destination web server


3. The browser receives the response as a text stream


4. The response is de-serialized into a document object


5. The page artifacts (images, video files, JavaScript and CSS files, etc) are identified, and latent HTTP calls are made for objects that are not available in the browser cache.


6. The browser paints the HTML page based on the document object and page artifacts

In an AJAX model, steps 4 - 6, are bypassed and the data from the request is simply inserted into the existing DOM. Three fairly expensive and unnecessary processes (because they were handled with a prior request) have been replaced with a lightweight operation. So how does an intrepid developer take advantage of this technological wizardry? Basically, the same object that is used by the browser to make server request is exposed and can be accessed through JavaScript.

It looks like this could get a bit long, so I think I’ll make it a trilogy. In the next part, I’ll dig into the hows and whys of the framework I put together for the aforementioned project, and see where it goes from there.

The Skills that Don't Pay the Bills

The other day I was asked a question by a friend/co-worker about what skills are important to have for a developer today. Reflecting on the question, I think he wanted to know what skills make somebody a more attractive candidate on the job market. Of course rather than answer the question, I went off on a tangent regarding a recent pet peeve of mine.

More and more, I'm finding that .Net developers are losing touch with how things work. Microsoft has abstracted many programming functions to such a high-level, many people in the industry have forgotten (or no longer bother to learn) what's going on under the hood. I suppose this is a natural and necessary evolution; the growth of the talent pool doesn't come close to the expansion of software development and implementation needs. By dumbing the work down with every release, Microsoft expands the pool of workers while fostering a need for the latest software and training.

The result is an increasing gap between programmers/engineers and coders. So my answer to the original question was--try to be the former rather than the latter. In many ways the distinction is subjective, but if I wanted to gauge somebody's programming abilities, here are some of the things I might ask.

• Write a function that sorts an array. Describe it in big-O notation.


• Write the pseudocode for a simple XOR encryption algorithm


• You have a table in a database with 1,000,000 rows and need to extract a single-specific record. What is the maximum number of records that will need to be evaluated to return the required row if the column you're looking at has a clustered index vs non-indexed with unique values.


• Write your own string-builder


• Design a custom cache mechanism

From what I've seen of the job market of late, it's unlikely these things are going to help one command a higher salary, but I think this depth of understanding is becoming a lost art in business software development.

So This is a Blog...

So I've been reading a lot of these blogs lately and figure it's time to throw my hat in the ring. My name is Jeff, and I'm a web software engineer currently living in San Diego, having recently moved from the Boston area. I've been a web developer for the past 8 years or so and have mostly worked in Microsoft technologies on the server side.

Some of the topics I plan to tackle are architecture and design, coding practices, hiring and interviewing, and emerging technologies. I'll also try to reflect on projects and jobs I've had in the past, and compare and contrast working on opposite coasts. With any luck, I'll stick with it for a while. Who knows, maybe somebody will even read it from time to time.