New Website

When I posted my article on creating a RESTful SOA it occurred to me that my own website was badly abusing every architectural direction that I was advocating. I decided to put my architecture to the test by creating a RESTful SOA for my own website. While I’ve recently applied the RESTful SOA to enterprise web applications I’ve never applied it to something so small.

So, I created a FrontController, ResourceLookup, and a full MVC utilizing a RESTful SOA written entirely in PHP and all for this one site! I even decided to keep the code backwards compatible with PHP4, so I didn’t get to use the cool reflection capability in PHP5, but I wanted to prove that I can still apply proper SOA fundamentals to PHP4.

I kept things simple and even managed to remove all dependencies to MySQL (which my previous site utilized). I only have three types of resources so it was relatively easy coding.

The end result: everything you’re seeing here can be represented as a web page, or as a service to be manipulated using HTTP methods, or as a news feed. I also added a text representation, which doesn’t serve much of a point except for demonstration.

The framework only took me a few hours to create, which was great, as it proves that a RESTful SOA is a flexible architecture that applies to web applications small and large. The problem was, I spent an entire weekend editing stylesheets, creating a new look and feel, and basically redesigning the entire site! That last part took longer than I wanted, and I’ve still got some bugs to work out.

Creating a RESTful SOA

I’m going to keep this discussion purposely language agnostic, however, I currently have two projects where I’m implementing a complete RESTful SOA (Service Oriented Architecture). One in J2EE, with a Model 2 architecture using Servlets, JSP, Oracle, and XSLT for the presentation-layer transforms. The other project is LAMP-based (Linux, Apache, MySQL, and PHP) using mod_rewrite and a custom front-controller.

It’s worth pointing out that a service-oriented design is different from object oriented design; and further, these two concepts are orthogonal to RESTful web applications. The PHP application I’m working on is not object-oriented (OO) while the J2EE project is rigorously OO. Both projects follow the same RESTful SOA pattern that I am presenting here.

Both projects have elements of aspect-oriented design (AOD), such as the authorization and logging aspects. I’m going to avoid this topic, and instead focus on the specifics of a RESTful SOA, but if you’re interested it’s worth understanding AOD as a way to compliment the traditional object-oriented design principles that don’t often lend themselves to things like REST.

On with the architecture!

Context of Use

The first step in building a RESTful SOA is to understand, and even better, document your context of use. An easy way to do this is to create a spreadsheet with three columns: Roles, Environment, and Goals. Create a list answering the following questions:

Roles: Who is going to use this site?
Environment: Where are they working?
Goals: What are they trying to do?

This is a simple exercise that is often overlooked. Ignoring this will just give you more work and a crappy product. The beauty of a RESTful SOA is that it provides resources that map to the different types of users and their goals. Don’t add anything beyond what your users need to do. You’ll find yourself making simple pages that provide a positive user experience.

Resource List

Walk through your context of use and create a list of content categories that meet your users goals. Even if you’re dealing with advanced applications, don’t complicate this process, tackle each of the users goals separately with different content sections.

Create a simple file hierarchy representation of your content, mapping closesly to your users and their goals. Here’s an example snippet:

/
/support
/admin
/it
/support/cases
/support/cases/num *
/support/request
/support/metrics

The REST is easy

You’ll find that in this process you’ve listed all of the resources that are needed to fulfil the needs of your users. Don’t invest your time into a CMS or a portal, the next step is to build your own application. You’ll find that a RESTful SOA will be quicker to build than configuring “Hello World” in a jsr-168 portlet!

Front Controller

This is critical, avoid JSF, and stop inserting random code into your html via JSP or PHP or whatever. You’ll want one small program that intercepts all traffic. Every language has a different way of doing this, whether you have to use mod_rewrite or servlet-mapping I don’t care; but funnel everything into one program and grab the path:

  class FrontController {
    path = getPathInfo();
    ...
  }

The Method Matters

The beauty of REST is that we leverage HTTP, URI, and our own web-server to do most of our work. Your front controller needs to process each RESTful HTTP method (GET, POST, PUT, DELETE) separately. Also make sure a request object gets passed in (query string, post variables, cookies, etc). You should have something like this:

  class FrontController {
    path = getPathInfo();
    ...

    function handleGet(path, request) {
      ...
    }

    function handlePost(path, request) {
      ...
    }

    function handlePut(path, request) {
      ...
    }

    function handleDelete(path, request) {
      ...
    }

  }

KISS your CRUD Goodbye!

I should say CRUDL (Create, Read, Update, Delete, List), but that didn’t sound as cool. The next step is to parse your path, this varies from one application to another, but generally your path will contain what I like to call a “resource” followed by “input”. For example:

/support/cases/50

50 is not a resource! However, /support/cases most certainly is. Also, if the user goes to /support/cases, I should be showing them a list of cases rather than details about a specific case.

The simple way to solve this is that when you process your GET request and you haven’t been given any “input”, just a resource name, you’ll want to call the “List” method rather than “Read”. This keeps it simple and very, very RESTful.

Put the DAO to REST!

The astute reader may have noticed I glanced over how you’re suppose to derive a resource and an input from a sigle path. Well, REST assured, this is trivial. Create a simple data model with a Resource entity that maps to itself (one-to-many) to allow for multiple children resources. So, “/support/cases/50” would involve two entries:

+-------------------+
+ NAME    | PARENT  |
+-------------------+
+ support | NULL    |
+ cases   | support |
+-------------------+

Obviously, you’ll want to use unique numeric IDs so you don’t have name collisions, but you get the idea. This will allow you to lookup unique resources and an optional input parameter. The “resource” entity should also contain things like “Title”.

The children resources represent the sub-elements that you may want to appear on a menu navigation. So you’ll want to keep these handy also.

Finally, you now have a unique resource, with optional input and one of four HTTP methods. This all maps nicely into one pretty package, which should be something like:

  class RESTController {
    path = getPathInfo();
    resource = getResource( path );
    input = getPathInput( path );

    function handleGet(resource, input, request) {
      if ( empty(input) )
        resource.list( request );
      else
        resource.read( input, request );
    }

    function handlePost(resource, input, request) {
      resource.update(input, request);
    }

    function handlePut(resource, input, request) {
      resource.create(input, request);
    }

    function handleDelete(resource, input, request) {
      resource.delete(input, request);
    }

  }

Presentation and RESTful Content

Probably one of the biggest architectural mistakes you’ll find with web applications is an incorrect MVC abstraction. It’s easy to say “Model, View, and Controller”, but rarely is this done right. Typically the architecture looks pretty, but on close strutiny calls for tightly coupled templates with functions that belong in the model. Do yourself a favor, don’t let your view talk to your model.

You’re already using XML

Whether you like it or not, you’re on the web, you’re producing html for your content and hopefully keeping your presentation in css. The great thing about XML isn’t that it’s easy to parse or to read. I hate XML, it’s ugly, stupid, and slow to parse. What it is good at is transforming from one schema to another (via things like XSLT).

When you ask your model for a “read” or a “list” operation, you should be getting back pure conent. Content in the form of XML. This is content without a presentation, and it belongs in your model not in your view!

Putting the Representational into REST

If your model is dumping out content in the form of XML, the only thing left to do is for your fancy front controller to transform this XML into whatever representation that the user asked for! It’s a good idea to make the default representation html. The code should now be looking like this:

  class RESTController {
    path = getPathInfo();
    resource = getResource( path );
    input = getPathInput( path );
    view = new ViewDispatcher( request.getType() );

    function handleGet(resource, input, request) {
      if ( empty(input) )
        model = resource.list( request );
      else
        model = resource.read( input, request );

      output = view.transform( model );
      print output;
    }

    ...

That’s it, you can take your presentation-less content and transform it into anything you want (html+css, flash, pdf, plain text). You could even make it a SOAP envelope, autogenerate a WSDL and now you have REST and SOAP without doing any extra work!

So there you have it, this is the basis for a RESTful SOA, that if done right, will allow you to build really cool web applications. Since it’s RESTful, every resource is accessible in a browser, or as a service to an API, or whatever REpresentation you want!

Funny thing, it seems that the whole point of REST is to do MVC the right way, and de-couple everything. I’m not sure why all the big CMSs and portals claim to follow an MVC (or Model 2) architecture but fail to provide a decoupled view.

Why aren’t you working on important problems?

I just finished reading the transcripts from Richard W. Hamming’s speech ‘You are Your Research’.

“If what you’re doing is not important… why are you working on it?” he asked.

In context of the latest GTD (Getting Things Done) craze, this is a critical question. A question that should keep you up at night.

As you’re processing your inbox, tracking your next action for each project, ask yourself: What are you working on, and why?

Do your actions connect, not just to an arbitary project, but to a meaningful goal, to the greater good of your life?

The first step in managing your life, in meaningfully getting things done, should be to follow Richard Hamming’s advice and ask yourself:

What are the most important problems in your field?

Put another way, what are the important problems that you could be working on? And why aren’t you working on them?

Stepford Geeks

One of the things that makes us geeks is that we used to have too much time on our hands. We developed technical and creative skills often times out of boredom. Alone, in a dimly lit room – we made web pages, wardialed, operated a BBS, wrote an OS, designed a computer, and so on through the history of geekdom.

Now we have no time. Those geek skills have given us cool jobs. We’re no longer bored, we’re creating a new world and there’s just never enough time to do all of the things that we really want to do. The latest trend is to use our geek skills to hack at our life so that we can get more work done.

But what if this is wrong? I’m also a movie geek, so let me put it likes this:

“When you come back, there will be a woman with my name and my face, she’ll cook and clean like crazy, but she won’t take pictures and SHE WON’T BE ME!” Stepford Wives

I need to make sense of this. Let me take a step back.

This week, I started my new job at Qualcomm. For an email client, I got to choose between Eudora and Outlook. I thought to myself, did I just go back in time… people still use this crap?

I almost forgot what old mail clients were like: New messages don’t reference the conversation thread from the archive. Wait, there is no archive. But there is a 500 MB quota; meanwhile my free gmail account just hit 2700 MB.

I’ve spent so much time life hacking I forgot how complicated email can be with the wrong tools.

Like any life hacker with proper geek skills, I designed a simple system that minimizes my reliance on Outlook, by dumping email into a private Subversion controlled directory that also works as my personal file space that can be shared/searched across multiple computers. Hooray, geek skills to the rescue!

In this process, I started looking through some of the old files that I’ve kept around. You know, my old web pages from 1995. Oh and a recursive decent parser for Firewall-1 I wrote for no reason except to figure out how to do a recursive decent parser for Firewall-1. I also found some “experiments” in Photoshop and 3DSMax. Really terrible stuff. My parser was horribly slow and looking at it now, it’s some of the worst code I’ve ever seen.

But this is the byproduct of my bored youth. I used to create web pages for fun. I learned Photoshop for fun. I learned HTML/CSS/Javascript for fun. I learned to program and make cool Java applets for fun. I learned how to manipulate data with Perl for fun. The list goes on and on to include real-time environmental sensors, databases, data modeling, but it also includes photography, graphic design, comic books, movies, writing, and even television. Basically, things that geeks get excited about!

Now that I’m out of time, I wonder if I’m using my geek “life hacking” skills to destroy the part of me that makes me a geek? I get excited about odd things because I’m bored with my life, but if my life hacking makes me so good at life that life ceases being boring… then what?

When you come back, there will be a geek with my name and my face, he’ll design and code like crazy, but he won’t take pictures and HE WON’T BE ME!

sigh…

I guess we should remember that if we’re going to life hack, we must always remain oddly discontented with our lives no matter how good at life we get. We should remember to hold such unrealistic and fanciful ideals for ourselves that our daily world is dull by comparison.

Good Design, Bad Code

I’ve been reading PHP 5 Objects, Patterns, and Practice, a great book exploring modern day design practices and the conventional wisdom of good software design. For serious PHP developers it’s a great update to the “gang of four” in a PHP context and really shows how PHP stands up as a serious development environment for the enterprise.

Unfortunately, all of this presents some serious misgivings about conventional software engineering methods that this book serendipitously stumbles upon. One of the code excerpts reads:

  class Army extends Unit {
    private $units = array();

    function addUnit( Unit $unit ) {
      foreach ( $this->units as $thisunit ) {
        if ( $unit === $thisunit ) {
          return;
        }
      }
      $this->units[] = $unit;
    }
   ...

I’m sure to the hot shit J2EE/EJB software architect this is perfectly good code in the context of a composite pattern. The above case represents a composite class that holds Unit objects, and the addUnit method simply adds Unit objects to the composite. The great part is this class will rarely, if ever, need to be updated since the business logic is separated into individual leaf classes and will be easier to maintain. Java programmers would be proud to see PHP like this.

On the other hand, every 15 year old hacker will now have something in common with old-school C programmers. They will recognize that this code is shit. It’s utter crap. It’s inefficient, sloppy, and scales poorly while being difficult to debug.

Have we as software engineers become so pig-headed that we no longer care about code efficiency and happily let a big-O(1) operation become a big-O(N) operation just because it uses the right pattern? When did software engineering mean you have to be stupid? Can’t I be a software engineer that follows good, disciplined, development methodologies while still favoring good code? We’re assuming that the intelligence of our design patterns will make up for the bad code.

For the J2EE crowd, PHP arrays are all associative, similar to a Vector class. In this case you’re adding individual leaf objects to a composite class; and every time you do you’re looping through an associative array to determine if the object already exists in the array. Each time you build this composite, which in a web environment is every page load, you’re doing a mountain of extra work. This is great for a small number of objects, but what if you have thousands, or even millions of objects at hand. This is what scalability is all about, right? When did our design patterns make us stupid?

The goal of software engineering is to produce good software, good software requires good code. Disciplined software engineering may have something to learn from the hackers who would recognize the bad code excerpts in this book.

Leaving Academia!

I resigned today. Quit my job. After several years I have decided to leave the University and head to the private sector. Before I leave, I want to remember the wisdom of Dr Ray Stantz:

“Personally, I liked the university. They gave us money and facilities, we didn’t have to produce anything! You’ve never been out of college! You don’t know what it’s like out there! I’ve *worked* in the private sector. They expect *results*.”

While at the University I have had the pleasure of working with some of the finest minds around. I have collected real-time coastal data at Scripps Institution of Oceanography. I have built distributed data grids for the Biomedical Informatics Research Network. I have worked on the Storage Resource Broker at the San Diego Supercomputer Center. I have architected data repositories and web applications for the Network for Earthquake Engineering and Simulation.

There is a culture within those ivory towers that I will miss. A spirit of intellectual pursuit that is a part of my very being. I am at heart an engineer as well as an academic. I am uncertain of what awaits me, but curiosity killed this cat.

I began a job search within the last couple of weeks. The initial response was tremendous! Since I completely missed the .COM era I was unprepared for the barrage of offers and interviews. In the past, I was lucky to get one interview. This time around I had a half dozen offers. I ended up creating a decision matrix with weighted discriminators to help me determine the best position.

In the end, I have accepted a fantastic opportunity with Qualcomm. I will be working with the MediaFLO group architecting the IT infrastructure, business intelligence, and data warehouse. Of course, as Ray Stantz reminds us: “they expect results”!

Balut!

Eating four hard-boiled fertilized duck eggs followed by a bottle of beer isn’t as bad as you’d think. Never mind the four dead baby ducks, four hard-boiled eggs is gross even without duck fetus!

I took tons of pictures while in the Philippines and also in Hong Kong, more on this later, but for now, please enjoy the galleries!

Centrifuge Earthquake Modelling

Visited Troy, NY (near Albany) for the NEES/UCDavis/RPI Centrifuge Workshop. Learned how to use the RPI centrifuge for earthquake engineering research, and learned more about data acquisition and sensors (LVDTs, strain gauges, pore water pressure sensors, etc) than I’ll ever remember.

Centrifuge Panoramicview pictures

Information Zen

Trying to keep up with Google Reader, and watching my inbox fill up faster than I could read, I realized that access to this wealth of information has consumed my soul! Don’t get me wrong, I love information, and lots of it; like when you’re walking in a library through seemingly endless knowledge and ideas. There’s a welcoming solace between the stacks of books I may never get a chance to read. The vastness of information is humbling; knowing that there’s more great ideas than I’ll ever have a chance to experience.

But unlike the library, my inbox, calendar, and growing list of tasks is not something I can easily ignore. It’s not uncommon that we spend an entire day just catching up and reading about the work we are supposed to be doing.

Think about it, how much of your day do you actually work? I don’t mean thinking about or planning your work, but actually doing something with a net result of work being accomplished. How often are you talking and thinking about some new idea versus implementing? Unless your work is incredibly boring and stupid, you probably spend a majority of your time thinking and planning. Thinking about new ideas is incredibly taxing. We often find ourselves overworked without having done any work.

This is the bane of our brave new information world. We have become more reactive and less focused. We bow to our inbox trying desperately to prioritize the enormous list of inquiries, tasks, complaints, and male enhancement spam. I’ve occasionally adopted triage techniques as a way to get through the massive volume of work collecting in my inbox. This is a valid organizational approach to handling an overactive inbox, but “triage” speaks of a crisis: not enough doctors to treat injuries from battle. This is a wartime solution applied to what should be a calming moment in my office.

So what do we do about it? Maybe we throw our computers away and learn to live off the land the way our agrarian ancestors intended?

Since I don’t know how to hunt or gather, at least not well enough to sustain my interest let alone survival, I’m stuck in front of the computer. In order to survive I sit in front of my inbox.

You have either to abandon it completely, or bow down and hope not to be overwhelmed. For survival, we learn new ways to manage, we learn not to assert control but to let it flow. There is an art to managing information, but it comes at a price.

Personally, I’ve become so good at managing information that I can quickly empty my inbox, updating my list of projects and actions, and scheduling appointments and travel. The key is to let the information flow, it is an ocean outside of your control.

Managing information is like surfing. Imagine a surfer trying to force the waves to behave, the surfer would most likely drown. The information in front of you is no different, you are just surfing and hoping not to drown.

Panoramia!

Recently, my work was gracious enough to purchase additional monitors for my workstation. I guess all my ranting about increasing productivity with multiple monitors payed off! Now I feel obligated to get more work done, but first I have to waste time by looking for panoramic wallpaper!

You’d think there’d be lots of tri-monitor wallpapers out there, something 4800×1200, but after an extensive search the results were entirely disappointing. It turns out I wasn’t searching with the right keywords. Don’t bother searching for anything with the word “wallpaper” or “multi-monitor”, instead just simply google “autostitch”.

That’s right, autostitch. Autostitch is a small program created by Mathew Brown, a PhD candidate at the University of British Columbia. Autostitch is based on research by Brown and his supervisor Professor David Lowe. The program automatically recognizes similar images and “stitches” them together to form a seamless panoramic. Despite the stigma of being “academic software”, autostitch is a revolution in creating panoramic images.

I have always avoided the desire to take dozens of pictures in a wide panoramic. Think of those idiots on top of the Eiffel Tower taking a series of photos in a slow rotation. I never thought there was a point to that activity, not without expensive cameras capable of capturing a real panoramic image. Autostitch proves that even cheap digital cameras can produce amazing panaormic images, perfect for my multi-monitor display!

Here is my first attempt with Autostitch:

Solana Beach – near my house

What’s more, I even managed to produce a great image of a simple street corner:

I’ve linked to the full-size images for those of you with tri-monitor displays looking for a 4800×1200 pixel wallpaper!

Overall, I am in awe of the quality of autostitch. Photography often fails to capture the simple beauty of an ocean, or a mountain, or even a street corner. Beauty is often found in the full breadth of view, the vastness of contrast between a simple street and an ethereal sky. Autostitch enables anyone with a camera to better capture the experience of a breathtaking ocean, or even a breathtaking walk around the neighborhood.

And now back to my multi-monitors!