Time and Pressure

We all drudge through life trying to survive, but why? What is so important for survival? What makes any of this life worth living? Is it a biological imperative to keep breathing for as long as possible? Or is there something we ought to be doing with our lives that motivates our survival?

The human condition seems to impress that there is a reason for our existence; a reason to survive if you will. We put ourselves in a survival mode in order to live by what we consider to be “good” or “right”. Many of us never really think about what that means and instead just hope to make it through the day without stress.

A steady pressure over time has shaped humanity into creatures compelled towards survival. The same evolutionary pressure has shaped our limited capacity for reason and logic, and that limited logic tells us that there must be a reason for our survival. We plan goals and imagine the desired outcomes. We are goal-oriented to fill an evolved need for a purpose in our life. Whether we are conscious of the goals or not we carry beliefs of what our life is suppose to be…

You plod though life towards your goal and whether you like it or not that goal becomes your purpose. If the goal has not been achieved then there exists a dissonance between your reality and your purpose; a continuing source of stress that will push you towards your goal. On the other hand, if you have achieved your goal then you’ve also achieved your life’s purpose; this can be depressing if you’re so successful that you achieve all of your goals!

This goal-oriented behavior results in either stress or nihilism. I propose an alternative to the goal-oriented life that is hopefully stress-free and nihilism-free!

Time and Pressure

Given enough time and pressure we have evolved into the goal-oriented and stressful species that we are today. Within an individuals life, all of their accomplishments are the result of discreet actions towards a goal. The goal is often emergent from the day-to-day actions. Most goals change over time, evolving to better suite our needs from the actions we’ve taken.

I would argue that you should never be concerned with a desired outcome or goal. There is no perfect state of existence in life but there is an ideal flow that is readily achievable. Your goals are often expressions of your own ideals or virtues and they tend to change. Ironically, the closer you get to your goal the more you learn about it and hence the more the goal will change.

Imagine a river rock that is perfectly rounded and perfectly smooth. This perfectly round and perfectly smooth stone represents a goal. In our goal-oriented perspective a rough non-rounded stone would be a source of stress compelling us to chisel and polish until it’s “right”. But a river rock wasn’t chiseled or polished; it was merely shaped over time by the pressure of the rivers water.

Many people let the water dry up and stare at rough stones planning just how round and smooth they’re suppose to be and never really make any progress. If you let the water flow you’ll always make progress.

Rather than focus on the specifics of your goal, focus on the time and pressure in your life and simply maintain that ideal flow. Improve with every step. Continuously refactor your life. Love every moment of the ebb and flow of this precious life. Remove focus from the goal and instead focus on the time and pressure. Make your life better, and better, and better without every worrying about what it’s suppose to be. You’ll end up with more accomplishments than the goal-oriented person would know what to do with.

An aside: Time and Money

Time and money are non-comparable. Money can be the result of time and pressure, but do not be so foolish as to compare time with money. Any fraction of time is worth an infinite amount of money. Or as one of my friends recently discovered: Time > $

San Diego Wildfires

Here’s a normal mid-afternoon day in my neighborhood

It’s pleasant. Fair sunny San Diego weather.

But then on October 22, 2007, it looked like this

Something seemed to be wrong.

Where you would expect the sun, there was just this sad orange moon

Indeed something was wrong.

As I was driving out of my neighborhood, I couldn’t help but to think that if the coast looked like this

then the rest of the city must be in flames.

Initially, I had invited friends to stay at my house since I figured it would be one of the safest places being so close to the ocean. I was wrong. The winds were blowing the fire directly towards the coast. I needed to evacuate. The air was dangerous to breath as I was right in the path of the smoke.

Fortunately, my house was safe when I returned home a few days later. There was ash and debris but otherwise everything was fine. Many people were far less fortunate.

Despite the tragedy of the situation I couldn’t help but to marvel at this moment; the dramatic beauty of the reddened sky juxtaposed with my expectations of a serene blue. And my intuitive response to help others quickly turned to needing to call on others for help. How quickly life changes… a miracle in every moment.

Aloha Akamai

Aloha –
Welcome, greetings, farewell, goodbye, love, mercy, … (Hawaiian)

Akamai –
Smart, clever, expert, skill, witt, … (Hawaiian)

I started a new job with Akamai Technologies. They have a small San Diego office and from what I can tell, it’s a very bright and motivated team with technical aptitude to match their creativity. A positive group with complex challenges that I’m excited to dive in to!

Buddhism has a concept of “right thought”, of freeing your mind of lust, ill-will, and cruelty. I’ve been thinking a lot about this lately. We surround ourselves with people that often reflect our own dispositions of thought. The environment we find ourselves in, for better or worse, is of our own doing. In many ways, this phase of my career represents a welcome “right thought” and I realize that “right work” is a necessary path to that state of mind.

And to my friends and family who have never heard of Akamai; you use their services all of the time without even knowing it. Primarily known for their content delivery network, which is a grid of compute and data resources covering multiple Internet backbones optimized to deliver content, to you, the end user. In other words, when you’re watching some cool video on myspace or previewing film trailers on apple.com, the content you’re watching is being managed by Akamai.


Organizational Intelligence

Organizational intelligence is the measure of an organization’s ability to comprehend and conclude knowledge relevant to its purpose. In simple terms it is the performance of a group or organization measured as a whole. For example, Basketball and Hockey require a high-degree of organizational intelligence in order for a team to win. The success of the team depends greatly on the team’s ability to work together, rather than based only on individual performance.

For software development, and most business organizations, the goal of an organization is to function at the highest levels of competency across all individuals. Unfortunately, a low organizational intelligence means that an organization is only as smart as the combined incompetence of all individuals. The net effect is that simple problems are harder for the organization to solve than they would be for a single individual.

Consider the following overly simplified skill summary:

In this situation, there are three developers and a single project manager. Of the three developers, one is a database developer, one an application developer and the other a WEB/UI developer. This represents a fairly standard cross-functional team, though greatly simplified.

In order to achieve the goal this group must perform at their combined strengths. This creates a positive emergent behavior where the group dynamic is able to achieve something that no one individual on the team could perform. This is a case of positive emergence and assumes a well-functioning team with a high degree of organizational intelligence:

On the other hand, the team could be functioning in an overly competitive or chaotic manner leading to a negative emergent behavior where the group dynamic is only as competent as the combined incompetence of all individuals:

This type of scenario is fairly common in software development and engineering. Concepts such as Analysis Paralysis, Design by Committee, and Feature Creep usually result in a low organization intelligence with a negative emergent behavior.

Emergent Behavior is the result of simple entities forming more complex behaviors as a collective. The complex behavior of the group is not a property of any single entity. Group/Organization behavior is often emergent, not the fault or credit of any one individual but the dynamic formed by the group itself.

Typically smaller teams like the one in the charts perform more on the positive side than the negative. However, as teams scale the challenges quickly become less about specific skill sets but instead more about understanding and exploiting the emergent behavior of the organization in order to achieve the desired business goals.

Consider the overly simplified version:

The individuals possess the correct skills necessary to achieve the goals of the organization and ideally will surpass those goals through a positive emergent behavior of team. On the other hand, the areas of incompetence are drastic enough that given a negative emergent behavior the team will surely fail despite individual efforts to the contrary.

Illusion of Stability

Sometimes we plant our feet firmly in hope for stability. Kids, mortgage, bills; the excuses are endless. Yet for all our static behavior life has a tendency to knock us right on our ass. If you’re alive and breathing, then there is no such thing as stability. Not only can your life change at any moment, it’s changing at every moment.

But what about all of the compromises? As stability becomes a value in life, where do we rank that with our other values? Everyone has a ranking of values. Some people will maintain honesty as a value even over friendship. Other people value friendship more and will gladly lie to preserve their friends favor. So what values are we willing to compromise for stability?

Maybe you’re a few months from vesting or just a couple of years from retiring? When do you stop rocking the boat and hope things settle down?

I hope the answer is never. The boat is always rocking whether you want it to or not. The water is guaranteed not to stay calm. Stability often appears as an imaginary value. The problem is not stability itself, but with the values that you compromised for something imaginary.

In my own life and in the lives of people around me, I’ve seen compromises that sacrifice integrity, competence, and honesty all for that illusive stability. I’ve watched good people do bad things, and observed my own bad behavior to make compromises in order to maintain a deceitful status quo. A lifetime of compromising your values and it’s no surprise where otherwise good people with well-intentioned values become bad people based on the ease with which they act out against their own principles.

I can’t speak of a perfect solution. It’s difficult if not impossible to just simply erase stability from our core values. The irony is, we want stability to protect ourselves and the people around us, but as we compromise in favor of stability we end up hurting ourselves and the people around us. My idea is to instead focus your life on the process and not the outcome.

We tend to focus on what we want and our minds fill with imaginary outcomes. Some people want kids and a house, and once they get them they want to preserve them, they want stability. I’ve seen people sacrifice happiness (which in my opinion should be the highest order value of them all) in search for stability. Try instead to focus on the process; on living in your home and raising your children rather than outcomes like kids, house, and bills. Or focus on living a good life rather than imagining a future retirement. Perhaps then we won’t compromise our values and our actions will reflect our good intentions.

Like waking from a slumber
I open my eyes and see clearly
The world around me
My place in it
The present
Where have I been?

Goodbye Qualcomm

It’s been just under two years and after a whirlwind of experience and excitement at Qualcomm I have decided to move on.

This was my first foray into the private sector and it was a blast! The academics had it completely wrong. I was warned that I’d be a cog in a machine. Not even close. The academic machine is a rusty axe compared to the shiny chainsaw of the corporate world. Each has its cogs. You know who you are.

There’s an old saying that originated from the Soviet Union: Initiative is Punishable (“Initsiativa Nakazuima”, thanks Katya!). I’ve learned in my career that this is true. Initiative is punishable, but as a friend of mine pointed out and looking at my own career I’d say that this same initiative has been rewarding.

Our lives, whether at a job or elsewhere, are filled with challenges. Sometimes we win, sometimes we don’t, but we should never lose focus of our ideals and our integrity. It is our ideals that inspire innovation. And it is integrity that keeps us on the right path. Often times we find ourselves so lost in details that we forget this larger context.

For anyone reading this far and finding any sense to this post: remember that we are defined not by our words but by our actions, and all too often the dissonance between our mouth and hands is the real source of our stress.

The Good, The Bad and the Ugly of Scrum

We all hear about and we all love it: the Rugby-inspired software development methodology known as Scrum. It’s fast becoming an industry buzz-word and causing many project managers to question their Gantt charts. For all the hype, what is the reality of Scrum?

Scrum is an agile-based software development methodology for project management. It is characterized by a prioritized product backlog that lists new features. Work is completed and delivered in time-boxed iterations known as sprints (e.g., two week iterations). Scrum teams are cross-functional and typically number 3-7 people each. Iterations begin with an iteration planning meeting and end with a retrospective to review what worked and what didn’t.

During a sprint each scrum team gathers for a daily stand up, which is a short meeting where each person describes what they did since the previous meeting, what they’re planning to do now, and any impediments. The team is self-organizing leveraging our instinctive behavior to work in small groups. The Scrum process is facilitated by a Scrum Master. That title is a bit of a misnomer since the Scrum Master carries no authority and is instead responsible for blocking any distracting influences that could disrupt the teams progress.

The principles of Scrum are well defined in the wikipedia article as well as in the book Agile Project Management with Scrum by Ken Schwaber. You can also shell out ten grand for an in-person experience with Ken. There is nothing like an expert talking about the work that you should be doing. As some of my friends and co-workers like to hear me say: Get back to work!

What’s so Good about Scrum?

Delivering working software. Working software is where Scrum really shines. It’s proving to be an excellent implementation of Agile Software Development with core values such as customer satisfaction and individual interaction.

There are four core values to the Agile Manifesto:

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan

These are all valid principles that are easy to ignore and have proven to be hard learned lessons despite how obvious they may seem. Projects typically fail when they ignore these principles. Good documentation has never compensated for crap software. Try telling the upset customers that you gave them exactly what they contractually signed in the Service Level Agreement. The principles of the Agile Manifesto should be held as software engineering law.

Scrum provides a very effective methodology that ensures these principles through an empirical approach to software development that embraces and encourages change.

If it’s so great, what’s the Bad news?

No one likes to admit this, especially not Scrum advocates like myself, but Scrum fundamentally conflicts with traditional PMO. It’s an interesting round of cognitive dissonance to watch a PMI-certified project manager attempt to rationalize Scrum. There are of course several ways to deal effectively with this dissonance but understand: these are fundamental differences which are not philosophically compatible.

Scrum will go far in delivering working software, but what about managing roadmaps? Better yet, what about resource allocation and God-forbid budget forecasts that are needed before a project starts? We need money and people to start a project and we’d like to know roughly how much something will cost before we agree to invest. In a perfect world we would know these answers and we wouldn’t need Scrum. But Scrum exists out of necessity from the failures of so many software development projects and reminds us that the entire enterprise of software engineering is often times more like scientific discovery than building construction (which is where PMI originated, and rightfully so).

The Ugly

Scrum delivers working software in chaotic environments. At the same time Scrum is a symptom of a larger problem in software engineering such that many software projects cannot be managed like construction projects else they face increasing technical debt, unhappy customers, declining quality, going over budget, and missing deadlines.

Migrating to a Scrum methodology typically has an effect of providing early visibility to problems. The wisdom being that it’s better to know that you’re failing one or two months into a project rather than years. While this makes sense and this transparency is a very valuable aspect of Scrum the reality is very ugly.

Transparency to critical problems often times stems from the fundamental conflict of traditional PMO and Scrum. These are problems that are unfortunately outside of the realm of Scrum or software engineering. In this situation it is easy to attack the symptom (Scrum) than it is to address the underlying issue, which is unifying project management (roadmaps, budgets, resource allocation) with the software development.

Visibility? Be careful what you ask for! If a project is going to fail, maybe it’s better to let it fail naturally than to induce ulcers in the Scrum Masters and the development staff. Remember: a good Scrum Master will be a burned out Scrum Master in most environments.

This isn’t an easy problem to solve, but done wrong Scrum can create an emergent failure. Take the following anecdotal quote from Brad Wilson:
Scrummerfall. n. The practice of combining Scrum and Waterfall so as to ensure failure at a much faster rate than you had with Waterfall alone.

On the other hand: Scrum, done right, has the potential for an emergent success given iterative and continuous improvement. A potential method for solving the real problem is to exploit the emergent behavior of a system.

Emergent behaviors are difficult to track, but analyze the existing software, processes, and development and determine whether or not they are evolving appropriately. A successful development process should continually improve the same way the code itself should continually improve. The process itself should be agile, responding to change to better produce working software.

Better yet, the individuals and interactions should be agile. It is the people that must respond to change.

Evolving Database Schemas

Master your Domain

It amazes me that while database schema evolution is one of the most critical factors in software development it’s also one of the most ignored and least understood aspects. Pretty much every interesting software development project requires an evolving database schema. From renaming tables to modifying relationships it’s as critical as any piece of source code but is often the least cared for part of any project. Your schema is what maintains order in your data, arguably this is the most important part of any enterprise (organizing your data)!

There are mountains of excellent (and practical) resources on how to manage software development projects. I can get certified in SCRUM or nearly any Agile-based development methodology but I can barely find any useful websites on managing my database in an Agile environment. When your development process encourages change this applies to your data modelling and schema design and not just application development.

There are some excellent articles from Fowler and Ambler that provide a good starting point. However, I found a definite lack of details and practical advice in those articles. It goes without saying that I want to run regression tests and version my work, but a database is distinctly different in that I have to deal with all that existing data and can’t exactly redeploy a schema while preserving the old data (not easily anyhow).

Existing persistence strategies often fail to accommodate Agile-based development leading to poor design choices in data modeling. In this article I’d like to explore practical solutions to Agile database development. Like anything Agile, there’s no perfect solution, but this is a pervasive problem across all interesting software development projects and I hope the discussion alone will yield better solutions.

To start, there is one important observation that I’ve found to be true across various software development projects: If the code smells it’s likely that the database smells worse! Let’s examine some common database smells:

  • Inconsistent relationship strategies; when your ER diagram starts looking like spaghetti and every piece of business logic introduces a different convention you’ve got a problem. You have object tables and three possible types of relationships (1-1, 1-N, N-M), your data model is only as complicated as you make it; pick a strategy for each of the three types of relationships and stick to it. I liken this to using GOTO statements in software, it’s unacceptable.
  • Inconsistent object model strategies; this is often the impetus to change your relationship strategies, that is, when I have inconsistent strategies for object tables it leads to very confusing relationships. You’ll see one table with a varchar(20) NAME and another with a varchar(16) NAME, is this the name of the object or does the object contain a “name”? Use a consistent strategy for tables as well as concepts like status, timestamp, IDs and alternate keys (such as name).
  • Inconsistent naming conventions; STAT_DATE, STATUS_DT, or STATDATE? Pick a convention and stick with it!
  • Inconsistent usage of the same column; a common example is a varchar column named TYPE that means different things to different applications. I’ve noticed that even good data modellers make this mistake.
  • Overloading fields; things like a comma-separated list of values where only the application knows what each value means. Don’t use a relational database if this is how you model – text files may work better!
  • F normal form; Johnny just took a class on database design and learned about normalizing a database and now you have 175 tables in what he claims is 6NF! There are appropriate times to denormalize just as often as there are to normalize.
  • Know when to OLAP; why are there summary tables attached to each of my transaction tables?
  • The Cauldron of Data; this is the crux of the problem, everyone is so scared of the data that they lose control of the schema and treat it like a bubbling cauldron too paranoid to make any significant changes out of fear of breaking a legacy application. This is the end result in any application where they didn’t manage their evolving database schema.

Let’s talk about some solutions!

First of all, this is a developer problem! Don’t expect your DBA or Hibernate to fix this for you – if you’re a developer this is your problem. This leads to the central theme of how I propose database evolution to be solved: Your development methodology must cover application and database development.

If your application depends on a database, then your development methodology better cover both application and database development! I know, you like building the code and leaving the responsibility of the database to someone else. But that brings you back to the Cauldron of Data scenario where you can’t make any significant changes to a schema because it got our of your control. And if you can’t control the schema you can hardly control the application that depends on that schema!

We tend to ignore database development as a way to simplify our application development – I suggest you make the application development suffer by adhering to a development methodology that works with database development! Think of it like this: you’re going to be the databases bitch if you don’t take this responsibility.

I know, this seems like more work from the application side but like anything done right it’s hard to imagine doing it differently once you get your development methodology to cover applications and databases. That said, how does one integrate their database development into a unified development methodology?

Let’s look at the differences between application development and database development (and what needs to change in the traditional Agile-based development methodologies):

  • Databases contain data that cannot be lost; this means you have to migrate production rather than reinstall
  • Data is easy to migrate when your data is not controlling you (see the Cauldron above)
  • Rebuilding your database is like compiling and deploying your code (this sounds like a maven target)
  • Databases should have unit tests, and not just for the stored procedures (more on this later)
  • Map your database development to your project lifecycle goals exactly like you would with application development (say, in Maven 2) but introduce the migrate step in the deploy target.

If I compile and build my application why not build the database schemas at the same time, just like I would with anyother dependent artifact? So, let’s get practical and talk about things you can actually do to accomplish Agile-based evolutionary database design:

Create a Database Schema Change Policy

Keep it simple and make sure you answer how you plan to address schema migrations planned and unplanned. Your process should lend itself to an emergent property of better schema design. This by itself requires you to not only support planned and unplanned schema changes, but to encourage them. Either do a big design up front (not-agile) or encourage change in all aspects of your development (including your data model). I recommend you clearly define a process for planned migrations (migrating from one version of the schema to another) and unplanned patches (critical fixes, the kind you get in the middle of the night).

Bring DBAs in Early

You’ll need their help, and you know it, best to get friendly with them early on – give them a chance to know what you’re trying to do on their database. I argue that you’ll find more resistance to agile development from software developers than you will from DBAs. Most DBAs have been on the front-line fixing smelly database code and are likely your strongest ally. Not only can they help with the development process they can (and should) assist with design.

Use Stored Procedures

Read up (separately) on “End-to-End Architecture”, if your schema is going to change then you better clearly define your endpoints and provide an API-like package to abstract the schema completely. What’s great about stored procedures is that they can (and should) be treated like application code. I recommend two types of packages, consider using a suffix of _PKG and _API. All of your object tables will likely have GET, PUT, and DELETE procedures. These should be autogenerated, if not, write yourself a script or invest into some software to autogenerate CRUDL stored procedures. Each schema should have a _PKG with CRUDL procedures for all object tables. You also have business logic, from complicated transactions to simple procedures like authenticate(user, pass). Procedures that encapsulate business logic should be in packages with an _API suffix and follow the same rigorous design that would be employed for any application API.

The naming convention of an _API and _PKG suffix is unimportant (any convention here would be fine), but what is important is distinguishing between your CRUDL procedures and your APIs that encapsulate your business logic. Once you have a convention that cleanly separates these concepts you now have a mechanism which can completely abstract your schema from your application and best of all, you’ve likely imposed some constraints and standardization on your object tables that lend themselves to easy autogeneration of the CRUDL procedures.

Version your Schema just like you would an Application

Versioning is a given for application code, why should database code be any different? Schemas, default data, packages, grants, everything should be versioned along with ALL other application code. Applications depend on a versioned database, just like any other versioned artifact – I would expect my build to fail if the dependent database for my application doesn’t exist.

Each of your schemas is like an application, and all of the DDLs should be checked into source control and managed as applications! Check in your test data (sql inserts) and you’ll easily be able to define a database-specific unit test environment!

Use the Right Tools

You’ll need more than a modeling tool, modeling tools are great at helping you to visualize your schema, but don’t get carried away. You need to track schema AND default data! Use tools that fit your process not the other way around – write your own scripts if you need, they’re not that hard once you have a working process. Between Maven and some sqlplus scripts we’ve gotten plenty of mileage at my current job with the following scripts:

  • drop_objects.sql; loops through all of the schemas and drops everything, there’s also a delete user approach but with the drop script you don’t have to redefine your tablespace; this script is never run in production
  • create_objects.sql; loops through all of the schemas and creates all of the tables and default data; this script is never run in production
  • create_pkg_spec.sql, create_pkg_body.sql; loops through all schemas and compiles the package specs and separately the package bodies
  • run_tests.sql; loops through all schemas and runs database unit tests, it uses stored functions with setUp and tearDown procedures similar to Junit; this script is never run in production
  • migrate_objects.sql; loops through all objects and runs a per-schema migrate script which is created based on the delta between two different versions of the same schema

Localhost Development

Why else do we have fancy development workstations? Stop assuming Eclipse is allowed to eat up all of your resources – let Oracle do it! The only way to empower your developers to be agile is to give them an environment where they can easily change the database schema!

We’ve gone so far at my current job to support localhost Oracle instances where we checked Oracle into our software version control (along with Tomcat, Java, etc). We tried using the Express Edition but it didn’t support all of the PL/SQL code we were developing so we’ve got the full bloated 10g running on all of the developer workstations (takes about 30 minutes to install on a new workstation). So don’t tell me you can’t run MySQL locally!!