Sunday, December 27, 2009

It’s All About Accelerating In A Curve

Some years ago I met an interesting person who had been a fighter pilot in the Korean War.  At that time, the jets (their first regular use by U.S. combat troops) didn’t have radar to see behind them: they had rear-view mirrors, like a car.  As he put it to me, “I don’t know anyone, including me, who didn’t have brownout underwear when they saw a MiG in the rear-view mirror.”

At moments like that, it’s important to be able to turn quickly while continuing to accelerate.  That’s the message that John Boyd, a fighter pilot extraordinaire, took away from jet fighter dog-fights, and turned into a strategic theory that was the basis for the second Iraq War: whatever one thinks of many aspects of that war, the initial invasion was an astounding success in terms of lives not lost on both sides.  For a more about Boyd and his theories, check out Robert Coram’s book.

Someone’s On Your Tail

And that someone is your competition.  If you can’t accelerate in a curve, you can kiss your product good-bye.  So let’s see what allows a software development company with a product (an ISV: Independent Software Vendor in the industry parlance) to accelerate in a curve, and conversely, what gets in the way.

The blocks to quickly changing direction in writing software are technical and organizational.  We’ll look at each of these.  But first, let’s identify what technical and organizational challenges have in common.

It’s All About Managing Complexity

If the software you are writing were not complex,  you wouldn’t need a team to develop it, and so this discussion would be pointless.  So we are starting with a given: the software is complex, at the level of the domain (the field of business, however defined, for which the software is some sort of solution), the business rules that govern data, and the User Interface (UI) that allows, and guides, interaction with the data.  How you manage those areas of complexity will determine whether your organization can change directions and get to new places fast enough to stay ahead of the competition.  The two areas where you can most influence the management of complexity are in technology, which can help us manage information in all its various forms, and in organizational design, where people interact with information.

A Good Workman Has The Right Tools

The old saying “a poor workman blames his tools” is, well, old: in software development, you either have tools that support quick and accurate development, or you don’t.  Tools that support quick and accurate development have (at least) the characteristics listed below.

  • Application development and runtime behavior is directed by metadata: the behaviors are in the framework, and get triggered by the metadata specified by the designer.  Contrast this with having to embed calls to methods, or worse yet embedding custom code, whenever 95% of common actions need to occur.
  • The application development tools allow for quick, interactive debugging, including the ability to instantiate objects and execute programs while stopped at a breakpoint.  The principle here is getting instant feedback from experimentation.
  • The application development tools must allow easy development and integration of custom-built wizards and builders that interact with the design objects currently opened in the design environment.  Repeated actions must be automated.
  • Related to the point above, the tools must allow easy programmatic opening of any object in the design environment, and programmatic closing/saving of any changes made. It’s all about automation.
  • The application environment must allow easy control over the build process, through normal language commands, so that a custom build process can be crafted to the individual situation of the r&d team.  It’s all about automation, crafted to your team’s needs and style.

Most development systems do not meet the criteria above.  Visual FoxPro combined with any of the application development frameworks available for VFP does meet those criteria.  And the VFP Developer Studio for .Net will meet these criteria.  Anything embedded in Visual Studio, even VS2010, will not meet these criteria: creating a build in VS is a bear, in spite of the various improvement in packaging.  Until the COM-nature of the design environment changes (the editor is in WPF in VS2010, but that’s not where builders and wizards shine), it’s going to remain a bear.

In addition, tools must be used to facilitate team knowledge and interaction, to wit:

  • Software tools will exist to break work down into projects, issues, check-ins, builds and releases.  Projects have features, and implement features through issues; issues can be implementation or bugs, but in either case have check-ins; builds have issues fixed and features complete; a release is a build.
  • All in-house information pertaining to development must be searchable with a tool at least as good as Google (tm) search.
It’s All About The People, Too

What is described above is a customized system for developing software.  Systems require people, however.  Let’s look at what the various people in the know have to say about the people.  A lot of this comes from Joel Spolsky, who has written extensively on this subject (since it directly affects his bottom line, and he is bright enough to realize this).

  • When a team gets above 10 people (I would argue 7, based on the group dynamics differences that apply between small and large groups), the complexity of communications begins to hamper efficiency.  The “2 pizza team” concept applies here: if it takes more than 2 pizzas to feed the team (a concept used by Amazon, and also reputedly by Apple), the team is too big.
  • Everyone on the team has to be really good (one of those 3% to 5% who get work done).
  • Decision-making comes by reaching agreement, not by fiat: when team members are forced to come together in decisions, not only do that have a stake in the outcome, but differences that would affect team functioning get ironed out in the reaching-agreement process.
  • Software developers have work arrangements conducive to deep concentration and focus.
  • Software developers who are kept on the team are well compensated, because only the productive ones have been retained.
  • Software developers are encouraged to expand their knowledge constantly (e.g., by being given Amazon cards for book purchases, and by having a budget for software purchases).
  • Software developers are provided with workstations that promote development efficiency: big, fast, and with multiple monitors (2 minimum, 3 preferred).

And Then There’s Time

My first “job” in software development was at a company that wanted to convert its product to Windows.  They hired me halftime, but the pay was as much as I had made as a psychologist (not much <s> – the economics of being a therapist is that the more efficient and effective you are, the less money you make), and it came with insurance.  Then it turned out they figured the job shouldn’t take more than 6 months.  Oh well…

Good software takes a long time to become good software: Joel says 10 years. Why does it take this long?  It has to do with articulating the domain, that is, in expressing (through the UI and through the business rules that constrain and affect data) that combination of knowledge of the domain, and knowledge of the users, in a way that best facilitates the interaction of the two.  Nobody can get it right the first time, because by the nature of complexity (read Gregory Bateson on this subject), we have to learn to articulate it through experience.  And doing it in one platform does not at all guaranty that one can do more quickly in another platform: what it does mean is that there is a leg up on getting to know the domain, which means there will be more complexity to program, not less.

That 10-year period of getting to deeply know complexity is something familiar to me from my time in psychology: it took about 10 years from the time of my internship until I felt I had a real handle on what was happening in my therapy sessions.  I did good before that, and I learned more after that 10 years, but the essential mastering of complexity took 10 years.  Software is no different.

Of course the software will be out the door in less than 10 years: 1 to 2 years, depending on the starting point, is reasonable.  Plumbing the depths of the domain and translating that into what fits the various user types takes time, however, and that’s where the 10 years applies.

And if your organization can not accelerate in a curve, it won’t be there to make that happen.

Friday, December 25, 2009

Why The Entity Framework Is A Crock

In an episode of The Bob Newhart Show, Bob, who is a psychologist, starts to wonder whether he is doing any good for his patients.  He goes back to the college where he got his degree to visit his mentor, Dr. Albert, played by Keenan Wynn. He finds Dr. Albert in the stacks at the library, and ask something like this: “Sir, I have been wondering if I do any good for anybody, and so I had to ask you, do we do any good?”  Dr. Albert says “Bob, I want you to write this down.  Are you ready?  Bob, it’s all a crock.”

Now, being a practicing psychologist at that time, it cracked me up.  And I think it’s a good concept to carry with one as one confronts the “latest and greatest.”  Perhaps it’s all a crock. <s>

The Entity Framework (EF) is the latest and greatest advance in .Net.  The idea has been around in Java for a while, so I’m not picking on the EF alone.  It’s the need for one that’s the problem, you see, not the thing itself.

What is the need for the EF?

Here’s a quote from the MSDN article which provides an overview on the EF.

When one takes a look at the amount of code that the average application developer must write to address the impedance mismatch across various data representations (for example objects and relational stores) it is clear that there is an opportunity for improvement. (italics mine)

What’s this “impedance mismatch” thing?  Well, you see, when .Net was designed, it was not designed around data.  In VFP, if I BROWSE a table, I don’t have to tell each control in the columns of the grid what type of data to expect.  It just works.  In .Net, I do have to tell each control in each column (or the equivalent thereof) what type of data to expect.  And guess what: you have to type every blessed use of data in the code, unless you are using a code generator (and there’s a good reason we gave up code generation in Foxpro very early on, as much as I enjoyed and appreciate Russ Freeland – the creator of Scrimage, the screen design code generator for FoxPro 1, before FoxPro had a form designer). The Entity Framework fixes all that, sort of.  You design your Entities in code and then generate the SQL Database from that; or in the next version of EF, I believe, you will be able to create your SQL Database first.

I put this on a par with the VMWare (tm) corporation admitting, in its own marketing materials (for ESXi, their bare-metal hypervisor) that original VMWare Server is not made for data-intensive applications.  If only they had told us this 5 years ago, right?  Why is it on a par?  Well, why would you have to correct the impedance mismatch (a fancy term for “we screwed the pooch for working with business data applications”) if that very same impedance mismatch had not been created in the first place?  As for “room for improvement” – oh, how it must gall to be forced to speak corp-speak.

So in Visual FoxPro (VFP), about to be abandoned by Microsoft (tm) except for extended support (as of 31Jan2010), we have no impedance mismatch, and in .Net we do, and now need the Entity Framework. 

Here’s An Example Of Why The EF Is A Crock

This very recent blog post by Rick Strahl, who straddles, expertly, the VFP (think Web Connection, his product that designs in VS.Net, and runs on VFP) and .Net worlds, describes a discovery he made about an impedance mismatch (my term, not his, but that’s what it is) between Linq2Sql and the Entity Framework.  The difficulty he found involved the differing requirements of L2S and the EF when it comes to the link table in a many-to-many relationship.  Quoting Rick: “It’s ironic that LINQ to SQL *requires* the PK in the middle – the Entity Framework requires that a link table have *only* the two foreign key fields in a table in order to recognize a many to many relation. EF actually handles the M->M relation directly without the intermediate link entity unlike LINQ to SQL.”

Now, when I’m designing with xCase, I have a choice: I can use a compound key of the 2 foreign keys (and even throw another column in there, like “sequence” or “priority” while I’m at it), or I can add a surrogate key.  I do the latter when I am going to be propagating the PK from the link table elsewhere. The problem with ORM’s in general is that they dumb down the relational concepts. thinking they are making the data model more accessible to the masses.  In fact, of course, it’s much harder to work around the exceptions caused by dumbing down than it is learn how to use the full relational model.

Why would I want to design in a dumbed down framework that doesn’t give me all the choices?  The answer, as I described above (the “impedance mismatch”), is that anything is better than nothing, right?  Well, guess what: dynamic data (VFP, VFP.Net) is better period: compared to working with dynamic data, the EF is a crock.

And Then There’s MetaData

Now, the MVC (Model View Controller) is supposedly a great addition to the .Net data arsenal.  What does it offer?  This explanatory article summarizes it well: it separates the metadata (model) from the display (view) which in turn is separated from the user input (controller).

I am going to focus here on one key aspect of the model, the metadata.  Where is this metadata (which after all, is data about the data, right?) stored: unbelievably (to a VFP guy) it’s stored in the code.  <gasp!> In the code!  Have they never heard of a data dictionary containing metadata?

Now, our (xCase, xCase2VPM, Visual ProMatrix(tm), ProSysPlus Libraries) metadata holds both the business logic metadata (e.g., must not be blank; field triggers, etc.) and the user interface metadata that is device independent (e.g., enabled; visible; etc.).  The latter corresponds to the device-independent piece of the Controller.  The VFP.Net version will have the same capabilities, although expressed quite differently in inner workings.  And where is this metadata stored?  In every case, it is stored in tables: you know, where data is supposed to be stored? <s> And then, of course, we have classes that take the metadata and apply it.  The classes can be subclassed at the appropriate places to apply to different devices, something that Frank Camp and I demonstrated in 2002, running the same app against dbf’s, sql server, and over the web to Web Connect and .Net web services, which in turn were running the same oApp object as the fat client. 

For a development system that has difficulty using tables to store data, the method of implementing the MVC may make sense.  For those of us with friendlier (to the developer) development systems, the method of implementation is unmanageable.  So, I’m sure the MVC is a great help for an embedded data system of metadata.  It’s just not much of a help when metadata is treated like what is is, data; and when the classes created are generic to utilizing the metadata, and therefore are easily subclassable (at least easy in VFP and VFP.Net) to work with different device contexts (fat client; empty client; web service; web page, etc.).  Compared to working with genuine metadata, and subclassable generic metadata handling classes, the MVC is a crock, also.

By Themselves, EF and L2S and MVC Are Quite Clever

Nothing I’ve written above should be taken to mean that these 3 data tools for .Net are not clever or not well-written.  At the level of their details, they are all far beyond my limited comprehension.  It does not take an engineer, however, to conclude that a Hummer is not the ideal car for errands about town (or perhaps anything, for that matter).  These tools are needed because data access is indirect in .Net, and because data types are static.  The tools are a symptom of the problems, not the problem.  And, of course, until VFP.Net, there has been no viable alternative in .Net comparable to the seamless data utilization in the original VFP.