Friday, December 23, 2011

Swimming With The Current

I had a dear friend (met later in life) who, in childhood, would dive into the Niagara River, about 2 miles below the falls, where the current was still very, very strong.  Strong enough, in fact, that at the place he dived in there was an eddy that could carry him 1/2 mile up the river.  So, he would jump in, get carried up the river, swim out of the eddy, and then be carried down the river.  Over and over again.  All without appreciable effort on his part.  Harry was a smart guy.  He figured things out.

Figuring Things Out

For over a year now I have been figuring certain things out regarding the changing scene in software development.  The store starts earlier than that, when it looked as though VFP could be brought to .Net, but that effort petered out, and I had solid reasons to reject the alternatives.  It’s not that there weren’t good tools out there, but rather that they didn’t fit my needs for tools that worked for software development by domain experts, rather than software development experts.

December 2010 was when I realized that The Time Is Now to make the move to web/device-based applications.  This year, for 2012, analysts’ projections are that 90 Million Tablets will be sold (50M iPads; the rest Android).  Add smartphones and the numbers are staggering.  Enterprises are not only accepting these mobile devices, they are encouraging employees to select their own devices and use them to extend work hours, away from the office.

Figuring out how to play in this particular playground has been a grueling exercise in discernment.  What I knew in the beginning was that:

  • Our apps have to run on every major device category/OS
  • We must be able to serve from the cloud
  • The toolset must make it possible to develop more efficiently than we do in VFP, using our modified, patched, and extended VPM framework integrated with xCase.
  • ORM’s suck (compared to working with data in VFP)
  • Statically-typed languages, even with improvements, have multiple disadvantages (e.g., having to use an ORM straightjacket in order to do anything intelligent with data), and few advantages (.Net never delivered on the promise of managed code everywhere; Java is behind .Net in features), relative speed of runtime being about the only one.  “Safety” is often touted for statically-typed languages: but the use of reflection in just about every serious, major statically-typed program negates the principles on which that claim of safety is made.

As I went along, I figured out more stuff:

  • JIT (Just-In-Time) compilers for dynamic languages, with call-site caching, have improved to the point where statically typed languages’ speed advantage is largely nullified. E.g., Python running on PyPy beats C# for speed on some metrics.  The difference is small enough to ignore.  And PyPy isn’t done optimizing yet.  And the JIT’s for Javascript in the browser show similar boosts in speed.  Lua’s JIT, with native co-processing, is fast enough for games with physics.
  • Any tools we choose must be in continuing development, and be in an environment where the needs of users are listened to, and acted upon.  Since companies that are belly-up can’t provided continuing development, that means startups that haven’t reach C-level funding are likely out.  That also means that Open Source projects must have a track record of continuing development, and community involvement.
  • Device-dependent programming may be needed now for games (although Lua’s speed disproves this), but even that advantage is dissipating.  WebGL is a game-changer, removing the advantage of native code for graphics display.  And if Google’s Native Client (NaCl) takes off, your favorite JIT will work on any device. 

A Drumroll Please…

In the category of “backend tools” one language stood out above the rest: Python.  With the data work in Dabo to start from, making our own framework is doable with low risk.  The Python community is among the healthiest in the Open Source world, thanks to how the Python Software Foundation (and Guido, founder of Python) manage community requests (anyone, you or I, can create a proposal in a PEP: it’s then up to us to gain support in the community).

In the category of “frontend tools” a winner emerged over the course of the year: Sencha Complete (Ext JS, Ext Designer, Sencha Touch).  (Sencha is a Japanese Green Tea, and sounds like “sen tcha” in “sen tcha a letter”).  Frankly, it was touch-and-go until late October.  They did not demonstrate staying power until then (when they received an infusion of 26M in VC funds, in a C-level series).  Just as importantly, there was no indication earlier in the year that the Ext Designer was going in the right direction: at the Sencha Conference in September, it became clear that there were in fact going where we needed them to go, a point confirmed to me by an Ext Designer PM. 

Sencha also made it clear, later in the year, that they will provide A-to-Z, soup-to-nuts, coverage of important development points that many other providers of similar development stacks gloss over: themes that match the devices on which they run, packaging for app stores (a la PhoneGap) built right into their product, and speed of device display, among them.  It’s the right attitude, and they have the resources to put that attitude into action.  Ext Direct handles, asynchronously and with easy integration to the web backend of your choosing, method calls from the Sencha client.  And, of course, there is no royalty payment leeching scheme.

Honorable Runner-ups

  • Mono/MonoTouch/MonoDroid: Miguel and team were dismissed from Novell when it was acquired, and then in 2 weeks put together Xamarin.com, and are cranking out great release after great release.  If there were no dynamic language alternatives, it would be worth looking at.  Their work with Google Native Client (see below on: The Future) is exciting.
  • xDev is Java in a rapid application development model.  While it shows great promise, uptake appears to be small, based on the size of the xDev conference attendance this Fall.  Along with the rise of Jython (see The Future, below), it bears watching if for some reason you need to deploy on the JVM.
  • There are a cajillion Javascript-based UI sets out there.  Some, like Pyjamas, build on the Google Widget Toolkit, which itself received a lot of loving from the Google devs during the year.  This is a great approach, and I would have like to go this way because it would have meant programming Python, front and back.  In the end, there was too much development left to do, in order to get to the point we need to reach, to be economically possible.
  • Others, like Dojo, have a good community, have funding from a stable source (IBM in Dojo’s case), and are reasonably complete.  But none of them are complete enough for me to set up a frontend development environment that allows me to graphically design my UI, today.  Only Sencha has that (and Designer 2 will have much more of what we want).
  • Cappuccino was an early contender: it is a clone of Objective-C, using in OSX and iOS apps, but in language they created, Objective-J, which compiles directly to Javascript.  I met a core developer, Randy Luecke, at a Starbucks near University of KY in Lexington, where he goes to school.  They created some beautiful programs with it, e.g., the defunct 280slides.com.  Then they were bought by Motorola Mobility.  Then Motorola Mobility was bought by Google (still pending judgment from the Justice Department, I think).  In any case, making an equivalent to Objective-C (which for a statically-typed language is considered to be pretty complete and easy to use) seemed to me like a step in the wrong direction.
  • If you want only to program for iOS (iPhones/Pads/Touch) and Android, Rhomobile is a very serious contender: they run a Ruby interpreter (sandboxed, cut down to meet app store restrictions) on the host OS, and have a reasonably complete set of tools.  They have nothing, however, for the web: if you want to development for “everywhere,” then you will need to go elsewhere.

More Efficient Than VFP + VPM + xCase + PSP

Our tools are designed to allow domain expert programmers (expert in their domain knowledge; average in their programming knowledge) to write software that has all the bells and whistles (dynamic, expression-based, UI properties; dynamic, expression-based viewing, editing, and deleting rights; dynamic, expression-based field properties, e.g., “able-to-be-blank”) and so forth.  In our current model, we build in Visual FoxPro, run a conversion routine, and end up with a SQL Database, with Update Tables to guide updates automatically from any prior version, with Remote Views automatically converted to SQL Server format, and an extended version of Visual ProMatrix’s Data Dictionary that is used to guide the application.  Developing in VFP is fast; conversion to SQL is painless.

Point of Pain: Metadata Creation

Well, almost painless.  <sigh>  What we have found is that our needs for complex SQL have increased as we have gotten fancier and fancier in what we do, meeting the needs our customers have expressed.  Right now, we design in xCase, go through the step of generating to VPM, and then check whether the sql does what we want in VFP.  Bad idea: when working with a small model, that can be fast enough to be no bother: we have 550+ tables at this point: we’re talking 1.5 hours for this step.  Major Bottleneck.  Now, our domain experts are good at taking that initial sql and then tweaking it to make it work and then changing xCase, but even that is a convoluted step.

So we will be approaching things differently here.  We will have a query designer that works in the development interface.  At any point the query will be able to be saved with a name, and after that used by name when specifying a datasource.  The metadata will follow (given a SQL Query, finding out the datasource for each field is relatively trivial, and only has to be done once for each query).

Point of Pain: Inherited vs. Specified Metadata

One of the strengths of metadata, in general (certainly not specific to our setup), is that metadata follows data.  A view field inherits metadata from the source (table) field.  At the UI level, this can be confusing: what am I inheriting for this property on this field (or table)?  Our UI will spell this out in some manner: what is inherited, what is added or removed at this level, etc.

Some of the Pain Point with inherited vs. specified metadata comes from the nature of the UI itself: we have been using xCase’s extendable, configurable UI.  What that UI is not, is programmable.  And that’s why we will be moving away from mandatory use of xCase in this new development stack.  xCase use will still be integrated, for those who want to use it to design data intelligently (there is still no better tool on the market for this purpose, and we have no plans to attempt to replace it for that function), but it will not be required.  If the user wants to design their data by defining tables, fields and relationships in our new UI, without the assistance of ERD diagrams, that will be possible.  If the user does it in xCase, changes will be applied when xCase is saved, without user intervention.

Point of Pleasure: Builders

One of the joys of metadata is the decrease in scut-work when creating forms (dialogs) containing many fields/controls.  Arranging them is one thing; placing them on the design surface, filling in properties, etc., is a waste of developer time.  With our current tools, I was able to create a form creator: specifying a table and picking related tables would build the dataenvironment for the right form class, picking the appropriate views to represent the tables.  That done, the regular form builders that select and place controls linked to fields would take over.  We’ll have the same capacity in the new system.

Point of Pleasure: Build Buddy

When we want a build of the application for testing, this can involve changing the SQL Database schema, new metadata, getting latest on all parts of the project, including pieces unsuited (think: 78MB helpfile), and finally building everything into an install file.  Currently, that is handled by the Build Buddy: check off the parts you need done, push the button, and wait. 

The trend today is toward continuous integration, and it’s a good one.  Features are developed (using Git, e.g.) by forking the project, build the feature, and then merging back to trunk.  Every merge to trunk triggers a build; if the build fails, the trunk reverts to pre-merge, and the developer gets to figure out what went wrong. At every point in time, if the forks are designed correctly (dependencies should belong in the same fork, even if they are forks of the fork), the trunk compiles and tests.

Did I say tests?  Oh, yes: tests.  Unit tests, that run automatically on a build.  Fail the tests, and the build fails.  Unit testing in Python, for our backend, is a snap: Python has a plethora of good testing tools. 

And then there’s the UI.  That has always been the bugaboo for testing.  How do we know that what we know works on the backend, will produce the right actions in the UI?  Fortunately, Sencha has introduced Siesta, created for their internal testing when moving from Ext JS 3 to Ext JS 4.  Siesta can itself be run in automated fashion.  End of problem.

The Future

In the next few years, I expect to see any programming language running anywhere.  The holdup has not been the mobile devices, but rather the browsers.  Perhaps the HTML5 sandbox will be expanded for the purpose of using other programming languages, or perhaps Google NaCl, where native apps can run safely in a sandbox, will catch on.  Something will, because the ecological pressure to do so (witness: the android-scripting project’s high level of activity).

In the immediate future, I expect to see the development stack I have described above take shape.  Our timeline for completion of V1, including releasing our first 3 or 4 apps (seeing how our own dogfood tastes, as it were) is 12 months, starting next month, pending one more approval.

My absolutely certain prediction for the future is that major changes will occur, and that some of them will be entirely unpredictable.  It appears that Semantic Computing will be a Next Big Thing: this description of a Semantic Command Line interface takes the “Search programs and files” textbox style of UI, and puts it on steroids.  Think mobile, where there is limited space.  The user types in “req” and at that time can pick from requisition and any other matches and then types in “rep” and gets “reports” and at that time has a list of choices from which to choose, including “Requisition Reports: Run”.  Picking that puts the interface associated with running Requisition Reports in front of the user.  Wow!

And even bigger changes will occur that provide more and better user experience, along with interesting challenges for those mere mortals (as contrasted with the engineers who created the Great New Thing) to use in a sane (not 100 lines of code to do what we do in 5 lines in VFP) manner.

That’s why the development team creating this product will not disband when the product as currently envisioned is done: if “done” is defined as “no more work required,” then this software development stack will never be done.  In order to swim with the current, there has to be continuing effort, continuing discernment, not to mention continuing joy in creating a working environment we would, and will, use ourselves. 

Currents and eddies change.  Keep going in one direction, and you can find yourself swimming hard and getting nowhere, going against the current.  Swimming with the current, along with noticing the eddies, maximizes the effects of your effort – and as in Harry’s case, gives you a chance to enjoy the scenery as it goes by.