Saturday, June 30, 2007

GWT, JSF, declarative markup, and lunch with Gavin

Just had an interesting lunch with Gavin King (original author of Hibernate and Seam), and came up with a somewhat crazy idea that needs more eyeballs on it.

EXECUTIVE SUMMARY

The GWT toolset lacks a way to do declarative markup. The only way to compose GWT components is through Java code. Joel Webber of the GWT team said at Google Developer Day 2007 that one of his personal ambitions for GWT is to build a declarative markup system.

The Seam toolchain lacks a way to build custom JSF widgets easily. Building a custom JSF widget with plain Javascript is extremely painful. The GWTJSF integration library goes partway towards addressing this, but doesn't currently support building composable trees of GWT-based widgets.

What if we brought the two together? What if we could create a JSF-based and Facelets-based markup system that enabled composability of independently developed JSF widgets implemented in GWT? This could both leverage an existing declarative markup system (Facelets) for GWT's purposes, and deliver a much better toolset for creating Seam applications with richer UI functionality.

That's the basic idea. If you Get It, then please skip to the DETAILS section. Otherwise, read on.

BACKGROUND

The first chunk of lunch was walking through my Seam/GWT example, clarifying what it does and how it does it. Basically, as the page explains, it takes GWT modules and encapsulates them in JSF components. This allows GWT modules to exist within the context of a larger JSF / Seam application, leveraging other JSF page components and directly communicating with Seam components through GWT RPC.

The main use for this, it seemed to us, is taking existing GWT modules and wrapping them as JSF components, to enable their use in a larger JSF application. This was one of the original use cases that motivated the Exadel guys (Sergey Smirnov and Alex Smirnov) to create their GWTJSF integration library.

Where do we take it from here, though? Originally it seemed to us that if all you need is GWT, then this JSF integration isn't of much use to you. Michael Neale has already landed -- in the Seam 2.0 beta -- a GWT integration that doesn't require JSF at all, but just lets your GWT app communicate with Seam components. (He hasn't, evidently, integrated my RPC refactoring in GWT 1.4, so it doesn't yet support native GWT-style RPC, but that'll be easy to do now that GWT 1.4rc1 is out.)

So, is the only use case for GWTJSF really for people who want to take existing GWT components and build them into JSF apps? Well, Gavin saw the code for the GWT module in my demo, and his feeling was, wow, this is a much better programming model than writing custom JSF components in Javascript. So what would it take to make GWTJSF better at creating reusable JSF components?

There are two main things that the current GWTJSF library lacks. One of them is support for Seam conversation propagation. This is important if you want to have GWT components that can participate in a Seam web application which uses both GWT-based AND non-GWT-based components. You want the GWT components to be able to carry the conversation state in RPCs, so they can participate in the Seam server-side conversation state management. (Yes, I know it is totally against the GWT religion to even mention the words "server-side session state." This isn't the main point, so don't get hung up on that.)

But that's actually not the most interesting possible extension.

THE DETAILS

The current Seam blogging example is basically a tree view of blog posts. Each tree node can be expanded to show the full text of a blog post, with an inline edit button that lets you edit the post's text right there in the tree.

Here's the GWT/JSF markup for instantiating this tree-view-blog-editing component:

<!-- This is a JSF component encapsulating a GWT module. -->
<bloglist:component id="main2">
  <!-- This widget routes GWT RPC requests to the Seam component named "gwtBlog". -->
  <gwt:gwtListener serviceBean="#{gwtBlog}"/>
</bloglist:component>


This creates a JSF component wrapping a GWT module. The GWT module exposes an interface which is implemented by the server-side Seam component named "gwtBlog". So the GWT module uses a single interface to communicate with the server.

The GWT service interface is this:

public interface BlogService extends RemoteService {
  public Blog getDtoBlog ();
  public List getDtoBlogEntryList ();
  public String getFormattedHTML (String wikiText);
  public void updateBody (String id, String newBody);
}


This interface conflates operations on the whole list of posts (getDtoBlogEntryList) with operations on individual posts (getFormattedHTML, updateBody).

Overall, this widget is great and all, but it really consists of two orthogonal pieces: a tree view component, and an individual-post-editing component. They're bound together in the current GWT code. This widget is monolithic; all it does is make a tree of blog posts. It can't be repurposed by tweaking the JSF page.

When showing this to Gavin, his feeling was that it was not nearly generic enough. The typical Seam way of doing this would be something like (edited for brevity):

<!-- Generic tree view, using a post-edit widget for each node. -->
<ice:tree id="tree" value="#{tree.model}" var="item">
  <ice:treenode>
    <ice:panelgroup style="display: inline;">
      <ice:commandlink value="#{ item.userObject.text}">
      </ice:commandlink>
    </ice:panelgroup>
  </ice:treenode>
</ice:tree>


This builds a JSF component tree that declaratively nests multiple widgets, using Seam expression language ( e.g. "#{tree.model}", "#{item}") to specify where the components get their data.

Now, here's the rub. What if you could write JSF / Facelets markup that instantiated a tree of GWT widgets this way?

Something like this:

<gwt:tree id="tree" value="#{blogEntries}" var="item">
  <gwt:blogpostedit value="#{item.blogPost}">
</gwt:blogpostedit>


What you'd be doing here is writing a JSF page that creates a component tree that is purely instantiated on the client. The only rendering would happen on the client via GWT. The GWT components would (WAVING HANDS MADLY) have some protocol for nesting themselves, and would have some data-binding style coupling to the server components from which they pull data. The RPC could be GWT-style RPC via service interfaces generic to each specific component.

This would essentially turn the JSF UI component tree into a structure that can be either rendered purely on the server (original-style JSF controls), or rendered partially on the server and refreshed using AJAX (the Ajax4JSF / RichFaces style of user interface), OR rendered entirely on the client (GWT-style widgets). This would also enable declarative construction of GWT-based interfaces, using the exact same syntax and UI component tree as used for other JSF interfaces.

This is good for Seam, because it enables a graceful path towards Seam-based JSF apps with lots of custom UI code, without losing the compositional good qualities of Facelets. And it's good for GWT, because it gives a declarative framework for building GWT applications, with a great story for coupling to server-side business logic.

ARE YOU CRAZY?

This is not quite a half-baked idea yet. In fact I'd say it's about 10% baked. But it's definitely interesting, and it definitely bears more thinking about.

There are many issues with it, just a few of which are:
  • This doesn't really let you mix arbitrary non-GWT and GWT JSF widgets; the only allowable children of GWT JSF widgets would be other GWT JSF widgets. (Unless you get really sneaky with partial rendering invoked FROM GWT, which probably isn't worth it....)

  • I have no idea at all what Joel Webber (of GWT fame) has in mind for declarative markup, so this concept may be totally wrong for his use cases.

  • On the face of it this idea seems to imply some kind of EL (expression language) parser implemented in GWT, which is no small chunk of work.

  • Who exactly implements this is a totally open question; I still want to get back to the offline Blogger client done with GWT/Gears, which has basically nothing to do with this ;-)

I don't claim to have answers to these; this post is just a starting point.

I'm going to post in the GWT developers forum, in the Seam users forum, and in the Ajax4JSF developers forum to see what kind of reactions this idea provokes.

Discuss!

Friday, June 22, 2007

Landed!

My first official self-commit to an open source project.

I've just checked in my GWTJSF patches to the Ajax4JSF trunk. You can now get the GWT 1.4-compliant GWTJSF project by following these directions.

No word yet from the Exadel guys; I'm 99% sure I didn't break anything but they'll tell me if I did, they're not shy :-) I do hope they will update their labs.jboss.org documentation to mention this project again, now that it actually builds in their tree. We'll see.

I included a reference to the googlecode repository in the gwtjsf pom.xml, so it should have no problems pulling GWT 1.4.10 (which it needs).

The old GWTJSF examples (which are not on labs.jboss.org anyway yet) no longer work, mainly because the Maven build (as my last post discussed) splits the sources into a separate jar, so the old build files break. I'll likely land at least one of my modified samples in the Ajax4JSF tree in the next couple of days, so people will have a reference point.

After that I'll get back to finalizing my Seam/GWT example and submitting that to the Seam project. Hopefully in another week, or two at the most, I'll have this whole thing wrapped up and then I can get back to the Gears/GWT offline Blogger editor project!

Onwards!

Tuesday, June 19, 2007

OK, now I see why people dislike Maven

Nothing like using something for yourself to clue you in to the buzz around it.

The buzz around Maven 2 is pretty negative. And now I'm seeing why. As my last post mentioned, I'm working on fixing the Maven build in the GWTJSF source tree. (Right now that source doesn't build at all.)

I got the basic Maven compile working. Spent tonight on trying to get the tests working. Finally gave up when it turned out there were two critical problems:
  1. The GWTShell doesn't like the Maven classloading pattern. Charlie Collins has a patch for this but it didn't make it into GWT 1.4.
  2. Running GWTShell requires access to the OS-specific GWT native libraries. The current means for doing that is with system dependencies in Maven, e.g. download the appropriate GWT version and point your Maven POM at it. This is totally contrary to the entire point of Maven, which is supposed to download everything for you. Will Pugh is working on GWT-Maven support for downloading the whole native GWT archive and expanding it locally, but it's not done yet.
So, fine, I gave up on getting the tests to run. Now I just want to verify that I actually have a working .jar file.

Turns out that I don't, because the GWT compiler needs not just the .class files but also the .java files to be in the gwtjsf.jar file. The original Ant build for the GWTJSF project had no problem doing this, it was totally obvious and trivial how to do it. So, it should be trivial in Maven 2, right?

HAH.

I found the plugin documentation page for Maven 2. This led to the documentation page for the jar plugin. Hmm, pretty sparse. How about the usage examples? Nope, nothing there about what the actual configuration options are, just some examples of signing (which isn't at all what I want to do). But look! At the top, it says this:
If you want to use advanced configurations you should have a look at the Javadocs for MavenArchiveConfiguration.
Oh joy! But wait. If you click on that link, what do you get? You get a page titled "Maven - Page Not Found", which amusingly enough includes a tag saying "Built by Maven".

That's what I call bad -- when your own javadocs for your automated build tool have automatically generated broken links!

It turns out that I missed a plugin that was actually already in the POM for the project, namely the maven-source-plugin. This generates a separate JAR file containing the sources for the project. That's not what I want -- I want the JAR file to contain both the sources and the classes -- but it seems to be the best I can get Maven to give me.

Overall, it really seems that Maven has an incredibly fragmented configuration model, plugins with very, very specific functionality that is hard to override, and documentation that is scattered and broken. Sigh. Oh well, once I land this I'll do my best to be done with it. Here's hoping the Maven team can fix some of this.

Edit: I will say that I like what Maven is trying to do. This POM Reference makes a lot of sense... having a higher-level description of your project is not a bad concept. But the execution and the documentation are fraught with peril.

Thursday, June 14, 2007

Perforce is FREE

Many people love Perforce. That includes me.

What a lot of people don't realize, though, is that if you are an individual programmer working on your own personal projects, Perforce is free.

See, you can download the latest release of Perforce and use it in an evaluation mode, where it's limited to two users and two clients.

If you're only using it to maintain your own local changes to one or more open source projects (say, GWT and Ajax4JSF and Seam), then that's all you need. There's no limit on the number of files or branches, and all Perforce features are 100% functional.

Right now on my Windows PC, I've got a Perforce depot which has:

- a complete download of the GWT Subversion tree
- a complete branch of that GWT tree, with my changes in it
- a complete download of the Ajax4JSF Subversion tree
- a complete branch of THAT Ajax4JSF tree, with my changes in it
- a copy of the Seam booking example
- a branch of the Seam booking example, with my changes in it

Setting my Windows Perforce client to use Unix line endings means that Perforce doesn't screw up the Subversion metadata.

The workflow is:

- check out from Subversion
- check in to Perforce
- branch in Perforce
- edit like mad in the branch, committing at will
- once done, integrate back to the Subversion copy
- check out the whole Subversion copy for edit
- svn update, svn patch, svn commit
- revert unchanged files in Perforce
- re-commit to Perforce

The truly wonderful part is that I can take changes from the Subversion copy to my dev branch, or vice versa, as circumstances warrant. Basically, it's the full power of Perforce branching, used to manage my personal development alongside the concurrent development of everyone who's landing in Subversion.

I can't recommend it too highly. Totally brain-saving compared to keeping multiple local Subversion working copies or whatever. Try it, you might love it.

Mavenizing GWT-JSF

As aforementioned here, I've spent a good chunk of this year integrating GWT, JSF, and Seam. The current fruits of that labor are here.

The original GWT-JSF integration was done by the Ajax4JSF guys. The code is hosted in the JBoss Ajax4JSF Subversion repository. The main problem is that they've converted the entire repository to use Maven, except for the GWT-JSF code.

So I'm fixing that. This post documents how. I'm going to try your patience by giving the blow-by-blow in real-time as I work on this. Consider this an object lesson in how to bang your head against a brick wall with MAVEN spray-painted on it.

First, get the Ajax4JSF code building with Maven, by following these directions from the Ajax4JSF guys.

Second, integrate a pointer to the GWT-Maven repository, along the lines of these instructions from the GWT-Maven team. Specifically:

- Add the gwt-maven repository to conf/settings.xml:
<!-- add gwt-maven repository -->
<repositories>
<repository>
<id>gwt-maven</id>
<url>http://gwt-maven.googlecode.com/svn/trunk/mavenrepo/</url>
</repository>
</repositories>

- Update the GWT dependencies in the main project POM to 1.4.10.

- Add
<scope>provided</scope>
to the gwt-user dependency, to keep it out of the WAR file. (gwt-user.jar is only for compile time; gwt-servlet.jar is all you need in your deployed GWT webapp.)

Why can't the gwt-maven repository live just inside the GWT-JSF POM? In fact, why don't ALL the necessary repository definitions just live inside the top-level trunk POM for Ajax4JSF?

Third, run "mvn install" in the ajax4jsf/trunk/gwtjsf directory. See it blow up because it can't find the 2.0 version of jsp-api.jar.

Wonder why this is, since the top-level trunk build has no problem finding it. Flail around adding a <profile> to the POM and watch it die in all kinds of other interesting ways. Back that change out again. Realize that you really have no idea how Maven works. Look at the base dependencies again...

... Turns out to be because the gwtjsf POM has the wrong groupId for jsp-api -- it should be javax.servlet.jsp but it was just javax.servlet.

And with that, the frickin' thing compiles! That's progress :-D

What's more, it's almost 1 AM and my baby daughter wakes up at 6:30 AM. (Get used to me saying that.) But hey, something works now that didn't work when I sat down here tonight. That's what hacking is all about: getting something new working. Get enough new things working, and anything's possible.

Time to commit all this to Perforce (see my next post) and then off to bed. Next hack night probably on Saturday, or if not, next Monday. Hopefully I'll be able to land this sucker, and then maybe even land my actual changes to the project! What a concept!

Quick GWT update

Well, Rob Hanson is as good as his word, and has a draft GWT ORM for Gears implemented. Meanwhile, the GWT Maven guys have put the latest GWT 1.4 RC into their Maven repository.

And me... well, I've been busy with the grandparents :-) Gonna get cracking again this weekend.

Hiatus

Sorry for the hiatus of the last few days. My parents are in town and there's been lots of grandparental fun, and I've also been kind of blocked on posting here.

I've been in a quandary about this blog, which is that I have too many topics I want to blog about, and not all of them are actually software-related. Should a blog titled robjsoftware.org have posts about religion, or family, or whatever else I might want to say that isn't really code-related?

After getting a lot of advice from various people (thanks Bill Harris, Francis Potter, Bob Lee, and my father Norman), I decided that no, it shouldn't. Part of blogging is developing an audience, and I'm not (quite) egocentric enough to think that everyone who's interested in my programming posts will share all my other interests. I'd rather have opt-in than opt-out, in other words.

So I've started another blog -- blog.robjthoughts.org. That's where all the non-software topics will go. I'll occasionally crosspost, but it'll be very much the exception. I've also cleaned up the labels on this blog to avoid the weirdness of a "non-software" label on a blog named "robjsoftware" :-)

Thanks for your patience. Now that my definitional issues are sorted, content will resume flowing shortly. (And I promise I'm done making new blogs now!)

Friday, June 8, 2007

Flex versus Pilgrim

Mark Pilgrim rants quite cogently about the evils of Flex, Apollo, and Silverlight.

I admit it. We're using Flex at my day job. We need to build a whole huge pile of rich client interfaces to our existing production system. And you know what? Flex simply has the best designer tools of anything we've looked at. We've got some great designers who are able to build interfaces in Flex that have real navigation and real interaction behind them, without needing more than a few smatterings of Actionscript. Flex Builder is just a good tool for building interactive UIs. Then the developers take those designer prototypes and refactor them into the actual application.

Designer-developer roundtripping is what Adobe calls it, and it's just not currently doable in the "web that works."

Now, Joel Webber on the GWT team said at GDD07 (I would post a YouTube link, but no one has posted video of the GDD07 fireside chats yet, why the heck not?) that he really wants to build a declarative markup system for GWT. With such a system, and with a good UI construction tool (that could be built in GWT itself for all I know!), GWT and Javascript generally might catch up with Flex. But right now, for building rich clients really quickly and maximally leveraging skilled designers, I just don't see how Flex doesn't win.

None of that has anything to do with the things Pilgrim is ranting about, and yes, proprietary is bad. I think Pilgrim missed that Flex, which Apollo is based on, is open source now. Not sure how effective or real that is, but it is technically true. And, of course, my company is building intranet apps, with only limited customer exposure, which is pretty much the ideal case for Flex right now.

Anyway, there's my stake in the ground: I'm not religious about open source, and I'm willing to use closed source tools when they've got compelling advantages. (But in my spare time, it's open source all the way!)

Also, so Pilgrim can laugh at me: how in the world do you subscribe to his RSS / Atom feed? I CAN'T FIND IT ON HIS SITE. Which means I must clearly be very dumb in some really basic way.

Thursday, June 7, 2007

Open source update: Maven is not my friend

Next in the Offline Blogger Editing saga. If anyone mentioned in this post would rather I shut up, please let me know.

Catch-22 with Maven: how can you create a project that has a dependency on another project that hasn't been stored in some repository somewhere?

I'd love to land my GWT/JSF integration work. But the Ajax4JSF team really likes Maven for some reason. And my code relies on GWT 1.4, which isn't final yet. So there's no way for me to check in a Maven build that uses it, because there's no repository in the world that has it.

Feh. I like what Maven is trying to do, but I would love there to be some middle ground here. Gavin King wants to do lunch next week to talk about the GWT/Seam stuff, though, which is cool.

Meanwhile, Rob Hanson is working on a GWT/Gears ORM implementation, and Ryan Dewsbury (author of the legendary gpokr.com, as well as a book about GWT applications) already has a GWT Blogger client written and working as part of his book. So it looks like gluing those two together would be an offline Blogger client in, like, no time. Ryan's looking into how soon Addison-Wesley will approve the source for some kind of release, Rob Hanson's going to have code working by Monday.

Given that the Maven stuff is slowing me down on finalizing the GWT/JSF stuff I did, it may be that Hanson and Dewsbury are going to put this offline Blogger-editor thing to bed before I can get my head out of my butt. Which would be fine -- GO GUYS GO! There's always more code to write, and no sense being selfish :-D

Blog dilemmas

  • Many small posts or few large ones?
  • Why does Blogger do such a terrible job of scaling your profile picture?
  • Should I tell everyone on God's green earth about my blog right away, or should I post a lot and then start spamming about it once I actually have some content?
  • How many people actually read through your back posts anyhow?
  • Given that there's only so much spare time, how much of it should I spend programming, how much should I spend blogging, how much should I spend reading blogs, and how much should I play one of the dozens of computer games that stare at me longingly from the shelf, wondering where my love went?
  • Is this template just too ugly for people to stand for even one minute?
  • Why is Blogger's edit window so d**n small?
  • Can I say d**n without the asterisks in this blog?
  • Is editing posts after they're published bad form, even if you think of lots of things you forgot to say?
  • Am I interested in -- and liable to blog about -- too wide a variety of topics to attract any readership whatsoever?
  • Who is this blog for anyway, myself or everyone else? Is blogging like art -- something that exists only in relationship to an audience?

Tuesday, June 5, 2007

Being Too Ambitious

Sorry for the logorrheic posting flurry. I've been wanting to start a blog for years but for some reason never did. Now that it's up, the dam has burst.

So this post is about my history as a programmer, and about some of the lessons I've learned in ambition and tackling the right problem.

First let me say that I'm hopelessly in love with programming. My basic definition of software engineering is "building machines out of ideas." And that's something that essentially has no bottom, and no top. There's no limit to software.

That flexibility is also a trap, because in reality software has to be built on other software. And overreach is the ambitious engineer's curse.

Let me give some examples from my own career. My first real post-college job, in 1990, was working on Project Xanadu. I'd read Engines of Creation in school, and it totally changed my life. I wanted to make hypertext happen, now.

So I flew out to California and started working for Autodesk with the Xanadu folks. What followed was one of the most exhilarating, and then one of the most disillusioning, periods of my life.

Xanadu was a glorious triumph of software design, and a consummate failure of software production. Picture it: the entire system was predicated on the assumption that hyperlinks would only work if they were fundamentally coupled to the individual textual characters that made up their anchor, such that any editing operations would exactly track which characters were linked to. The design sought to create a serverless, versioning system for generic hyperdimensional multimedia data, on top of which was an addressing scheme for arbitrary hyperlinks. All this was built on a super secret data structure, essentially a multi-coordinate splay tree (with adaptive optimization for all lookups). An object-oriented data store provided the persistence. A pipelining message protocol provided the networking And all of this was written in Smalltalk, compiled to C++ for efficiency.

IN 1991.

Do you remember how slow computers were back then?

The World Wide Web came along a couple years after the project finally fell apart, and took over everything. Because the fundamental assumption of Xanadu -- that the critical problem was tracking all anchors across all content edits -- was wrong. That problem wasn't critical. And Xanadu's now a footnote.

On the bright side, I have a lot of friends from that era, despite my hastily chosen and immature words in the Wired article that came out in 1993. Dean Tribble and Ravi Pandya now live in Seattle, working on the Singularity OS research project at Microsoft. Mark Miller is at HP now, thinking more deeply about secure mobile code than anyone. Life goes on and ideas flower in their time.

So from that, I decided to reduce the ambition level a lot, and went back to Autodesk to work on Autodesk Animator Studio, a Windows video paint package.

For Windows 3.1.

Anyone remember what watching video in Windows 3.1 was like? On a 1994 Pentium with 8 megs of RAM? Now imagine editing it.

(It looks like the only little video I saved from those days no longer plays in Windows XP. Sigh, bit rot creepeth ever unto the past.)

Yes, this could also be called overly ambitious. We did ship, but the whole experience drove my manager at the time, Jim Kent, out of the software biz for good. Fortunately, six years later, that turned out to be a very good thing for the public version of the Human Genome Project.

But hey, shipping was a big step forward!

A brief detour into charting software, and then I decided I wanted to get into the computer game business. So I worked for a small company named Protozoa in San Francisco. Protozoa was doing performance animation software, and wanted to make a game with their technology. I learned a lot about 3D and realtime programming there. I also learned that if you don't have a total focus on what makes your game fun, your game will not in fact be fun. I ALSO learned that building a realtime 3D game on DirectX 3, in 1995, is not the best idea. Again, ambition bites, hard.

Years later I learned that one guy I worked with there, Bay Raitt, was destined for glory. Bay was a super genius character modeler. He was agonized by the problems of trying to build characters with only 200 polygons. Exactly how agonized I only realized years later, after seeing the second Lord of the Rings movie. I bought a book about how they created Gollum, and lo and behold, Bay was Gollum's lead facial animator. In retrospect, seeing him work on a 1995 3D computer game was like seeing Picasso trying to work with a half-dry jar of finger paint.

Alrighty then! Time to move on from computer games, I thought. But the stings of Xanadu had healed somewhat, and I wanted to learn about networking. And a number of the Xanadu folks had re-convened in Cupertino, at a startup named Electric Communities. EC was seeking to take some Xanadu-era ideas about capability security and secure mobile code, and to build from them a distributed, peer-to-peer, cryptographically authenticated, secure-identity-supporting, commerce-capable virtual world.

Using Java version 1.0.

In 1996.

Are you seeing a pattern here?

EC failed for reasons similar to Xanadu: the software was too big, too ambitious, and too slow to make a compelling product. Not only that, but users didn't want virtual worlds nearly as much as everyone thought. Years later, and with 21st-century hardware, only Second Life has somewhat cracked the nut, and even their infrastructure is suffering pretty deeply right now (word has it).

Again, though, I met a number of super smart people who are now doing great things elsewhere, including Doug Crockford, moderately (ahem) well-known in Web 2.0 circles. Also Arturo Bejar, security ninja at Yahoo; Brian Marick, a major testing god; and Chip Morningstar and Randy Farmer, also now at Yahoo, who have a quite candid assessment of the whole era on THEIR blog. And I got to spend the better part of a year figuring out how to create a secure, capability-based protocol for virtual-world object transfer between untrusting hosts. Getting paid for that? It was really, really fun. Until the money ran out.

EC was really the end of my over-ambitious days. In 1998, after EC, I went on to work for Helium, a now-defunct consulting company founded by an old Protozoa buddy (Gever Tulley) and a number of his oldest friends. We did a bunch of work for Quokka.com (a truly classic dot-com flameout story -- they picked the wrong sports to cover and created all their sites by hand, but we had fun doing Internet livecasting of sailing races and the 2000 Olympics!). Then Quokka folded in 2001, Helium reformed, and we went on like that until 2003.

Finally in 2003, I begain working for my current employer, Nimblefish. A number of my old Helium and Quokka friends joined me there, along with other great folks. I've been there for three and a half years now, an all-time record for my career, and we've got a working production system that just keeps getting better and better. Hibernate, Flex, and other modern tech drives our code; we've got real customers, and we're growing; and we've got a clear strategy that we're executing on at full bandwidth. I think I might have finally kicked the curse of over-ambition.....

...or maybe it's just dormant! :-)

I'll close with the word that best captures how I feel about programming:

Onwards!

Monday, June 4, 2007

Pluggable types, objects as services, and Google Gears

Warning: rambling language research post ahead!

I admit it, I am a static typing weenie. Having gotten hooked on autocompletion, real-time code error highlighting, and advanced refactoring support -- IntelliJ is my king -- I find that it's hard to imagine doing without.

But that's not to say that type systems aren't sometimes a real pain. Particularly ornery is all the machinery around making changes to the type system. Not all of us can be Scott Blum (GWT compiler ninja), and compiler hacking is not a well-known art. But without access to compiler internals, type-system hacking is very hard to come by.

Gilad Bracha, while he was at Sun, had some interesting things to say about pluggable type systems. Now, that paper is somewhat of a gripe fest, in that while it may be clear what he's complaining about, it's not entirely clear what to do about it. What does a pluggable type system actually look like?

I occasionally scour the recent proceedings of the ACM for interesting papers. (Any language lawyer who's not a member of the ACM's Digital Library is seriously missing out. Also, I've got to catch up on Lambda the Ultimate.) Recently in OOPSLA 2006, Andreae, Noble, Markstrum, and Millstein gave some real body to the theory with their paper on JavaCOP, a framework for building pluggable type systems in Java.

One of their examples is a @NotNull annotation to mark variables that can't have null assigned. Check out their syntax for verifying that no bad assignments happen:
rule checkNonNull (node <<: sym) { 
where(requiresNonNull(sym)) {
require(definitelyNonNull(node)):
error(node, "Possibly null expression "+ node+" considered @NonNull");
}}
Working through this, this defines a constraint on all symbols that have the requiresNonNull property (which they implement with the @NotNull annotation on the symbol). All such symbols further require that the AST node assigned to them definitely has a non-null value. Their system implements all this as a layered extension to javac, that is able to check eight or so extended type properties in no more than double (at worst) the compile time.

Very interesting stuff, which points (to me, anyway) in the direction of generalizing the research to re-express Java's whole type system in rules like this. In other words, they're starting from Java and layering more type rules on top, but they could also work downwards and re-engineer Java's existing rules in their framework.

What would be the ultimate goal here? Well, giving more flexibility to juggle the nominal versus structural issues that Gilad talks about would be one reason. Another reason would be found in a more recent Gilad paper (apparently presented at Google last fall), about implementing offline objects that provide local support for centralized software services.

Now, if you read this paper after the recent shipment of Google Gears, it has all kinds of interesting ramifications. Gilad proposes a generic synchronization system implemented by reflective logging of local object modifications, supporting reconciliation on the server by change log replay. (This is actually very similar to the Dojo Offline model of synchronization, as presented at GDD07 last week. I wish there were a text post describing the model, because all these video posts and podcasts give me a pain. ) I'm not entirely convinced that model is as generic as he would like, and I will need to bang my head against data-based synchronization some more before I go to operation-based synchronization, but it's nonetheless thought-provoking.

Pluggable type systems come in when he discusses upgrading local code in a modular way. It looks to him like a fully static type system such as Java's has inadequate flexibility for changing shared type definitions at runtime. This is getting into the arena of major magic -- I'm personally very queasy about the whole concept of upgrading code modules that are in mid-execution. I'd prefer to treat that problem via an Erlang-like model of strict isolation through message passing -- the model that the E programming language implements. (And by the way, if you haven't bought into capability security yet, read one of Mark Miller's recent papers already and get with the program!)

At GDD07, I ran into Doug Crockford, who mentioned that he and Mark had discussed taking Google Gears' support for isolated worker threads and using it to implement securely encapsulated downloadable code. That squarely jibes with Gilad's recommendations, and is something that could be prototyped Real Soon Now. Of course, Gilad's whole pluggable-type machinery is definitely not on that agenda, but we must walk before we can run.

So where does this leave us? Seems to me there are several directions to pursue:
  • Gears provides a reasonably deployable framework for experimenting, not only with offline applications in general, but with an encapsulated framework for mobile code, and with the overall synchronization / update model Gilad discusses.
  • Pluggable type systems in general, along the lines of the JavaCOP system, provide a rich variety of new data points on the traditionally impoverished "static versus dynamic" axis in programming language design.
  • The right kind of type system to implement extensible, upgradeable local software objects -- particularly local objects with the ability to have their message formats, persistence structure, and inter-module interfaces -- is a huge open question, and pluggable type systems give us a much more fine-grained and flexible toolset for experimentation.
I find myself wondering what a JavaCOP framework built into the GWT compiler would look like....

(Does anyone have any links to work on JIT compilation for Javascript? If Javascript is moving in the direction of being a generic intermediate language, it'd be good to know how fast it can be made.)

Why Blogger Needs Offline Editing Support, and How To Build It With Google Gears

I'm writing this in Notepad while riding the BART train on my normal morning commute into San Francisco. This commute takes about forty minutes each way for me. Since there's no Internet, I can't connect to work at all. Hence this is dedicated time for reading, or writing, or hacking on things that don't need a full development environment (my laptop is a bit too old to cope with our entire code base from work anymore).

So of course blogging would be a natural. And sure, writing in Notepad is not torture. But it's also not the right thing. What I want is to be able to write blog posts, edit them, save drafts, etc., etc., and then the minute I get back online, upload 'em all to Blogger and worry no more about it.

To do that requires some kind of offline application with local storage. As my last post mentioned, I originally wanted to build such an offline app as a standalone Tomcat-based Seam webapp that would run in a local webserver. But Google Gears offers a much more direct route to the same destination.

So this post is a breakdown of how to construct such a client, based on conversations with various Googlers last week at Google's Mountain View developer day.

The Requirements

Must-have requirements for an offline Blogger editor:
  • be a web application running in the browser (duh!)
  • support rich text editing at least at the level of Blogger's own online editor
  • support authentication to Blogger
  • support local storage of some or all of the user's blog posts
  • support creating, viewing, drafting, and editing of posts whether online or offline
  • allow synchronizing any offline edits to Blogger once back online
  • allow synchronizing of any posts edited on Blogger down to local storage
  • allow very minimal reconciliation of conflicting edits to posts on Blogger
  • require minimal installation of plugins or other application-specific software
Nice-to-have requirements for an offline Blogger editor:
  • support archiving / backing up of a user's entire blog
  • support re-importing some set of content to a new blog that supports the Blogger API
  • support migrating content between blog services
  • support deleting postings ahd handling synchronization of offline deletes
  • provide a nicer browsing/reading UI (arguably Google Reader has this job sewn up already)
(Needless to say, the nice-to-haves don't get done until all the must-haves are working!)

Explicit NON-requirements:
  • No need for this application to render any static HTML (the blog can presumably still be viewed and searched through Blogger itself; this app can be *just* an editor)
The API Implementation

Clearly this is exactly the kind of application Google Gears was created to support. So let's assume we're using Gears. Further, since I like GWT a whole lot, let's also assume we're using GWT.

I originally thought that this app could be built as a GWT/Gears webapp connecting directly to the Blogger API. Dan Morrill of the GWT team set me straight on that. The Blogger API is very pure in its REST-fulness, meaning specifically that it uses all the HTTP verbs. However, XMLHTTPRequest, the entire foundation of AJAX, can't do anything other than GET and POST. So whatever we do is going to have to involve some kind of servlet bridge to the Blogger API.

(Possible workarounds for this: 1) get the Blogger and GData API teams to make GET/POST-only variants of their PUT/DELETE API methods; 2) get the browser vendors to implement PUT/DELETE support in XMLHTTPRequest. Clearly #1 is a lot more doable than #2 given the dreaded words, "legacy support". You listening, GData team???)

Once we've accepted the need for some kind of bridge servlet, we'd better make sure that bridge servlet is fairly scalable and generic. Whoever is hosting this bridge servlet is going to have potentially a lot of traffic. So the bridge servlet needs to be stateless, re-entrant, and as close to a simple pass-through as possible. It shouldn't have per-session state; it should be just a message relay.

We'll also need some GWT modules for communicating with the bridge servlet, including beans that map to the data objects defined in the Blogger API, and service interfaces that map to the Blogger API's methods. And in fact, this suggests that the servlet itself should simply use the Java implementation of the Blogger client API, exporting a GWT-friendly version of it (via GWT RPC) to the GWT client application.

The persistence implementation

Gears has this lovely local SQL database. But how do we get things in and out of it? We could write to the JDBC-like Gears SQL API. But I have tasted Hibernate. And if I'm programming in GWT, then writing JDBC-like code by hand would be like going back to the pre-Hibernate days. And I can't do it. I CAN'T DO IT, I TELL YOU! I can't go back. I WON'T go back!

Now when I first started brainstorming this with Ray from timepedia.org at GDD07, we got wild-eyed and started contemplating porting Hibernate to GWT. Saner heads rapidly prevailed. It turns out that Bob Vawter of the GWT team is doing work on a GWT-generator framework for generating a variety of useful artifacts (starting with Javascript-library method exports) from GWT beans.

So let's use his framework, only instead of exporting to Javascript libs, we'll make it generate dead simple CRUD operations for our Blogger beans to the Gears data store.

It looks like some other folks on the Gears list have had similar thoughts. Check out this Gears ORM implementation. But since that's not based on GWT, it's not directly applicable. Still cool though! But let's assume we'll proceed with Bob's stuff, just for fun. This is all still early days and all experiments still deserve love.

The todo list

So, what do we have?
  1. GWT beans mapping to Blogger API data objects.
  2. GWT service interface mapping to Blogger API methods.
  3. Bridge servlet implementing GWT service interface and using Blogger Java client API implementation to relay messages to Blogger.
  4. Simple persistence code generator (using a GWT compile-time generator implementation) to take the GWT beans and store them in the Gears local database.
  5. Business logic to implement the client-side synchronization to the Blogger server. (That'll be another post, or maybe lots of them!)
  6. GWT UI code to let the user use all of this infrastructural goodness to meet the must-have requirements!
If we get it right, it hopefully lands in the Google GWT API project. And once Blogger does this, Calendar is next, and then some.

Sounds like a fun project, huh? Interested? Chime in in the comments. Or join this GWT-Contributors thread on the topic.

Onwards!

Friday, June 1, 2007

Why start this blog?

[ObNewBlogIntentions] Will try to update often. Will try not to spend so much time farting around with templates that I never get around to posting. [/ObNewBlogIntentions]

So, with that out of the way, hello. I've been planning a blog for a year and a half now. I realized in early 2006 that I'd spent my whole life programming for companies who owned everything I did, and I wanted to do something open source on my own time. So I thought, what about a blog system? Clearly the world needs more of those :-)

(The nice thing about programming on your own time is you don't have to justify anything to anybody! Except your wife. But if you cut back on the computer games, so you're not blowing your personal time budget away, then it's all good.)

The thing is, I really wanted a blogging system that would support offline editing and offline backup of my blog content. I ride the commuter train, and I don't entirely trust any single system (even one run by Google) -- I like lots of redundant backups that I personally maintain.

So I started on a project to implement a distributed blog application. Which, of course, since I'm a Java weenie, meant that I really needed a Hibernate-based peer-to-peer distributed object version control system.

I built most of that over the bulk of 2006 (or prototyped it, anyway), using a Seam / EJB3 / /Tomcat / JSF stack. Then I thought, how do I build the UI? I started looking into GWT and liked it. I even found a JSF / GWT integration library. But it didn't handle GWT RPC properly.

Then GWT got open sourced. I started working on fixing the RPC issues in GWT in early 2007, and that led to my submitting what I think was the first major external patch to GWT. I also put together a demo of Seam + JSF + GWT -- adding a GWT blog-reading interface to the Seam blog example.

Michael Yuan from JBoss mentioned this at his JavaOne BOF this year, and I got to demo it at the GWT hack session at JavaOne, where I had the pleasure of meeting Bruce, Joel, Kelly, Bob, and the Dans (Peterson and Morrill) from the GWT team. Then I took a couple of weeks off to recuperate -- my wife is expecting our second child in late July (our first child is two and a half), so there's plenty of other distractions keeping me busy!

So that brings us to yesterday. Google Developer Day in Mountain View. And the announcement of Google Gears and GWT for Gears. Now, remember that my original use case was to build a blogging app that could work offline. I was using a technology stack that was Seam + EJB3 + Tomcat + JSF + GWT + Postgres. So to use this app I was building, you'd have to be running a local Postgres install and a local Tomcat install, and you'd have to do a bunch of Java deployment. I might have wound up with a few dozen users, or maybe only myself. (The code could have wider applicability, but direct product usage would've been low.)

Along comes Gears. Suddenly there's a whole new stack option: Gears + GWT + Blogger. Offline functionality usable with any Blogger blog, and a much, much, much easier deployment, with a possibly much larger user base as a result.

So yesterday at GDD07 I brainstormed about this a bunch with Dan Morrill, Miguel Mendez, and Bob Vawter from the GWT team. It looks like a go. My next post will be about the detailed plan for that.

Meanwhile I still have to finish landing the JSF+GWT code, and documenting and submitting the Seam example. This whole project has been one surprise after another, and it's only getting more fun so far :-)

Onwards!

Welcome!

Hello folks. This blog has been too long in coming, due to many complicated reasons of my own self-inflicted devising. Better late than never. Enjoy!