Message for you, sir!

Helping Don spend $100

john-cavnar-johnson — Sat, 18 Mar 2006 23:13:00 GMT

While I was writing that last post, Don Box asked for some help spending "100 engineering dollars" at Microsoft on HTTP, XML, and REST. Now, I would rather give him some advice on spending $100 on the other side of his job (WCF, WSE, SOAP and such), but since he specifically asked for this, here goes:

$50 on fixing the XSD mess. Specifically, develop a well-defined subset of XSD that is useful for defining the structure of business documents (and really I think you can cover 80% of all business documents with 20% or less of the current XSD spec). Build a really cool visual tool that uses this subset of XSD to help application architects design distributed systems around business documents. Make that tool part of Visual Studio Team System. I don't want to prevent anybody from using all the XSD spec, if they really want to and can completely explain all the ramifications of UPA. I just don't want to be bothered with all that extraneous crap.
$45 on putting all the XML coolness of VB9 into C# 3.0. This is strictly altruism on my part. I use VB, but I have good friends who, for reasons totally inexplicable, like the whole case sensitive curly brace lifestyle.
$5 on tool support of SSDL. OK, I'm cheating here. This is SOAP related, but it wouldn't cost much and maybe Tim Bray wouldn't even notice.

I really don't think any of the rest of the stuff he mentioned in his post really requires any additional investment beyond what Microsoft has already committed to.

How to build a distributed application: Why Practice Matters

john-cavnar-johnson — Sat, 18 Mar 2006 22:45:00 GMT

"In theory, theory and practice are the same. In practice, they aren't even close."

I have no idea who said that first, but they are words that an application architect should live by. As I mentioned in my last post, I really do think it is important to have a strong theoretical underpinning to serve as a guide for developing a distributed application. It is also true that our ultimate goal as application architects is deliver working code that solves real business problems. Building actual systems is a continuous process of trade-offs, compromises, and imperfect choices. With this post, I start applying my theory to real application scenarios. The rest of this series will focus on a mix of scenarios from a slightly abstracted version of an application that is currently in development and pieces of applications I've helped build in the past. My goal is to show how to apply a simple set of constructs and processes to solve real-world challenges familiar to most business application architects and developers. The goal of the application architect is to supply an underlying infrastructure that lets developers concentrate on delivering business functionality without getting lost in the morass of conflicting forces that must be balanced to design and develop an effective distributed application.

Business Processes: User Tools, Documents, Messages, Services

I think of distributed applications as a set of technological artifacts constructed to support one or more business processes. I'm not going to attempt to define a business process. If you are a business application developer or architect, I hope you know what your organization does and you have a pretty good handle on the business process(es) that your application is supposed to support. If not, you're reading the wrong blog. From an architectural perspective, I recommend modeling your distributed application as a combination of user tools, documents, messages, and services. No doubt, this will strike many of my readers as a bit odd. There are no objects, protocols, databases, frameworks, APIs, or platforms in sight. All of those are implementation details. We'll get to them eventually, but you should start at a higher level. Let's look at each of the four constructs in more detail and try to make a few technological decisions along the way. Because most of my experience has been on the Windows platform, my examples will use Microsoft technologies, but the principles apply more generally.

You may be more comfortable with terms like user interface, presentation layer, or rich/smart/thick/thin/dumb/browser-based client. I think it's more useful to think of the tools your users need to fulfill their roles in the business process. This puts the focus on your users and what they need, rather than concepts driven by technical constraints. Ideally, you want to choose a computing platform and developer tools that let your business developers deliver the best possible user experience. In reality, many times your choices are limited by things like corporate standards, application requirements, and developer expertise. This is the area where I've experienced the most painful of those trade-offs, compromises, and imperfect choices that I mentioned above. You have to weigh a wide variety of conflicting forces in a constantly changing environment. For the purposes of this series, I'm assuming one of the best possible environments for developing a distributed application: a corporate environment with managed desktops running Windows XP SP2 with automated deployment of the .NET framework 2.0. Given that environment, there is no doubt that Windows Forms is the best way to deliver user tools for internal users. I'll assume the use of a cross-browser compatible web application for our user tools for external users. If your environment is substantially different, your choices for user tools may be quite different and it can have a substantive impact on your architecture. I'll try to point out some of the differences as we work our way through various scenarios.

Documents

Most business processes revolve around users manipulating documents: purchase orders, customer information, vendor information, service requests, etc. Fortunately, most of the technical world has settled on a single technology for representing business documents that is straightforward and well supported: XML. Unfortunately, most of the technical world has settled on a single technology for representing the structure of business documents that is overly complex and unevenly supported: XML Schema. I recommend embracing XML in your architecture and frameworks. Some application developers will want an OO façade around XML documents and others will want to work directly with the XML document. Let them make that choice and do the work. XML Schema, on the other hand, should be treated like MSIL. Folks should know it is there and understand how to get to it, but nobody should be building it by hand and you should realize that you can accomplish everything you are likely to need by using only a very small part of its capabilities.

Messages

Your user tools and services will communicate using messages. Typically, messages represent interactions focused on documents (i.e. requests for documents, commands with documents, etc.). Most discussions about distributed application design get bogged down in arguments about how to best send, receive, and represent messages. Mostly these are what I like to call "artichokes and roses" arguments. If you don't know what I mean, just ask the next 10 people you meet which are better, artichokes or roses? You won't get a coherent answer (and most people will decide that you are a bit loony) because the question is meaningless without some context. Handling messages and network communications is where the application architect should really earn their keep. In my next post, I'll show you how to design a communications API that is easy for developers to understand and fosters good distributed application design. I'll use SOAP as the underlying messaging model (mostly because it gives us a good interop story), but completely hide it from the application developer.

Services

Services are where most of the work of the distributed application happens. Services allow you to harness all the power of server-based computing to solve real business needs. The biggest challenge facing most service developers is the difficulty of multithreaded programming. Answering that challenge will be the ultimate goal of this series. I just wish I knew for sure how it turns out in the end.

Next Up - A Communications API

In my next post, I'm going to try deliver something of real value in building a distributed applications (about time, after all this theory and modeling stuff): A message-based communications API for user tools. I'll follow that up with a complementary API for services to use.

How to build a distributed application: Why Theory Matters

john-cavnar-johnson — Mon, 30 Jan 2006 22:18:00 GMT

After looking at the comments on my last post, I have decided to write a few words about why I think it's important to have a theoretical framework for how to build distributed applications. Our industry is beset with faddism, magical thinking, and tribalism. We spend an enormous amount of time debating best practices, patterns, and "the right tool for the job". Unfortunately, we usually end up with nothing more than slogans, silver bullets and shibboleths. Without a coherent mental model for what makes things work, we can easily get lost in a jungle of received wisdom, industry trends, and vendor claims. I like to think of the OODA loop as my "machete of truth". It's no guarantee that I'll find the right path, but at least I can clear out the most obnoxious undergrowth.

A few specific examples will probably help explain this more clearly. Suppose you hear about a new tool or technique that is supposed to help improve your application development process or application architecture. How do you decide if its potential value outweighs the costs of changing the way you do things? Here's one I've started looking at: FIT. I take a quick look around the web and I like what I see. I have a lot of respect for the work of Ward Cunningham and James Shore. On the other hand, I also know that the real superstars of development are much more productive than I and my team members are. Superstars often make the mistake of assuming that it's their tools or techniques that make them productive, but that is usually not the case. What works for Ward and James may not work for me. If the OODA Loop has predictive power and can tell me why some tools and techniques work better than others, I am in a much better position to decide whether or not to push for incorporating FIT into my preferred process. In this case, FIT fits the OODA loop model very well. Now, I'm faced with the hard work of really learning about it, figuring out how hard to push my organization to accept it, and when to introduce it into the process. Has anybody done any work to hook FIT into Team System? Sam? Anyone?

Using the OODA Loop as a mental model can also help you understand how to use a potentially dangerous tool (and all powerful tools are potentially dangerous). As you might expect from a VB-loving Mort, I absolutely adore "Edit and Continue". I'm certainly aware of the dangers of that feature, but I've also used its power in extremely productive ways. Writing code in a late-bound, loosely typed, verbose language with an IDE that supports auto completion, edit and continue, and excellent framework support allows me to engage in exploratory coding that approaches "coding as thinking". This technique allows me to iterate through a variety of potential solutions to a particular problem without the friction inherent in the more traditional method I use in developing production quality code.

I would like to posit the "spinning balls" from MTS/COM+ as an example from distributed application architecture. Back in the day, I spent many hours becoming intimately acquainted with the ins and outs of MTS. At the time, I didn't understand why IT managers and business users were so fixated on those darn spinning balls. Now I understand that, for most of them, this was their first exposure to distributed applications that provided them with insight into what was happening as the application ran. Understanding the OODA loop allowed me to appreciate the importance of this feature. The vast majority of distributed applications are completely opaque to the people who pay for them and the people who support them day in and day out. As application architects considering code hosting environments (e.g. ASP.Net, BizTalk, Windows Activation Services), the first question we should be asking is "Where are the spinning balls?" The ability to see what's happening as your systems run is so vital that no modern distributed application can afford to be without it. Building that capability on your own can be devilishly difficult.

These examples are really just about how I use the OODA loop in the small. My main goal is to develop and propound a theory that unifies the process and the architecture of distributed applications. I don't think that's really been attempted before (this statement is my attempt to elicit evidence to the contrary). Ali Pasha asked what other frameworks had I considered. The closest thing to what I'm trying to do that I'm aware of is feature driven development. Unfortunately, FDD is very closely tied to OO in ways that I think are inappropriate for distributed applications. Of course, that doesn't stop me from appropriating all the things about FDD that I like. On the process side of the fence, I think Scrum has the best theoretical underpinnings. I will borrow heavily from it and the ideas of David Anderson. I would be remiss if I didn't point out that one of my favorite bloggers has considered the idea of connecting agile development with SOA and pretty much dismissed it out of hand.

How to build a distributed application: Introduction

john-cavnar-johnson — Thu, 19 Jan 2006 23:06:00 GMT

If you set out to devise a theoretical framework for both the design of distributed applications and the process of building distributed applications, it seems unlikely that you would base it on a theory for armed conflict. Even though we may talk of "death march" projects, I've never experienced any real life or death struggles in the world of software development. And yet, I've come to use Col. John Boyd's OODA loop as my framework for how to build a distributed application (this will seem doubly ironic to those folks who know me well). Boyd had a laser-like focus on destroying the enemy in the most efficient manner possible by improving "our" OODA loop while disrupting "their" OODA loop. In software development, there is no "them" (even if we developers, in fits of extreme perversity, sometimes think that our customers or the system administrators are enemies). I use the OODA loop with a completely inward perspective. That is, I use it as a measure of how well my team or application is working.

What is the OODA Loop?

A complete discussion of the OODA loop is well beyond the scope of this blog, but Boyd's fundamental thesis was that the quicker you iterate through the loop, the more effective you become. The most common misimpression is that his thesis is simply about speed. To the contrary, the real key to understanding the loop is that quicker iterations come primarily by combining an initially accurate orientation, which allows you to favor implicit guidance and control (the thick green arrows in the diagram above) over explicit decision-making, with the flexibility to accommodate changes in your orientation based on observation. Automated unit testing makes a good example from the world of software development. Many developers believe that writing unit test will inevitably slow them down. If you take a static view of the process, this seems obvious:

production code + unit tests > production code

But once you experience developing with effective unit testing, you realize that you can actually write better code faster. In OODA loop terms, you've made the creation of unit tests part of your Orientation ("requirements") phase. Each time you make changes to your code, you run the whole set of unit tests (this is implicit guidance and control) and you can observe whether the changes have adversely affected your conformance to spec (red light/green light). You gain the ability to tighten your coding loop which, in turn, leads to an overall decrease in the amount of time it takes to deliver the right code.

Another way to look at this diagram is to use it to model the architecture of a distributed application. In this view, the decision diamond represents a user decision, the "implicit guidance & control" boxes represent application logic, "outside information" represents external system inputs, and orientation represents application state. I suspect trying to visualize a distributed application this way will be very disorienting to most application architects and developers because it is initially very difficult to relate this view to our normal mental constructs for application design. I find it extremely useful, because by concentrating on designing applications that can speed up this OODA loop, I can discover application patterns and techniques that provide more effective business process automation. Using this model leads directly to distributed applications that are, in today's parlance, service oriented.

Where do we go from here?

This post is designed to lay the groundwork for a series of posts that will translate this somewhat esoteric theoretical framework into practical advice. Before I write those posts, I'm interested in some specific feedback. Would you like me to explain more about the OODA loop and how it's commonly used outside of software development? Do the examples I used make any sense at all to anyone besides me? Does anybody think it's important to a theoretical framework for building distributed applications (whether or not you think mine's full of hot air)?

How to build a distributed application: Preface

john-cavnar-johnson — Thu, 12 Jan 2006 20:54:00 GMT

With this post, I'm starting a series of blog entries describing how a distributed application should be built. I'm not making any claims that how I build distributed application is the only way, or even the best way. Instead, I'm trying to do what I described in my initial post on this site: Create a context in which people can think about building distributed applications.

There are two distinct ways to think about a "How-To" approach. One is to focus on the architecture and design of distributed applications. The other is to focus on the process for building distributed applications. I plan to focus on both aspects of the "How-To" because I've come to understand that they are fundamentally related. In fact, I use the same underlying mental model for both (more on that a little later).

One thing that's clearly true about distributed systems design and the software development lifecycle is that they are both primarily realized through a collection of "best practices" rather than as an expression of any coherent theoretical foundation. The best information available in both domains comes from the experience of successful practitioners. Unfortunately, it is often difficult to distinguish which practices really contributed to success and which were merely incidental. Moreover, advancing the state of the art becomes both tedious and risky. Without a theoretical framework to guide us, we stumble around in the dark looking for ways to improve our efforts.

In an effort to shed some light on these topics, I'm going to put forth a model that will serve as unifying model for discussing how we build distributed applications. As always, your feedback is welcomed.

We interrupt our regularly scheduled blogging …

john-cavnar-johnson — Thu, 22 Sep 2005 13:39:00 GMT

to evacuate the Houston area. I've temporarily relocated to the Dallas area and will be participating in disaster recovery operations for my day job. Blogging will take a back seat for awhile.

What Matters to Mort at the PDC?

john-cavnar-johnson — Wed, 14 Sep 2005 14:07:00 GMT

The PDC is an incredible whirlwind. If you're a Mort and at the PDC, I need your help. I'm doing a BOF session for Morts, but there's no way I'll get to all the sessions that might matter to Mort. Here's some things I think we might want to talk about at the BOF:

Office 12, does the Office team finally get it, that they are a platform for Mort?
Avalon, do you think we'll be using any of that glitzy stuff in IT apps?
Workflow, can Microsoft get this one right? (I'm attending mostly of the workflow sessions today)
Visual Basic, looking ahead at VB9, does LINQ and the other enhancements turn you on? Would it make you give up C# (if you went that way when you migrated to .NET)?
Indigo, um… I'll let somebody else talk about that, I've had my say.
What am I missing?

If you have ideas about what matters to Mort, post them in the comments or catch up with me at the PDC. I'll be in a PluralSight shirt (but remember, I'm not an employee, just a very satisfied customer).

Dear Rich

john-cavnar-johnson — Sun, 11 Sep 2005 14:25:00 GMT

<Aside>This is good time to remind everyone that I don't work for PluralSight and they are not responsible for the content of this blog. Aaron Skonnard (who does work for PluralSight) graciously offered me this spot with the only proviso being that I use it. The opinions expressed here are mine alone, etc.</Aside>

You're welcome. Let me take a moment to emphasize what I think is the biggest problem with your whitepaper. In your response, you restate the aim of the paper ("to help you make the best choices when deciding how to build distributed systems today on the Microsoft platform") and then you go on to say that guidance about how to actually go about designing distributed systems is beyond the scope of the paper. I simply can't fathom how you propose to help people make the best choices in deciding how to build a system if you don't address how to design said system. Would it make sense to write an article to help people make the best choices when making fudge without including any recipes? Would you talk about how it's important to use chocolate, sugar, and butter without describing the relative portions to use? You could even describe how the molecular nature of peanut butter affects the consistency of fudge, but how useful is that without a recipe? To extend this analogy close to the breaking point, Microsoft may just be the store that sells the ingredients, but your success is directly dependent on whether or not people like me can cook up some tasty fudge.

Understanding our difference of opinion about that allows me to answer your final question. Your paper is neither full of worthless trash nor will it help the reader decide when and where to apply which technology when building their system. Your article is full of accurate information without the necessary context for your readers to put it to full use. My original response to your whitepaper was my attempt to provide the appropriate context that would allow people to successfully use Microsoft's technologies to build distributed applications. To return to my analogy, my original response to your article was my attempt to provide a recipe for some tasty fudge. I take your point that I wasn't entirely clear about my recipe. This is due, in large part, to two different factors. First, I devised my response to follow the structure of your article and supply my guidance as an alternative to yours. In retrospect, I see this was a mistake. In the end, it obscured some of the points that I was trying to make. Second, designing distributed systems is harder than making fudge and I really have two goals. I want to describe my favorite recipe for building distributed systems and how Indigo makes it more difficult to build systems they way I want.

Now, to respond to a few particulars in your response. You say that I appear to assume that you favor RPC style communication. Actually, I'll state that outright. When I use the term RPC, I mean exactly that: Remote Procedure Call. Of the approximately 4200 words in your article, 90% of them are spent talking about RPC-style technologies (.NET Remoting, DCOM, ASMX, and Indigo) and 10% talking about messaging-style technologies (which I define pretty broadly by including MSMQ, BizTalk, SQL Server Service Broker, SQL Server Notification Services, and HIS). You clearly state that distributed component technologies are a powerful way to build tightly coupled portions of your application. This is, at the very least, a partial endorsement of distributed objects. I strongly disagree with this, by the way. Let me be clear that I am quite cognizant that it is certainly possible to build powerful systems using distributed component technology. I've done it a few times myself, but there is a more excellent way.

Let me clearly articulate my alternative to distributed object technology and RPC-style programming models. I explicitly reject your formulation of the messaging-style approach. Message-based programming models need not be concerned with the low-level details of transports or the processing of XML streams. Once again, you trot out the old canard that message-oriented programming is not as productive as RPC-style programming. I would argue that for most business programmers, message-oriented programming is far more productive than the traditional OO approach. Message-oriented analysis and design focuses a few simple constructs: Messages, services, and events. It is very straight forward to map distributed business processes to these constructs in a way that never worked well with RPC. Here is *precisely* the opportunity that I think the Indigo team missed. At the heart of the internal model of Indigo is a very powerful concept of a message. Unfortunately, this conceptual model is never exposed to developers and architects directly. You can only manipulate it indirectly, either through low-level XML processing or by applying a set of opaque attributes to a .NET type. Clearly, for interoperability reasons, the wire format for this message will be XML and the structure must be expressed in terms of XML Schema (despite its complexity). But I see no reason I shouldn't be able to work with this message in the .NET programming language of my choice. The Indigo team has already built the infrastructure to make the transformation. All they need to do is to expose it to developers instead of hiding it behind those attributes or forcing us to grovel in the angle brackets.

Once we have a way of dealing directly with messages, we need a straightforward way of connecting the initiators of message exchanges with the responders. For the sake of simplicity, I'll call the initiator the Message Sender and the responder the Message Receiver, even though in many cases both sides send and receive messages. We normally think of the Message Receiver as a service and the Message Sender as the caller. The Message Sender and Message Receiver need to agree on at least the following things:

The shape of the initial message
The message destination
The shape of the response message (if any)
The response destination

I think anybody who's worked with the System.Messaging API can imagine a pretty straightforward API for these programming primitives. Instead of being limited to the MSMQ proprietary protocol and infrastructure, this API could be layered on top of all the interoperable goodness of the Indigo infrastructure.

Finally, I want to point out that Rich and I agree on a lot more than we disagree. Developers should spend their time building business apps, not infrastructure, for all the reasons Rich lays out in his post. Building a messaging infrastructure is devilishly hard and it is best left to experts like those who work for Microsoft. The only major point of disagreement is how to best expose that to application developers. I'm not even asking that the Indigo team throw out their RPC-style interface because I know that despite all the accumulated experience of the last 15 years of distributed application building, some folks still want to do that. All I'm really asking for is an alternative high-level API that allows me to map asynchronous messaging-based business processes directly on to the asynchronous messaging-based infrastructure in Indigo.

John

P.S. I liked the first version of your post better, but I understand why you edited it. Since we're both going to be in Los Angeles for the PDC, let's get together and share a cold beverage. This is an instance where I think synchronous communication might really be useful. Especially since I suspect we both would prefer an ESB from these guys to one from those guys.

Serendipitous, Indeed

john-cavnar-johnson — Sun, 04 Sep 2005 18:18:00 GMT

Wherein our narrator makes a fortuitous discovery in a most unexpected place.

One of the struggles I face in trying to explain my objections to the Indigo API is creating concrete examples of the abstract, and sometimes seemingly abstruse, notion I have that the Indigo API itself leads developers down the wrong path in building distributed applications. It's easy to come up with scenarios that demonstrate the ideas, but these scenarios often seem forced and are always open to the criticism that they're contrived specifically to prove my point. Imagine my delight and surprise when a post showed up in my aggregator that perfectly illustrates my argument and that post is written by a Microsoft expert in API usability. Please take a few moments to read and ponder the significance of this post by Steven Clarke. Now, I think Steven drew the wrong lesson from his experience, but before we get to that, let's take a look at the situation he set up. He has some UI programmers using data binding to link the UI to a backend Indigo service. This is a key scenario for SOA. In this case the problem domain is an album list. I'm going to assume (based on the names of the methods in his interface) that this is a personal list of music albums. Steven's understanding of his problem is that he made errors in the way he maintained state within his service and the assumptions that the UI programmers made. To understand how fundamentally wrong this is, we need to think a bit about the problem domain, the interface he exposed, and how the Indigo API enabled him to create the situation.

An album list is rather obviously a document. Except that to many software architects and developers, that's not obvious at all. The database folks will immediately think of a relational model and the OO crowd will think in terms of objects and their methods (state + behavior), but normal people will naturally have a conceptual model of a document. It is absolutely critical to maintain the document oriented view if you want to be build excellent distributed applications. The main reason that service orientation is the best way to build distributed applications is that it provides an effective way to map document-oriented business processes onto the physical constructs of a computer network. The basic principles of service orientation have been around as long as the pyramids of Giza. Indeed, they enabled the ancient Egyptians to build the pyramids. The system they used was based on the same asynchronous messaging architecture that I use to build systems today. Granted, their messages were either verbal or written on papyrus in hieroglyphics, their network used people to physically move those messages, and the services were all performed manually, but from an abstract systems view, it is the same.

Unfortunately, the current Indigo API doesn't encourage a document oriented approach to service design. Take a look at Steven's service contract:

[ServiceContract(Namespace = "http://Usability.Task")]
interface IAlbumService
{
[OperationContract]
Album[] GetAlbumList();

[OperationContract]
void AddAlbum(string title);

[OperationContract]
int GetNumberOfAlbums();

[OperationContract]
void SellAlbum(string title);

}

Now, if you're a veteran of the global struggle against rampant unscalability, you'll immediately see issues with this design. The AddAlbum method is the very definition of a chatty interface. And if you don't think scalability matters for an album list, I'd just point you this guy. More fundamentally, this interface confused the UI developers it was supposed to support. I don't think you place the blame for these problems with Steven Clarke. He's clearly a very intelligent guy. I've learned a lot from his writings on API usability. Nor can we write this off to inexperience. He's been working with Indigo as long as anybody. I think the fault here clearly lies with the API itself. What I find most interesting about Steven's post is Steve Swartz's advice. Unlike most Indigo developers, Steven has direct access to the Indigo architects. Steve's advice is very standard RPC thinking. He presents two choices. The first is add Open and Close methods to the API to represent the state management. The second is to add a parameter to each function to represent a session identifier. Both of these solutions pollute the service interface with details that are irrelevant to the problem domain and force the service to relinquish some its autonomy to the service consumer.

There is a better way. Microsoft's own Patterns and Practices group calls it the Document Processor pattern. If you apply that pattern, you get an interface that would look more like this:

[ServiceContract(Namespace = "http://Usability.Task")]
interface IAlbumService
{
[OperationContract]
AlbumList GetAlbumList(string Owner);

[OperationContract(IsOneWay = true)]
void StoreAlbumList(AlbumList Albums);

}

This is a huge improvement over the original API. No need for Open and Close methods, no extraneous session id parameters, and no chatty calls. But there is still a problem. I wouldn't want to tie up my pretty Avalon UI while waiting on Brad's 1000+ album list to come back from the service. Other implementers of the service caller might choose to block on that call, especially if the service caller isn't a UI. Unfortunately, the async design of Indigo is totally messed up. It's based on the async pattern of the .Net framework which is a pattern for calling synchronous local methods asynchronously. There is a much simpler approach that's possible when you realize that all network traffic is fundamentally asynchronous and that only the caller is able to make the choice whether or not its code should block waiting for the response. I'll outline this approach in my next post.

Love's Labours Lost

john-cavnar-johnson — Fri, 26 Aug 2005 13:34:00 GMT

In the comments, a couple of folks have responded to my criticisms of Indigo by suggesting alternative technologies for building distributed applications. I'm not looking for alternatives to Indigo. I'm working towards a better Indigo. I'm convinced that the underlying Indigo infrastructure can and should support the programming model that I envision. I want to help Microsoft avoid missing the opportunity to provide an entire generation of programmers with the tools to quickly and easily build scalable, reliable, and manageable distributed applications.

I think the Indigo team is operating under a set of assumptions that are invalid. They appear to believe the following things to be true, but I believe these things are false:

Asynchronous messaging is inherently more difficult than RPC.
Building business applications is just like building shrink-wrap software.
Service orientation is achieved on the wire, not in the programming model.
Ease of code migration is more important than wire compatibility.
Mapping between a service's external messaging interface and its internal representations is a mechanical detail that's best handled by the plumbing.

I want to explore each of these notions in future posts as I develop my proposed alternative programming model for Indigo.

Mort at the PDC (Updated)

john-cavnar-johnson — Wed, 24 Aug 2005 13:20:00 GMT

I'll be there. Although some folks think that Mort doesn't go to conferences, I've always felt that going to the PDC was essential. The PDC is important for two reasons, from a Mort perspective. First, Mort needs to know where the Microsoft is platform is going because there's really no other mainstream platform for Mort (except for those of us who are still doing COBOL on the mainframe which despite what the industry cognoscenti would have you believe is still very much a mainstream platform). VMS and DECBasic are dying off. Java never had a place for Mort. The Linux community is virulently anti-Mort. Apple's always focused on ease of use for the non-programmer (not that there's anything wrong with that, it just doesn't make it a good platform for Mort). Morts know that the Microsoft platform is our natural home. When they start planning on major renovations, we need to be prepared.

Second, Mort needs to be at the PDC to remind the product teams how important we are for the success of Microsoft's platform. With the exception of the Visual Basic team, most of the product teams at Microsoft don't understand Mort. The PDC is our chance to let them put a few faces on that persona. Most Morts don't have the time to blog, speak at conferences, or talk to the industry press. We really prefer to deliver value to our employers and spend time with our families. The PDC is our best chance to provide direct feedback to Microsoft on what we really need from the platform.

<ShamelessPlug>If you're a Mort (or think you might be … I really need to do the Jeff Foxworthy thing for Morts), join me for a BOF session to discuss whether this year's PDC will have a real impact or not. <Update> This session was accepted. Join me at 9 PM on Thursday, September 15th. I'll update again when I know the room.</Update></ShamelessPlug>

Indigo has a Simple Messaging Model?

john-cavnar-johnson — Tue, 16 Aug 2005 14:20:00 GMT

Steve responds to my Indigo complaints by claiming that Indigo, contrary to my assertion, has a simple messaging model. Although he doesn't specify exactly what he means, I assume he's talking about decorating methods with the [OperationContract(IsOneWay=true)] since that's what usually passes for simple messaging among the Indigo crowd. I completely reject that notion. A remote procedure call is still a remote procedure call, even if you don't get a response. You still have a proxy. You're still incorporating the service's method into the caller's domain. I will grant that with the appropriate discipline and design you can build a simple messaging model on top of this one-way RPC semantic, but Indigo really doesn't help you here.

In one respect, I think Steve's view and mine are both correct, if you understand the difference in our perspectives. Steve, as an Indigo architect, looks at what happens inside the Indigo infrastructure, when you use the one-way semantic and sees a clear messaging orientation. I'm looking at the API that's presented to Indigo's users and see a continuum of RPC semantics with nary a messaging API in sight. I would like the Indigo API to reflect more directly the messaging orientation without the RPC overlay. I want to use Indigo to send structured messages to a variety of endpoints without necessarily having to map them to a particular service method. This is clearly possible with the current Indigo API but is just as clearly not the preferred programming model. It's difficult to describe exactly how my ideal high-level API would differ from the current API without some sort of pseudo-code examples. I'm working on exactly that, but it'll take some time because I have a family and a day job that come first.

As to Steve's other points, his description of Indigo configuration led me to look deeper there and I see that I had a mistaken view based on the samples and demos I've seen. Now I have a different set of concerns about Indigo configuration that I'll have to address in later post. I'll just withdraw my last point. It was an unsubstantiated assertion. Of course, I still disagree with Steve about svcutil, but that's part and parcel of our disagreement about what are and what aren’t RPC semantics. I am still concerned about the way Indigo is presented to developers. After all the early emphasis on service orientation, I expected Indigo to present an API that is consistent with those principles to developers. Instead, up to this point, the articles, the samples, and the demos have consistently been all about RPC. Sure, I would prefer that Indigo not even have an RPC interface. It would be quite easy to replace that with a request/response messaging model, but I know that you have to support developers who want their distributed object model in spite of a couple of decades of experience showing it's a bad idea. My main point about being a Mort is that it's not the Morts of the world who want this. It's the Elvis-style developer who wants it. Real business apps map very directly onto a messaging model, much better than they map onto a distributed object or RPC model. I think Indigo is missing an opportunity to create a "pit of success" for building distributed business apps.

5 Things I Love About Indigo

john-cavnar-johnson — Mon, 15 Aug 2005 13:19:00 GMT

Tom Fuller left a great comment on my last post. Mostly, it reminded me that I ought to be just as explicit about what's great about Indigo as I am about what's not so great. To answer Tom's specific question, the point of all this negativity is to influence the design of Indigo before it's released. The Indigo team has a tremendous opportunity to improve the quality of an entire generation of distributed applications by providing a solid infrastructure for application architects working on the Microsoft platform. I want to do my part to help them succeed. I think they're missing the boat by not providing a more straightforward, Mort-accessible messaging interface. Without further ado, here's what's great about Indigo.

The people behind the technology. The Indigo team, as a group, is the most talented, passionate, and driven collection of geeks I've ever seen. [side note to my British readers: That sentence is grammatically correct in American English even if your way (team are) makes more sense.] Their willingness to engage the community (even grumpy old curmudgeons like me) is impressive.
The core technology for distributed programming is truly a thing of beauty. Deep down under the covers, Indigo has a well-designed engine. This gives me hope that the more cosmetic problems can be fixed.
The message layer programming model. Yes, I realize this is also on the list of things I hate about Indigo, but you can't have a simpler messaging model without a quality lower-level programming interface. I'm very happy that the Indigo team built this layer on top of the core technology.
The extensibility model. Even though I never plan to use it, I'm glad they did such a good job because it means the Einsteins of the world can extend Indigo for my benefit. If you're a Mort, this is vitally important.
[MessageContract]. This attribute is the closest Indigo comes to providing a simple message-oriented approach. Even though it's way too limited, the fact that it is there at all gives me hope.

The fact that this list is shorter than the first shouldn't be taken to mean that there's more bad than good in Indigo. These five things are vastly more significant than the 10 things I hate. Besides, I stretched the first list to get in an indirect reference to "The Taming of the Shrew". Next up, I'll take up Steve's challenge to be more specific in a slightly different way and try my hand at designing the simplified messaging interface I'd like to see from Indigo. It may take awhile. Designing API's is very tough and this blog is something I do on my own time.

10 Things I Hate About Indigo

john-cavnar-johnson — Sun, 14 Aug 2005 06:03:00 GMT

Steve is still confused, so I thought I would spell it all out for him.

The examples, especially the accursed Calculator example that's infected MSDN. The worst part is not that it is stupid and inane (right, you're going to go across the network to add two numbers together, Hello, you have a computer right there in front of you). The worst part is the subtext that tells the unwary programmer that using Indigo is just like a local method call. I really thought, based on your comments at PDC 2003 ,that you guys understood that location transparency is bad, but apparently not.
The client programming model that makes network roundtrips look just like local method calls (result = Calculator.Add(value1, value2). Why don't you have a simple messaging model for the client?
The service programming model that encourages you to directly expose methods to remote callers rather than responding to messages as events.
Typed services because they replicate RPC semantics. To quote the book, "working with typed services feels similar to RPC-style distributed object technologies such as .NET Remoting". This where you expect most developers (especially Mort) to work and it is the worst possible place.
Untyped services which are just slightly less tightly coupled RPC's.
Message-layer programming which where I'd like to be because down there it's obvious that Indigo is sending the messages for you, but it is so freaking complicated that mere MORTals will get hopelessly lost. Seriously, when we adopt Indigo at my day job, I know I'll have wrap this layer with a simpler, less complicated interface. It will look a lot like System.Messaging but it won't be limited to MSMQ. That's what I hoped Indigo would provide.
Attribute abuse. I've already harped on this one, but let me add that by using attributes you're insinuating your code into my domain model in completely inappropriate ways. I want to use Indigo explicitly, not implicitly via attributes.
Configuration is good for some things, but not others. You guys seem intent on repeating the mistakes of the past. In this case, it reminds me of MTS. We all learned pretty quickly after it was released that allowing someone to change the transactional model of a piece of code after deployment almost always ended in disaster. Likewise, several elements (transports, security, etc.) in Indigo are simply too fundamental to be usefully configurable.
Svcutil is evil. This is the tool I was referring to in my earlier post. Just like the attribute-driven approach, Svcutil infects my domain model with your code. Svcutil always imposes distributed object/RPC semantic (at least as far as I know)>
The "One Ring to Rule Them All" philosophy. The whole "unification" strategy looks more and more like you guys just took a whole bunch of unrelated and, in some cases, incompatible technologies and threw into one big bag without any thought as to the underlying architecture.

Let me sum things up like this. I want Indigo to stay out of my domain model. I would like a vastly simplified version of the message-layer programming model promoted as the right way for most developers to use Indigo services. If you took the best parts of System.Messaging and WSE, you would just about have it. I hope that clears things up.

Mort Gets the Message

john-cavnar-johnson — Fri, 12 Aug 2005 21:54:00 GMT

Wherein our plucky narrator reveals a shocking secret and makes a plea for understanding.

I am Mort and I write distributed applications. I have decided that this simple fact is the primary factor in the difficulty I have communicating my concerns about WCF (nee Indigo). I've had several occasions to interact (in person and via blogs) with various members of Team Indigo. They are some of the smartest, most professional, and dedicated folks I've ever had the pleasure of talking to and I've always felt there is a huge impedance mismatch in our conversations. Reading Steve Swartz's comments on my last two posts made me realize it is my fundamental Mortness that is the problem.

In case anyone doubts my Mort credentials, I will re-iterate them here. I prefer Visual Basic to C#. I don't have CS degree (English and History instead). I was introduced to programming through that gateway drug of my generation, Lotus 1-2-3 macros. I eventually graduated from Paradox to Visual Basic via Microsoft Access. I am an opportunistic programmer and I expect my tools to let me focus on delivering the business functionality my employer pays me for, not write a bunch of infrastructure code. I got to be an architect not because I'm a great programmer (I'm not, I'm Mort), but because I can visualize how to solve business problems with computer technology.

With that in mind, I'd like to respond to Steve's comments. I know that Indigo supports asynchronous messaging. I just don't like the way Indigo supports asynchronous messaging. Indigo, in an attempt to make asynchronicity easy to use, hides it from Morts like me. This is the exact same mistake that DCOM made with location transparency. It will have the exact same consequences that DCOM's location transparency had, distributed applications that won't scale. This is the most hideous trick that Microsoft can play on Morts. We'll make designs that look just like the Microsoft examples, work great in the development environment and blow up in production. Thanks a lot. And redefining what Close() and Dispose() mean doesn't help either.

I know that Indigo supports translating public contracts to application contracts. I just want direct control over the process, rather than being stuck with the output of your tool. Elvis may think that mapping the external message to the internal model is stupid grunt work that can be relegated to a command-line utility, but Mort knows that it is a crucial piece of business logic that needs it's own separate place in the world so that it can change independently of the rest of the application.

I know that Indigo supports XML programming (Tim and all the other Einsteins will be happy). I want to do XML programming about as much as I want to do IL programming (which is not at all, in case you're not sure). I'm glad it's there under the covers, but don't make me look at it.

It's not so much about what I want Indigo to do as it is about what I don't want Indigo to do. I don't want it to interpose an RPC model between the underlying asynchronous message-based truth of Indigo and overarching asynchronous message-based truth of the business applications I write every day. I don't want to have to dip down into an untyped bag of XML goo just to get a message instead of an object. I want Indigo to allow me to say, "Send a message containing this purchase order to that service over there and notify me when I get the response back with P.O. number in it" without making it look like a method call. If I want the client to block while waiting for the P.O. number, that's my choice, but it's always an asynchronous operation across the network and it should always look like an asynchronous operation across the network.

The main thrust of this post is that Morts get message-oriented development. Messages (documents and action requests) are fundamental to business programming. We don't need the OO façade that Indigo pushes on us, but we need more than SOAP. We need an easy way to send and receive structured (but not necessarily typed) messages. For all the talk of the Indigo big tent philosophy, I feel like Mort is left standing in the rain.