Gui As Conversation

Someone asked on the MindControlWithDerrenBrown page, why can't my pc "use the entire communications context with it up to that point"? Heres some thoughts on that larger picture, some of them less specific and more handwavy as it goes along.

The 'conversation' I have with a GUI is a conversation with a schizophrenic amnesiac -- it doesn't remember from second to second -- from dialog to dialog, from window to window -- what we've talked about, what I've been doing.

'Dialog' boxes implies I'm having a conversation, and sometimes it does seem that way: some apps have well-designed dialog boxes with well-written text. Over time, Apple and Microsoft and others have gotten better at building in little features that give the illusion the computer 'understands' me and what I'm doing (e.g. OSX sorting filenames with a numeric portion numerically, so that 1Apple comes before 2Billie comes before 10OSX; Windows 7 saying "you're trying to copy a music file to this device, wanna convert it first?", etc.). But this way of doing things is developmentally hugely expensive, and makes for a more brittle user experience: when the illusion shatters and it becomes clear, once again, that 'my pc' doesn't know what it's talking about.

Some specific examples:

I've had the 'this file is in use' dialog pop up in Windows with recently created directories I'm trying to rename. I don't have any files open in the directory, I don't have Explorer windows browsing inside it. I've had this message pop up for files I don't have open.

There are solutions to this specific problem, to be sure: WhoLockMe? and ProcessExplorer? can usually get to the bottom of why a file or dir is 'locked' (sometimes a process that hasn't ended cleanly; in one case it was Explorer trying to build thumbnails for .AVIs, etc.)

But here's what seems so strange to me, wearing my user-hat: someone had to go in and write that dialog text in english, triggered on certain conditions written in code, to ultimately produce a dialog with zero semantic content for the pc. Why doesn't my pc 'understand' what a file 'is'? What it means that it's 'locked'?

More to the point: Why can't I query it on what it means by the file is 'locked', ask it who locked it, etc. I.e. why can't this be an ongoing, stateful, meaningful conversation?

A similar example is all the different dialogs in all the different apps that offer a 'don't ask me again' option. Each one, in each app, is separately programmed, may store its setting who knows where, have a different mechanism for 'un'setting. Again, wearing my user-hat, that doesn't make sense to me. The whole 'remember this setting / don't ask me again' type thing should happen at an OS or 'OS servant' level, with some place where those 'settings' -- those conversations -- have a record.

Same thing with app notifications, which recently have gotten a little better in presentation (modeless temporary popups in system tray instead of system modal dialogs for example) -- but it still lacks semantic content, and it lacks a persistent record. Again, specific tools can be built to handle this specific case, maybe...but I want one tool, I want maybe a layer between me and the OS, unobtrusively modeless in a sidebarred text window with CLI perhaps.

It's as if I'm dealing with a bureacracy, or a badly organized corporation. Every time I go there, I get told something different. I didn't bring the right form. Sorry, that's not my department. Sorry sir, I'm not sure who you talked to yesterday but you were misinformed.

I'm a customer, I want to talk to the guy in charge. I want to talk to the same guy I talked to yesterday. In fact, I want one guy who's my contact, my interface to the company, who's going to know what we've been talking about and what my concerns are. I want a conversation, one that builds over time.

I know, I know: also, a pony! But I do have some concrete ideas, and some that are getting more concrete over time.

There's at least two things I'd like to see, as a user: 1) My pc 'remembers' what I've been doing. E.g., when learning a new piece of software, I've often had the experience of trying to get a specific result by changing all kinds of settings (let's say, file sharing between windows boxes with firewalls; or creating .ico files in Photoshop; or troubleshooting Active Directory replication issues). Maybe they're not un-doable, maybe I've been too busy experimenting to undo them. I'd like to be able to say, "dammit, where was that setting/option/tool i just was using, that dialog I just was in?". My history -- not just with this app, but with the OS, should be there and accessible. I picture maybe a modeless text window sidebar with a CLI; "You started Excel. You chose File->Open. You chose Format->Cells->Conditional Formatting. You chose Insert Column."

2) The much more difficult, and much more powerful, idea of the computer having some 'understanding' of what i'm doing, of there being semantic content underneath what it 'says'. Yes this is vague. Bear.

I'd like my PC to 'understand' common file operations in apps, so that the above sidelog could read "You opened foobar.xls in Excel". I'd like to be able to teach it about specific apps, so that the above sidelog could read "You opened foobar.xls in Excel. You entered (+expandable node here)some data. You selected (+expandable node here)cells and gave them conditional formatting. You inserted a column." And for this text to be a representation of underlying 'semantic' content, so that I could ask "which column?" and in that particular context, it could give me an answer.

I have notions of being able to build up an 'understanding' of let's say, what an mp3 file 'is' -- it consists of audio data (which can be played by tools X and Y and edited by tool Z) and metadata, this is what the metadata looks like, which is similar to wma metadata (superclassable) -- and have that 'understanding' become part of the OS. Then, combined with already existing notions of files, being able to 'tell' the computer to set all the ID3 artist tag based on top directory. Or to create a playlist of all 'genreX' songs. Of being able to save that mp3 file 'definition' in an format so that libraries of filetypes, capabilities and operations can be built up.

I'd like to be able to tell my pc, most apps have their 'settings' in this 'tools->options' menu. Some in 'edit->preferences'. Make a note when I change those (i.e., when dialogs get launched, and values changed, and OK clicked).

I'd like to be able to persist such "explanations" or "definitions", exchange them with others, build up libraries.

Wearing my developer hat, I don't yet have a clear idea how to develop all of this. I have some specific ideas for specific parts. And I have the general notion that this should be a bootstrapping type of deal, an extensible, expandable system that once off the ground, will build itself. There are aspects that are similar in LanguageOrientedProgramming and IntentionalSoftware?. Maybe the place to start is in building developer tools, and wanting to build better ones, and wanting to build better ways of building better ones better, etc. --CliffStabbert