Graphical Programming Language

The term "visual programming" seems to have been appropriated to describe GUI builders for conventional textual languages [for this see VisualProgramming, VisualProgrammingLanguage].

This page is about truly graphical languages, in which the source code is itself graphical in nature and does not principally consist of text. Normally, a chunk of code is a directed graph of some sort, where the direction of the graph edges indicates control or data flow.

There are somewhat weirder versions, where for instance graphical primitives are locked into a grid (like in the old MacOs game ChipWits), or some other structure, but these tend to be used for special-purpose scripting applications. -- BillTozier

Examples:

Teaching languages: Links: See also: VisualProgrammingBook.

For discussion of advantages and limitations of graphical languages, see GraphicalProgramming.

Upon more research in existing work, I found a system called bounce. It doesn't appear to be a general purpose language (it's meant for VR programming), but it does have some interesting stuff. Check out http://catalog.com/hopkins/lang/bounce/bounce.html

JaronLanier was working on this problem when he invented VR.

Does BefungeLanguage count?

A good question. BefungeLanguage, SnuspLanguage, or Ward's BiotaLanguage (and TomRay's original Tierra language, for that matter) certainly rely on the structure of the code in a way that is unlike traditional source. But I'm not sure whether the people who (for instance) write their Ph.D. theses on graphical programming would lump them together with Prograph, say. Mayhap the natural lump/split line lives between "Graphical Programming Languages" and something like "Structural Programming"? Or is it between "Drawn Programming Languages" and "Typewritten Programming Languages"? -- BillTozier

Would that make PythonLanguage, with its significant whitespace (SyntacticallySignificantWhitespaceConsideredHarmful?) a GraphicalProgrammingLanguage? -- EarleMartin

Certainly not. That would water down the term to the point of almost being meaningless. We like to have pages laid out in some kind of nice 2D format regardless of what is being displayed, but the information presented by properly indented CeeLanguage or PascalLanguage is no different from the syntactically required whitespace in PythonLanguage or HaskellLanguage.

This kind of thing just shows that it is difficult to create strict definitions that capture true intentions and creates a 100% correct dividing line between true category members versus non-members. That's an issue with all definitions, not just this one.

There is one neat language with graphical syntax: Aardappel by prolific language designer WouterVanOortmerssen.

http://wouter.fov120.com/aardappel/index.html http://www.google.com/search?q=cache:wouter.fov120.com/aardappel/

The [graphical language] that I use and like is LabView [i.e. GeeLanguage]. Initially, HP came up with the idea, but they thought it would never fly. When NationalInstruments? [http://www.natinst.com] picked the idea up and made it a pretty big success, HP thought they could resurrect their own in HP-VEE. Labview did start out with modules written in another language, but it has extensibility and allows you to bring in code from other languages. -- BenWillems

Prograph is the closest to a true graphical language I've ever heard of, which used the DataflowProgramming paradigm for algorithm description. Fairly recent page of links at http://www.tritera.com/prograph.html and as far as I can tell it is still in commercial production although the company Pictorius looks as though they mainly do Java now. http://www.pictorius.com/ [BrokenLink]

Another interesting product was Serius, bought by Novell in 95 and renamed Visual AppBuilder? which used visual programming to tie together components written in more conventional languages. There's a Byte article discussing Serius, Prograph et al: http://www.byte.com/art/9407/sec12/art2.htm

I have a longstanding fascination with different languages and development paradigms but never had a chance to try either of these (my mild amount of self-restraint comes down to only using stuff in a commercially-oriented project, I have a life!). -- AndyDent

I know that the Quest3D engine by Act-3D uses [a graphical language]. The program is for visual presentation of CAD designs, and many also use it for 3D games and things like that. Judging from the results this seem to work quite all right - especially for beginners.

And there are also GraphicalProgrammingLanguages in several high-end 3D applications like MaxonCinema? and AliasMaya? (AFAIK). This is really a big help to create simple dependencies, that would otherwise need you to learn a special scripting language.

And also there are several simple GameMakers? that use a graphical approach - like ClickTeam?s MultimediaFusion? (although different from the LabView idea), among others. -- ThomasSchmall?

Not to mention one really, really big one coming. UnrealEngine 3.0 will be using a multimodal GeeLanguage-style GraphicalProgrammingLanguage called KismetLanguage?. Kismet will be primarily graphical, but include a text-mode for writing procedures that are easier in text (like mathematics). Keep in mind that this is the UnrealEngine, which is notorious for having the entire classmodel of the whole engine represented in the scripting system (although many of the internal classes are "hands off"). In about a year, a very large and complete graphical programming application will be hitting the market (release for UnrealTournament 2007 is slated for 2006). Plus, another, less popular but similar gamemaking app called VirTools? already exists based on a similar multimodal language, although VirTools? is considered a prototyping toy by gamedevs. -- MartinZarate

See also FlowBasedProgramming. For some sample diagrams, see http://www.jpaulmorrison.com/cgi-bin/wiki.pl?DrawFlow. By the definition of GraphicalProgrammingLanguage given above, it is only a partial one, as the graphical language is used to combine components written in various procedural languages, but we have found that the diagrams provide an excellent communication medium between users and developers, especially when most of the components have simple, self-explanatory functions, e.g. readers, writers, sorts, merges, etc. In addition the "port" concept allows these components to be generalized functions that are prewritten and pretested. Even the application-specific components can be given names like "generate report", "summarize", "access such and such a database", etc., that help with the visualization process. -- PaulMorrison

On the term "visual programming language":

I think that the term "visual" itself is pretty poor. All programming languages that I know of are visual. You look at the text. I'm sure that there are some blind programmers out there, but I shudder to think of them using C++. Imagine what that would sound like? Or, feel like in braille?

I think the nomenclature predates the prevalence of the GUI, so it's a historical name. I agree that all languages are "visual" in the sense you mean - the distinction here is between languages you type, and languages you draw. The latter ones are "visual" in the way the term is technically used. -- BillTozier

MicroSoft has been using the word "visual" in VisualBasic and related development platforms for some time, but frankly the only thing that is "visual" about them is a handy way of building the layout of interface components. Sadly, over the years this has undermined the efforts to produce real visual/graphical languages like those referenced above.

I know that in the past graphical programming languages have been a joke. But recently I've been wondering if there was a way to create a serious graphical programming language. I think that such things could have numerous advantages. One possible one would be to be able to more intuitively see how a program works. Another idea was that we could get rid of the textual vagaries that make smart development environments a pain to write.

The first thing that would need to be done differently is that the idea that this would be simpler than text programming has to go. What else would have to be different to make it work? Is anyone else doing research on this area?

(moved from FutureOfProgrammingLanguages)

I'm not sure the future of programming languages is even a "language." I imagine that we'll be connecting components on a bus architecture. We'll rely on visualization, because the layout of the code will be much more schematic than it has been. We've moved in computing from more linear algorithmic (flow-charts) to the more divergent (UML ClassDiagrams, for example.) I can only see this continuing: The relationships between modules will become increasingly less linear, and we will find that our expressive technology will be a problem.

When we finally make a good GUI platform, we'll find that our languages suddenly leap into the realm of schematics.

(What sucks about GUI technology right now? It's so primitive, that's what! Sure, it's great if all you need are buttons, text boxes, combo boxes, and trees. But good GUI technology will make it easy to program by setting icons down on a surface, rotate them, give icons awareness of neighboring icons, and how they are situated with respect to one another, automatically mirror internals in the external display, allow people to easily compose icons, shift scale, etc., etc., etc.,. The GUI technology we have now is peanuts compared to what we need. The only reason we're still using text, isn't because we love linearity. No, it's because it's the easiest to navigate and work with, right now.)

-- LionKimbro

I'm going to be contrary, and say that the primary representation of programming will continue to be textual, though there may be visual views that are useful in certain circumstances or other alternate views. (For instance, the modern class browser is graphical, but is still fundamentally text based. Dialog boxes are borderline, but while the dialog box itself is probably best understood graphically, there's still an underlying text program that is the core of the functionality.)

While I normally hate to analogize computer languages to natural languages, computer languages won't become visual for reasons almost identical to the ones that have natural languages moving away from visual representations, not towards, even today. Other than visually pleasing typography, playing with visual representations like spiral typesetting or dense visual symbology is mostly the domain of poets and artists, not people seriously trying to communicate. Looking and seeing visual information is easy. Manipulation is fundamentally more difficult; so much so that I've never seen anyone come even close to a truly visual programming paradigm that actually works usefully beyond the "hello world" example. Too much software just isn't manipulable visually, and showing the few cases where it may be hardly disproves that.

This is another one of those "In fifty years" predictions that keeps receding at the rate of one year per year. The reason is that it sounds really cool if you just hand-wave it into existence, but there are fundamental flaws with the approach that severely limit its usefulness.

That said, there's significant room for the continuing advancement of the re-factoring editor that understands more about the actual syntax of the program and may be able to do wild visualizations that may even be helpful; my point is not that no visualization progress will happen but that for serious programmers, the text representation will continue to be the "real program".

I'd predict that we will move to a multi-modal style. Sometimes a textual view is better, sometimes graphical. As tools improve, we'll have various ways to "frame the code". Dependency graphs and visual gui editors are excellent complements to textual code.

I can't resist putting in my 2 bits: FlowBasedProgramming (FBP) is a very visual coordination language that allows the programmer to combine asynchronous components into a 2-dimensional network. Because communication is via ports, it lends itself to prewritten, pretested componentry, so it cries out for a visual interface, which could easily be extended to have quite a lot of smarts, e.g. knowledge of what ports a given component uses - put them up on the screen, connect them, run it, and maybe even watch the data as it travels (test mode, not production!).

The components can be written in any language that understands data packets - granted that the components are coded in text right now, but if they are reusable components, who cares what language they were written in? However, some business transforms can be very visual - some components in the network will eventually be generated from pictures themselves. FBP has been in continuous use at a bank for 30 years - we were using it for production work long before we had decent graphics support (we used biiig sheets of paper!). Now we've got the technology: anyone want to step up to the challenge? The thing might finally take off! -- PaulMorrison

I've been thinking about it. The gap between your book and the issue of what support to put into a programming language is fairly large. I'm accustomed to bridging such gaps, but still...any comments on that? -- DougMerritt

What kind of support? Organizational? An IDE? Money?! Could you expand on your question? Bridging gaps is good - that's what Pontifex means. :-) -- PaulMorrison

I just mean language constructs to support FBP natively. Or similarly, the ideal set of library functions to support FBP. -- dm

...

Never mind, I'll just stick with the FBP primitives I invented for the 1983 version of the language in question.

Can you point us at them? Maybe put something up on the FBPWiki? -- pm

That said, I could see having a small number of mini-languages running in the FBP environment whose job is to shift stuff from input ports to output ports, perhaps with small transformations, subtotals, etc. Approximately the expressive level of RPG, or less. In fact, as I suggested elsewhere, the logic could even be specified graphically, given an appropriate IDE (imagine a business-oriented ToonTalk :-)). Remember that FBP is primarily a coordination language, so there is no need to push a language (any of the ones used for writing components) beyond its comfort level!

More generally, I think a possible approach would be to establish a lowest common denominator for communication between FBP components, and I think this should probably be ASCII strings, probably with embedded delimiters, which I guess nowadays sounds a lot like XML! Except that the overhead involved means that you should only do this at the boundary between component vendors - between components that are more closely related we need a convention that is not so CPU-intensive! However, FBP is a natural environment to support streaming XML parsers. I wrote one using the so-called "Babel Parser" (http://www.jpaulmorrison.com/cgi-bin/wiki.pl?BabelParser) for the project we did in JavaFBP (the Java implementation of FBP), and we multiplexed it (3 ways) to take advantage of the multiple processors on our machine.

If we assume also that, in the PC world, code components can always be turned into DLL/.so libraries, then we can use FBP to hook together any languages that understand ASCII strings, and that can be compiled to a DLL/.so. I am also assuming that most compiled languages can handle pointers to ASCII strings... Languages that require VMs would require native interfaces. In such an environment, some threads would be running essentially machine language (generated by the compiler of your choice), and others would be interpretive using VMs - then hopefully there would be no need for a VM to support multiple threads. This would then give us the rather neat ability to write individual FBP components initially in a scripting (interpretive) language, and then switch to compiled code for performance reasons at a later stage in a project's life-cycle (avoiding PrematureOptimization).

Looking into this, I realize that I didn't publish the JavaFBP component API yet, although it is in the jar file on my web site. I've put the APIs for THREADS and JavaFBP (the C++ and Java implementations of FBP, respectively) into http://www.jpaulmorrison.com/cgi-bin/wiki.pl?FbpApi.

I think Rocky's Boots [http://www.warrenrobinett.com/rockysboots/] and Robot Odyssey [http://members.aol.com/Fractal101/odyssey.htm] deserve mention here.

Labview uses true graphical programming (G language). It's a very intuitive way to program. In my opinion, about 10 times faster than old-style text-based programming for most applications.

I'd be interested in learning what it would take to integrate graphical programming with a more traditional text-based programming language. I don't believe that graphical programming entirely lends itself to general purpose programming (e.g. they are generally inadequate for conceptually atomic processing steps... such as mathematical functions and initial data description). However, the use of graphical languages have proven very natural for describing flow of any sort: WorkFlow, DataFlow?, and SignalFlow?. They essentially allow the internal processes to be nameless, and abstract away much of the nasty interfacing between processes and objects. It also allows for automatic zoomable interfacing, going from the 2000 meter view (inter-business processes, web services) down to the 10 cm view (the exact flow within a particular procedure within a particular process within a ...). Such things are good for communicating to PointyHairedBosses.

How difficult would it be to create a graphical programming language that handles those tasks but corresponds directly to the use of certain underlying text programming, such that the graphical programming language is just a DomainSpecificLanguage for workflow, dataflow, signal flow, and that sort of thing, and a possibly typeful sublanguage of another language (Lisp, Ocaml, Haskell with Monads, etc.). I imagine something like a common set of wrappers could do the trick... a limited set of domain specific languages that are readily and intuitively translated to and from graphical components, and that integrate with an underlying language in some well-defined way. However, that still leaves the difficult parts: creating/identifying and integrating these DomainSpecificLanguages that are good for graphical programming, ensuring they cover everything you and other users will possibly need and that they provide for extensibility (graphical macros?), and creating an ObjectBrowser for all of this.

Key to finding what DSLs we want is finding: What are the LimitsOfGraphicalProgramming?? It's easy to make a graphical language that is turing complete, but at what point will users stop gaining and start losing? Are the inherent limits different between 2D and 3D graphical programming?

I posit that the limit of graphical programming is conceptual atoms -- values, mathematical (pure) functions, and calculations.

KismetLanguage? (the language of the upcoming UnrealEngine 3.0) has the system you described - a traditional, flow-based GraphicalProgrammingLanguage with a c-like text language that can be embedded into blocks within the flow-code to handle things like arithmetic that are much better described in text.

And I think that the "managerial view" is unlikely to be as nice as you think. Consider how well managers can read Cobol. Consider how little you learn from one function of the outermost code of your system - flow-based programming is functionally the same as traditional functional programming languages except that the threading is automagic. All it is is "take results from call Foo and send it to calls Bar and Baz as argument Quux" - whether that's phrased as 

 tempfoo = Foo();
 Bar(tempfoo);
 Baz(tempfoo);
or a line drawn from Foo to the Quux input on Bar and Baz is just semantics. Theoretically, you could selectively "expand" functions just like one could theoretically "inline" important methods to make the logic more clear in a code view in a text language... but notice that nobody does.

"Consider how little you learn from one function of the outermost code of your system - flow-based programming is functionally the same as traditional functional programming languages except that the threading is automagic." -- Do you mean procedural? Technically, there is no flow when working with functions (in the mathematical sense); '1+1' is completely equivalent to '2'. This is part of why I posit that functions/calculations/values cannot be readily transformed to graphical languages. If you've fully described the flow, above, then: 

   myBlock = { tempfoo = Foo();
               Bar(tempfoo);
               Baz(tempfoo);    }
is completely equivalent to: 
   myBlock = {}
This is because nothing is flowing out of the block.

Now, if Foo() returns an object instead of a value, or Bar or Baz have side-effects, these facts simply aren't being represented. I.e. you cannot tell by looking at it whether tempfoo is a value (immutable) or object/actor/process/etc (which can send and receive messages). You also cannot tell whether Bar and Baz send/receive signals to and from tempfoo, the world, both, or neither. You can't even tell whether Foo() sends and receives any signals to and from the world or not.

These things would be represented in a flow-based language. It would be misleading to call the procedural code above the same as outermost code of a flow-based language because there is no indication of what is flowing or how it is flowing. And while one can produce a language that is computationally equivalent to a turing-complete procedural language, it wouldn't be right to call that language a flow language unless it is representing the right sorts of things -- flows in particular.

A language that actually captures the flow should allow you to learn a great deal by looking at the outermost code of your system, even if you 'box' certain processes (e.g. with Bar and Baz) to allow for something of zoomable interface and software componentry. In particular, you should be able to learn every single source and destination for messages to and from those boxes (or some abstraction on that, e.g. if input data contains references for connections). You'll have graphically captured the side effects, or at least identified their potential sources. You should be able to figure out where signals are going, and where signals are coming from. For fully typed flow-based language, ideally, you should also be able to look at it and learn both legal protocols (message patterns between two boxes on a single, connected line of 'flow' that don't cause runtime exceptions to propogate out of the 'box') and identify some emergent behavior (patterns of connections, for example).

I imagine that type Bar(Foo) in a flow-based language would simply be an abstraction that hooks up an object with a Foo interface to a process Bar. Connecting the WorldObject (or RootObject) to Foo and Bar would be a separate task... a separate connection, a separate flow.

You might want to look into SubtextLanguage. It doesn't loose the flow - quite the opposite. It may be only partly graphical in that it still uses functional expressions (except for the schematic table part), but it does'' make all flow visible (more so than the code itself). See the demo at http://subtextual.org/demo1.html -- GunnarZarncke''

I think some graphic/visual languages are missing from this wiki:

Limnor Studio: A visual programming language for .NET currently free (April 2011) http://limnor.com/

Tersus: Commercial and open source versions (MS Windows, Linux) development of Windows, Linux, web, Android and Iphone applications http://www.tersus.com

Google App Inventor: Open source development tool for Android OS, http://appinventor.googlelabs.com/about/

CategoryUserInterface

EditText of this page (last edited December 2, 2014) or FindPage with title or text search