Graphical Programming
Graphical ProgrammingSee GraphicalProgrammingLanguage for a definition of that term, and for particular languages and systems.
Discussion of limitations and advantages of graphical programming (or of compositional programming models that could be represented graphically) goes on this page.
I've thought about this topic a lot and it is my opinion that we cannot have a truly GraphicalProgrammingLanguage until we separate the concepts of algorithmic programming from compositional programming. Currently, all programming languages are algorithmic and slightly compositional. In the best world, one would develop component patterns using an algorithmic language. A Component Pattern (or template) could be something like Collection. It would have many fields one could fill in like favor size over speed, access elements randomly, always add to the end, and such. The Component Template would then be instantiated and used by a compositional language. Depending on its template arguments, the Component Template would be instantiated into any number of components - a single linked list, a linked list, an array, a sorted array, etc. If you think about it, this concept is very different than parameterized types but does extend on the concept of genericity.
While the algorithmic language would have features that allowed these types of programmers to express algorithms, a compositional language would explicitly prevent the programmer from doing anything algorithmic. This constraint actually allows applications to be more correct. The compositional language would have special features for composing and integrating components into full-blown systems. The algorithmic language could only product Component Template (or Patterns). It would be very difficult to product a complete system in the algorithmic language. Finally, the compositional language would be visual. However, the visual metaphor would not be the User Interface, but things like States, Classes, Roles, and Connections.
The idea here is once again my idea of being a good this doesn't mean you have to be a good that. The best application programmers may be awful at low-level algorithms and data structures while the greatest component builders may not be so good at building a high-level system. If some sorting routine in an application was too slow, the compositional programmer would not replace an insert sort with a merge or heap sort. Instead, she would simply change the Component Pattern for the sort routine to favor speed over memory. She need not know anything about merge, heap, insertion, radix, or selection sorts. What she does know about is creating great user experiences. In this sort of development system the source code really is the design. Or more correctly, the design is the source code.
Imagine a warehouse of Component Templates. Some of them are generic abstract data types, others are collections of business rules. The application composer simply grabs the required components and starts arranging them using something similar to a sequence diagram or Harell state chart. She may deal with objects but rarely with classes. Instead, she is concerned with roles, there connections, and state transitions. Instead of finding bugs in the algorithms, the bugs are found in the system design.
If the composer needs a new Component Template, she merely logs onto one of many on-line Component Template Repositories to find what she is looking for. Each Component Template is optimized and debugged for each of its many instantiations. Some Components Template could be very sophisticate, like those in Oberon where, in addition to their template arguments specified by the composer, they could even survey the machine they are on and optimize their instantiation for that machine. Each Component Template should produce components that are interpreted when instantiated. However, this is not a requirement of the idea. There is a bunch more but the basic idea is to separate algorithmic programming from composition. Then, do make the visual metaphor of composition the actual components, roles, and states instead of the User Interface. Each platform would garner a User Interface appropriate for its native Look & Feel. The composer is, instead, interested in the fact that a Loan Applicant submits a Loan Application that transitions the state of the Loan Applicant to the Approval Pending state. The composer can choose from many different modeling languages to represent these roles, states, and transitions.
My main heretical utterance is that Application Development is not (or should not be) programming. It should be integration and composition. It is Component Template vending that is programming. Applying programming to solutions that require construction results in unmaintainable and buggy systems. Unfortunately, we do not yet have the tools to prove these ideas.
If you want a set of inputs mapped to some blackbox functionality mapped to some outputs, you'd probably best go with FunctionalProgramming, not ObjectOrientedProgramming or ComponentBasedProgramming. Make each little widget a function. -- SunirShah
If you think the algorithmic language sounds like functional programming, I probably did a bad job of explaining in my attempt to simplify things. FunctionalProgramming is certainly great and getting better all the time, but it doesn't have any of the facilities needed to create the Component Templates I have in mind - much less instantiate them. Something like Oberon and its compilation model is a little closer. But it too has problems that would make it difficult to create Component Templates. Each component template is more like a mini-language than it is like a traditional Component, Class, or lexical closure. Instantiating one is like executing a compiler. In this way, they are more like Classes that produce other classes when instantiated. Something like this could never be done with a bunch of #mapcar type features. Actually, you couldn't even create the simple Container component-template with FP functions. However, I'm sure an appropriate language would definitely have some time of closures or blocks in its arsenal. -- RobertDiFalco
Robert, are you wanting something along the lines of a declarative language for building applications? Something along the lines of DeclarativeGuiLanguage, only much more general? -- CurtisBartley
I think that is a big part of it Curtis. There are really two pieces and both need each other. You can think of it as a Producer/Consumer relationship. The programmer produces component templates using an algorithmic language and a composer consumes these by instantiating them into full-blown components and modeling them into applications.
While analogies have been a recent PITA for me I will hazard a try... Think of the composer (for lack of a better word) as similar to a construction worker or house builder who using beams, nuts, bolts and various other bits that are made to specification. S/he is the consumer. The creator of the nuts, bolts, and beams is the supplier.
Now, how many construction teams mill their own 2X4's? How many forge their own nails, bolts, and beams? Even worse, how many actually formulate their own alloy for their nails? Would you decide to not hire a builder because they did not know enough about chemistry to create the alloy used in their nails? Of course not!! It should be just as ridiculous to expect a great application developer to know what a Radix sort is. I'm not saying this is possible, certainly not now. But this is the way it should be. The skills required to create great algorithms are almost at odds with the skills required to create great user experiences. That's not to say they can't be possessed by the same person. I'm only saying it's a shame that many great applications builders are never found because they have no feel for algorithms and vice versa.
So, in the future, a programmer will use some type of algorithmic language to create Component Templates (or maybe Component Compilers) that are individually tested and profiled for load, correctness, availability, etc. These "templates" are then instantiated into full blown components based on the specifications provided by the composer. The composer or builder specifies the component compilation criteria. The composer then models the states, roles, and interaction between compiled components. It is very important to remove all algorithmic features from this compositional modeling medium - it shouldn't even be called a language. I think the most important feature of the compositional medium (in addition to compiling components) is its modeling features. It should allow the application composer (or component integrator) to model states, interactions, and roles without any programming. Further, a role does not map to a class or object. These concepts don't exist...it is just a role. I imagine an environment that looks like anything between the UnifiedModelingLanguage (and all its diagrams) and OakTreeDiagrams. The visual metaphor must not be the user interface. Of course, the modeling idiom is just a mapping and a system modeler should be able to use any modeling idiom they prefer - just as long as it allows them to model states, roles, and interactions. -- RobertDiFalco
It seems that you are describing an ArchitectureDescriptionLanguage and the idea of SoftwareIntegratedCircuit?s. There has been a lot of academic research into this topic over the last 20 years or so, and it has been used in some commercial projects, including inside Phillips TV sets. -- AnonymousDonor
Some of this sounds like GenerativeProgramming. See for example http://www.extreme.indiana.edu/~tveldhui/papers/dagstuhl1998/dagstuhl.html.
It seems to me that we will want to build big components out of little components, which means the two languages would need to alternate. See also AlternateHardAndSoftLayers. -- DaveHarris
Yeah, I'm sure its not an original idea. I just never heard it before, and it has always seemed right to me. If you perform application integration using a compositional system that could not express algorithms (only instantiate/compile them), but was able to do this in a way that was not limiting, we would be much closer to making programs somewhat quantifiable. Of course, this is predicated on the quality of the Component Patterns created by the programmers. I think it is our natural reaction to think that one would need to switch between the two only because it is such a foreign concept to us. None of us have ever created applications in this way and there are no tools to even try. However, I'm certain that if the modeling/compositional medium was rich enough as far as modeling states, their transitions, roles, data, and interactions, one would not need to develop algorithms. They would also need a rich enough repository of component patterns. These need not all be huge or aggregate business logic units, there could even be a component pattern called Add, Multiply or, collectively, Math. The pattern inputs are the data types and values. In the graphical modeling/integration language these would be specified and related to other fully-specified components.
There's some math behind these ideas and maybe one day I'll get around to writing a decent abstract on it. However, it's always seemed such pie in the sky so other work has always taken precedence. Actually, I think AspectOrientedProgramming also has potential in this regard. It still requires the wedding of algorithms and integration features, but it is a good stride in my direction. -- RobertDiFalco
"None of us have ever created applications in this way and there are no tools to even try." Robert (I assume), I just have to respond to your blanket statement: I have been doing exactly this for 30 years with FlowBasedProgramming (FBP) and its descendants. One of the basic ideas in FBP is that it should be possible to build quite interesting applications just by hooking together prewritten, pretested components, and specifying parameters (which can be scaled up to mini-languages). FBP's InformationPackets are structured data packages (like ReferenceObjects), and so imply a certain level of granularity. My early experiments persuaded me that going to a smaller level of granularity in the data would result in excessive complexity of the networks. OTOH some of the references on these pages suggest that one might be able to usefully represent the logic of FBP components graphically, but IMO it isn't necessary. See also my answer to Sunir lower down on this page. -- PaulMorrison
The thing is, most of the time textual syntaxes have enormous advantages over graphical ones. In particular, most graphical syntaxes do not make it easier to understand the program.
(This lost some context during refactoring, but it was lacking support on the original page as well. Anyone want to support it?)
After meditating on graphical programming for a little bit, I've come a distressing conclusion: GraphicalProgrammingLanguages increase complexity rather than reduce it.
If you consider software to be a path/trajectory in n-dimensional space where n is typically large, it can be reduced to a vector due to its somewhat linearity. This vector can be projected onto one dimension very easily by serializing it. This is reflected by textual languages.
Adding a second dimension with graphical programming should logically increase the code density by a factor of four (2 squared). However, while a vector is guaranteed to serialize with no problems, it is not guaranteed to project onto two dimensions as a planar graph. This means those little connectors between the components may cross, causing great confusion and ugly diagrams.
Just to add more complexity, programmers have given meaning to spatial relationships between those boxes over the years:
+-----+
| |
+-----+
|
|
+-----+
| |
+-----+
does not mean the same as:
+-----+ +-----+ | |---| | +-----+ +-----+(one looks like a parent-child relationship and the other looks like a sibling relationship)
Consequently, programmers spend more time prettying up the diagram than actually solving problems.
Other problems abound, like the inefficiencies in using a mouse, or following process flow (like branches) in two dimensions versus one, or the information overload that your brain's massively parallel image processing hardware gives you versus the linear speech centre that processes written text.
Thoughts? -- SunirShah
Programs are only linear/sequential if you thinking in terms of a VonNeumannArchitecture. This view is probably quite appropriate for "mathematical" programming, e.g. calculating the evolution of a star, but there is a large class of program (esp. in business apps) where a better metaphor is that of streams of data being sorted, selected, merged, transformed, etc. The diagrams that are appropriate for this metaphor seem to ramify naturally in 2 dimensions, and the mind seems to have no trouble handling this - in fact, it has been found to be a very good communication vehicle between programmers, users, and (gasp) even managers! See FlowBasedProgramming. -- PaulMorrison
To me, graphical programming is representing chunks of functionality as icons or pictographs, as long as the representation has some properties of Lego (c) in that you can stick pieces of it together. I've tried to think of an efficient representation, but I always run up against the problem of drastically increasing complexity. The idea that struck me as most usable so far was Raymond Feist's system of TattooRuneMagic?.
It seems to me that true advanced graphical programming would require users to learn many pictographs like oriental languages, and to have the separation from the system of script kiddies.
This is a common misunderstanding of Asian languages. They are not pictographic, and this is not a terminology issue. All written languages have pictographic roots, but by the time a written language becomes sophisticated enough to be able to record essentially any spoken sentence, it becomes phonetic in nature rather than pictographic.
Thus in Chinese, the characters do not stand for pure ideas, they represent spoken words, just like in English. The pictographic ancient roots of some characters (e.g. used to be a picture of a house) have relatively little to do with the modern usage of the character, except sometimes as mnemonics to students or to scholars of language history.
Whether pictographs unlike those in Asian languages may be useful in graphical programming, their record so far is unimpressive, but certainly icons are in common use. Look at the top of the browser you're using right now. Shrug. Some of these icons work well, others don't, and in general there's no evidence that they work very well for expert users, and only limited evidence that they truly work well for beginning users. But they're pretty and flashy. Oooh, shiny! Ship it! :-)
Chinese characters do have pictograph roots and many characters still bear physical resemblance to what they represent. For example, the symbol for "door" looks just like a swinging saloon door pair. The symbol for "woman" is shapelier than the symbol for "man". The symbol for "ocean" (or was it "water"?) has a splash to it. The symbol for "middle" is a square with a line down the middle. However, many don't reflect physical things due to historical accidents and phonetic cross-overs.
And the letter "A" that I just typed is derived from a pictogram of an ox head with horns. And the late Egyptian cursive Demotic script is ultimately derived from pictograms, despite this being less obvious than in the case of hieroglyphs (which were still largely not pictograms, with some exceptions). And studying the history of scripts of any of these languages is informative. But the modern symbols used in Chinese and Japanese are not currently pictograms any more than they are in English, despite a nearly universal folk myth to the contrary with Westerners.
I assume you know all this, but most readers don't, and you weren't making this clear. -- DougMerritt
This has been a back-burner project of mine for years. I've been looking for a tool to do this over the course of those years this page is the closest thing I have ever found. I might have started earlier if I knew anyone was interested. Here are some of the ideas that I have culled over that time (not all viable, perhaps; consider this a brainstorming):
First of all, the general concept. Imagine trying to create an electrical circuit using just words. It would be obfuscated and ridiculous. The electrical schematic says it so much simpler. What we're looking for is the ability to create a dataflow diagram analogous to the ANDs, ORs, half adders, and so on. In an introductory computer course, we once had to create half adders, and full adders from half adders, the half adders being included from our previous work as just a box. Similar to how functions could be related to each other, passing each other values. With the half adders, we could specify a value for the input wires, and see the result we get out, or just leave it blank. It would be nice to specify values on the inputs of functions and see the data flow through. Perhaps even to save off assertions about those values and thereby implement a DesignByContract or UnitTesting sort of thing. It would be nice in the small, but consider VHDL in relation to your electric circuit example.
Second of all, it is conceivable that more flexibility may be possible with true visual programming than otherwise. The first time I ever cogitated on the idea of yielding and blocks was with my little code pictures. It took a little while to recognize that ruby yields and blocks were the same thing as I was visualizing, but once I did, it made me realize that this is truly something if it can help me come up with a (to me) new programming syntax that I later find out is a valid syntax newly implemented. Other possibilities include more flexible entry and exit into/from loops, realization and implementation that instance and type are more of a range than conventionally considered (the specific object usually called an instance just has all the holes filled in, an intermediate type/instance having some holes filled in), arrays are to variables what loops are to functions, and so on...
Third of all, though looking at individual hydrogen bonds may give a complex layout, zoomed out you might just see the flat surface of a pond. It would be nice to be able to zoom while programming. It would be nice to be able to descend into a function seamlessly (yes I realize that some RAD tools do this). Similarly, all those UML diagrams and structure diagrams that people say are just there to help the code but not actually get included, those I say should be visible when viewing the zoomed out structure, at least to some extent.
If there are some people willing to brainstorm with me and have some diagram swaps, I'd love to start or be in an emerging open source project with some of these ideas to make a GraphicalProgrammingLanguage.
half adders.... see figure 15.6 in http://www.jpaulmorrison.com/fbp/method.htm. I think you touched on the key point when you talked about building a dataflow diagram. IMHO visual programming adds very little when doing procedural code, although I have been known to jot down a block diagram when confronted with some particularly tangled logic. I wouldn't want to enter logic graphically, though - StructuredProgramming is much better!
For some examples of such diagrams see http://www.jpaulmorrison.com/cgi-bin/wiki.pl?DrawFlow. -- PaulMorrison
Thank you, Paul, for making me a page. It should have occurred to me, but didn't.
It is true that conventional programming does do a fairly good job at procedural programming. While it may not seem obvious how conditionals and looping structures would fit into a dataflow type of programming, I have devised some syntactic (if I can call them that) conventions that work quite well, I think. One of the big payoffs, though, in visual programming comes when you have many independent operations occurring to a certain object, usually to prepare it to be passed to another function. In this situation, you have an object hub, and the untangling that happens in helping you see the purpose of the code is huge.
I have since found some projects that are attempting GraphicalProgramming in a way close to what I envision (and I'll probably end up joining one of these projects:
Moved from VisualLanguage:
GraphicalProgrammingLanguages are languages and/or environments where the constructs for specifying programs are made up of visual elements (e.g. pictorial, iconic, line drawings, graphs).
Advantages:
One nice thing about GraphicalProgrammingLanguages is what makes a language good - it is a readily understandable representation of the problem and/or solution.
Some people say that GraphicalProgramming will succeed because SoftwareHasShape.
Disadvantages:
One bad thing about GraphicalProgrammingLanguages is that they do not integrate well with the programmer toolsets we have built up over the years. How do you search for patterns in a graphical language "text"? What is a good coding standard for readability and easy reviewing? I like the concept [...], but I wonder how those interconnected webs of links can be reliably archived let alone communicated to someone through printed documentation (maybe that's not important in the future, but for now it sure is).
-- JeffMantei 2000-11-28
Some people say that GraphicalProgramming is doomed to failure because SoftwareHasNoShape.
Would be nice to have software similar to TheIncredibleMachine that let you fit graphical objects together to produce a program, even just a prototype. (Borland) ObjectVision? was kind of like this in the early 90s but still required a lot of parameter definition/coding. TheIncredibleObjectEnvironment? could have predefined objects (files, string, vector etc as usual) and the ability to create your own. There has been talk of executable UML (see http://www.executableumlbook.com/overview.htm) perhaps that will be close but I'm thinking fun icons. Some Computer Telephony IDEs are like this but are expensive. With svg or similar it might be possible to make an OpenSource visual object development environment hostable over the web. It would run directly not just do CodeGeneration. People might then get a better feel for what object oriented development (and thinking) should be. If they could map their MentalModels with the tool they would be able to see it work right away.
Executable UML is a reality. There are domains where UML models generate all of the code. See http://www.projtech.com/. GraphicalProgramming has come and gone as the "next big thing" several times. Each cycle of activity ends with people realizing that GraphicalProgramming is more difficult than textual programming. Logic, reason, instruction, etc. are by-products of language. They can be represented graphically, but only with difficulty. My current employer uses Informatica to define data transformations. It provides only a visual editor for these. Developers create boxes to represent each manipulation of data and draw lines between the boxes to indicate sequence. Moderately complex transformations explode into huge rats nests of little lines. The same behavior could be expressed more compactly and clearly with text.
Wasn't there an example in SqueakSmalltalk where you drop the steering wheel on the racecar's heading slot, and that ended up making the user able to turn the car via the steering wheel?
[SqueakSmalltalk looks interesting - will have a closer look. VisualProgramming is alive and well if you look at VisualStudio, VisualWorks etc the problem is mostly the user interface is visual, the business objects and back-end processes (files, networking etc) aren't given the same treatment.]
But VisualProgramming is not the same thing as GraphicalProgramming.
[Perhaps now that object technology and ideas have evolved it is time to have another go. If the tool is designed well, it should reduce difficulty for the user (in this case the programmer). Language is a by-product of interacting with a visual world by transitivity; logic, reason, etc., should also be definable in 2D by those who prefer to create in that mode. Not necessarily as a "big trend" but as an option. If Informatica does certain things badly that could be an opportunity for them to fix things for the next version. Is their tool even based on an OO approach? As well that is why I think an OpenSource tool would be better - many developers would contribute to it and it would only exist if it was useful or interesting, and evolve in response to the communities needs.]
It isn't that Informatica does anything poorly. The problem lies in trying to represent any software solution in just 2 dimensions. It's trivial to say that A references B in text because our brains can create an n-dimensional model where the line from A to B doesn't intersect the lines between all the other components. To do the same in 2 dimensions (or even 3) invokes complex routing and visual clutter. There's a strong instinct that tells us visualizing software graphically will help us manipulate it, but I believe that instinct only applies to very simple software, or small slices of complex software. If you reduce the number of elements in each view to a manageable level, you're not far removed from traditional source code.
So you believe UmlDoesntWork?
UML works for some things. Some folks love BridgePoint? (http://www.projtech.com/). Most folks agree that UML is good for communicating some kinds of ideas without programming language specifics. I wrote the Java and SAP OO/ABAP round trip engineering components for a major commercial UML modeling tool. We used UML informally on whiteboards, and we used MOF to define UML in the tool, but we never maintained UML models of our day to day work. It was too much trouble for not enough benefit.
I'm not saying traditional source code should be abandoned, or this method should be used for BigIron projects. However many people reading this page will never touch SAP, or have access to commercial UML tools. A simple, documented, easy to install opensource visual tool embodying executable OO concepts might help many decide that OoFitsOurMentalAbilities instead of the opposite and that ProgrammingIsFun instead of ProgrammingIsNotFun.
Perhaps. My experience has led me to believe that GraphicalProgramming is not as much fun as textual programming, but maybe something will change my mind.
And you are probably right however the fact that you have created such systems makes it seem overkill to you :-) If you yourself use such a tool to drag an icon you know a MouseDown?() event has fired (but not yet a MouseUp?() ), that the OnPaint?() event is being handled, what the code is doing to create tracking rectangles etc. But for someone trying to figure out what an InnerClass is for the first time if they can just right click an existing class icon and find they can create a class within a class intuitively they know what has happened even if they can not yet put it into words. They can then dig into real code and learn more details. Certainly they should move onto text programming but it might mean the difference between becoming a real programmer (and being able to work with ABAP) or choosing something else like social science - not that there is anything wrong with that. Or maybe move from social science into programming.
I don't think you understand the nature of the problems encountered in GraphicalProgrammingLanguages. And I know you don't understand that ABAP is not for real programmers. :-) Read some of the criticisms earlier on this page. A GUI may help understand what an inner class is, but it makes it harder to use. You have to connect data and behavior with lots of little lines. With just a few simple interactions those lines intersect each other, making it very hard to grasp what is connected to what. This is one reason circuit board and chip designers are moving away from visual representations of their product toward textual languages that generate them.
I feel like I'm raining on your parade, and if you're the one person who might figure out a good way to do GraphicalProgramming I shouldn't be discouraging you. In the mid 80s ByteMagazine did an issue devoted to GraphicalProgrammingLanguages and I was convinced that was the next big thing. It seems like every few years a new group of people become convinced of the same thing. Investigate them, but know that there are good reasons they haven't been successful. Our instincts tell us that images should convey more information and be easier to work with than text, but our instincts are often wrong.
[As EwDijkstra puts it: "A formula is worth a thousand pictures."]
Or as BenoitMandelbrot? put it, "z := z^2+c". Actually, the formula was discovered much earlier, but Benoit was the first to use a computer to visualize the MandelbrotSet.
It seems to me that my WEB site http://executablepictograms.com – the concept for the theory in programming which I call Calculus χπ or Calculus “Executable Pictograms” can cover some of yours gentleman concerns. I’d like those of you who still interested in this topic GraphicalProgrammingLanguages to contact me or tell in this article what you think and provide a contact link to me.
-- Regimantas Streimikis 10/14/2013
See also FutureOfProgrammingLanguages.
EditText of this page
(last edited October 14, 2013)
or FindPage with title or text search