Parallel Programming Discussion

I think there is an issue about boundaries as to what is ParallelProgramming. I had tried to be clear about it when I started ParallelProgramming, but it seems that I have not been clear and that shows up as a problem worth discussing.

There are pages which link from DistributedComputing which discuss issues of the patterns of programming. They include quite a lot of topics as can be seen from the fact that it has 56 back links.

I intended ParallelProgramming to be about taking a serial program solving some numerical problem and turning it into a parallel program which will run on several computers in a closecoupled way. Maybe it is wrongly named, in which case it could be renamed something more specific. I now suggest ProgrammingForParallelComputing. -- JohnFletcher

I think of DistributedComputing as distributing a task among separate computers where it is expensive to share resources between nodes. I think of ParallelProgramming as exploiting fine-grain parallelism in a problem, where sharing resources is easier. Examples: MultiThreading? to take advantage of SymmetricMultiprocessing, and decomposing algorithms to use SIMD (SingleInstructionMultipleData?) to take advantage of vector units such as AltiVec or CellProcessors. -- IanOsgood

Here are some definitions from WikiPedia.

On the page http://en.wikipedia.org/wiki/Parallel_computing.

Parallel computing is the simultaneous execution of the same task (split up and specially adapted) on multiple processors in order to obtain results faster. The idea is based on the fact that the process of solving a problem usually can be divided into smaller tasks, which may be carried out simultaneously with some coordination.

A parallel programming model is a set of software technologies to express parallel algorithms and match applications with the underlying parallel systems. It encloses the areas of applications, languages, compilers, libraries, communication systems, and parallel I/O. People have to choose a proper parallel programming model or a form of mixture of them to develop their parallel applications on a particular platform.

The items listed include ParallelVirtualMachine and MessagePassingInterface.

On the page http://en.wikipedia.org/wiki/Parallel_programming

Parallel programming is a computer programming technique that provides for the execution of operations in parallel, either within a single computer, or across a number of systems. In the latter case, the term distributed computing is used.

On the basis of the above I think that what I am discussing can be described as ProgrammingForParallelComputing which has to use a ParallelProgrammingModel. That has the virtue of being consistent with the WikiPedia usage. -- JohnFletcher

Category and Concurrency

There are two categories CategoryConcurrency and CategoryConcurrencyPatterns.

There are also related pages called SynchronizationStrategies and ParalleliZation. There is some discussion about how these topics are related.

Concurrency is when two tasks or threads are active at the same time.
Synchronization is when the two tasks need to interact.

There is an EditHint on CategoryConcurrencyPatterns that SynchronizationStrategies is the same thing and that they should be merged.

I think that ProgrammingForParallelComputing is a distinct area from the above which could have a category of its own but does not yet. I would welcome suggestions or contrary views. -- JohnFletcher

There is also the interesting book PatternsForParallelProgramming.

CategoryDiscussion