Programming For Parallel Computing

ProgrammingForParallelComputing proposed new name for much of what I had put on ParallelProgramming. -- JohnFletcher

Programming for Parallel Computing is programming to run on several processors at the same time. This is something which has been possible for many years. Now it is a real possibility but there are great barriers for programmers wanting to take advantage but having to grasp new ideas to be sure that the results are reliable in the new situation. There are several libraries which do the basic InterProcessCommunication and also a lot of higher-level code for particular problems such as LinearAlgebra. What many people need is a roadmap of how to get something running. -- JohnFletcher

If a program is to run in parallel, there must be at least one task running on each processor, and they must be able to communicate through InterProcessCommunication.

Programming for this involves a choice of ParallelProgrammingModel.

Discussion of ParallelProgrammingModel moved to that page. -- JohnFletcher

Not only is it "a real possibility", but it will increasingly become a necessity if one wishes to fully leverage the capabilities of multithreaded and multicore CPUs. -- MikeSmith

Q. Is there an accepted delineation (or term) between parallel computing on single machine systems (i.e. where one presumably has command over the [synchronous] execution path) vs. p2p parallel computing done over several machines where there is not synchronous, controlled execution (like the SETI-at-home project)? --MarkJanssen

A. The former is typically known as multi-threaded programming or multiprocessing, the latter is typically called distributed computing or parallel distributed computing or grid-based computing.

A2. Most 'single machine systems' these days have at least two cores in the CPU which are sharing the main memory but not faster cache memory directly attached to each core. This means that compute intensive tasks are most efficiently carried out by software which is able to adapt to the structures available. There is even software which is able to find out what is the structure of the CPUs and their caches and provide that information in ways which can be used by higher level software.

CategoryProgrammingLanguage