Streams As Sequences

Any set of values delimited by iterators is a representation of the sequence concept. For std::vector that pair is

 vector::begin() and vector::end();
for an istream I containing T objects that pair is
 std::istream_iterator<T>( I ) and
 std::istream_iterator<T>();
and for an ostream O that will store T's the pair is
 std::ostream_iterator<T>( O )
and the last iterator written to.


How about a coroutine or generator (like python or CsharpLanguage 2.0).

yield 3; yield 4; yield 5; would return a stream.

In HaskellLanguage, a stream could be a tree rather than sequential list.

Is StreamProcessing? a good idea? I don't know what to call this pattern, so please refactor if you have a good name for it.

Its easy to write a function to:

   - read a text document from a stream (i.e. from a file or remote web server)
   - parse it (i.e. xml)
   - transform it
   - do something else with the results

Is it a pattern or antipattern to do the above as a series of stream processing steps, so that execution goes between reading, parsing, transforming, etc repeatedly as the output is generated, or better to simple read the whole document, parse it, transform it,l etc.

That depends how much data you have, and what kind of transforming you need to do. If you have more data than will easily fit in memory, and your transformation does not require random access to the data, then it is a pattern to perform the work as a series of stream processing steps. However, if your transformation includes a random access step (e.g. sorting) then it is often easier to read the data into memory up front. If you have more data than memory, and you also need random access, you usually need to get more sophisticated.


EditText of this page (last edited July 3, 2004) or FindPage with title or text search