Librarian of Alexandria

If I write my own shell—which I may very well do at some point—there's a particular process model I'd like to embed in it. To wit: in UNIX right now, each program in a pipeline has a single input stream and two output streams, with files/sockets/&c for other kinds of communication.

A pipeline of functions foo | bar | baz looks kind of like this

keyboard -> stdin\   /stdout -> stdin\   /stdout -> stdin\   /stdout —+
                  foo                 bar                 baz           |
                     \stderr -+          \stderr -+          \stderr -+ |
                              |                   |                   | |
                              v                   v                   v v
                            [..................terminal...................]

Which works pretty well. You can do some pretty nice things with redirecting stderr to here and stdin from there and so forth, and it enables some nice terse shell invocations.

I'd like that basic system to be preserved, but with the ability to easily create other named streams. For example, imagine a hypothetical version of wc which still outputs the relevant data to stdout, but also has three other streams with these names:

                 / newlines
                 | words
stdin -> wc-s -> | bytes
                 | stdout
                 \ stderr

You can always see the normal output of wc on stdout:

gdsh$ wc-s *
       2       3       6 this.txt
      10      20      30 that.c
     100    1000   10000 whatever.py

But you could also extract an individual stream from that invocation using special redirection operators:

gdsh$ wc-s * stdout>/dev/null bytes>&stdout
3
20
1000

We could also have multiple input channels. I imagine an fmt command which can interpolate named streams, e.g.

gdsh$ printf "1\n2 3\n" | wc-s | fmt "bytes: {bytes}\n words: {words}\n nl: {newlines}\n"
bytes: 6
words: 3
newlines: 2

We can then have a handful of other utilities and built-in shell operators for manipulating these other streams:

  # the `select` command takes a stream name and outputs it
gdsh$ wc-s * | select words
3
20
1000
  # here we redirect stdout to the stream X and pass it to fmt
gdsh$ cat this.txt stdout>&X | fmt "this is {X}\n"
this is 1
2 3
  # the same, using file redirection operators
gdsh$ fmt "this is {X}\n" X<this.txt
this is 1
2 3
  # the same, using a shorthand for setting up a stream by taking
  # the stdout from some command
gdsh$ !X='cat this.txt' fmt "this is {X}\n"
this is 1
2 3
  # the same, using a shorthand for setting up a stream by just
  # reading and outputting a file
gdsh$ @X=this.txt fmt "this is {X}\n"
this is 1
2 3
  # using a shorthand for filling in a stream with a string directly
gdsh$ ^Y=recidivism fmt "Y is {Y}\n"
Y is recidivism
  # redirecting each output stream to a different file
gdsh$ wc-s * words>words.txt bytes>bytes.txt newlines>newlines.txt
  # using a SmallTalk-like quoting mechanism to apply different shell
  # commands to different streams
gdsh$ wc -s * | split words=[sort >sorted-word-count.txt] bytes=[uniq >uniq-bytes.txt]

This could also enable new idioms for programs and utilities. For example, verbose output, rather than being controlled by a flag to the program, could be always output to a (possibly unused) stream called verbose, so the verbose output could be seen by redirecting the verbose stream (or by logging the verbose output while only seeing the typical stderr messages):

  # here we only see stderr
gdsh$ myprog
myprog: config file not found
  # here we ignore stderr and see only the verbose output
gdsh$ myprog stderr>/dev/null verbose>&stderr
Setting up context
Looking in user dir... NOT FOUND
Looking in global dir... NOT FOUND
myprog: file not found
Tearing down context
Completed
  # here we see stderr but logg the verbose output
gdsh$ myprof verbose>errmsgs
myporog: config file not found

Or maybe you could have human-readable error messages on stderr and machine-readable error messages on jsonerr:

  # here is a human-readable error message
gdsh$ thatprog
ERROR: no filename given
  # here is a machine-readable error message
gdsh$ thatprog stderr>/dev/null jsonerr>stderr
{"error-type":"fatal","error-code":30,"error-msg":"no filename given"}

Or you could have a program which takes in data on one stream and commands on another:

  # someprog takes in raw data on the stream DATA, and commands
  # on the stream CMDS. Here we take the data from a local file
  # and accept commands from the network:
gdsh$ @DATA=file.dat !CMDS='nc -l 8000' someprog
  # ...and here we have a set of commands we run through locally
  # while taking data from the network:
gdsh$ !DATA='nc -l 8001' @CMDS=cmds.txt someprog

There are other considerations I've glossed over here, but here are a few notes, advantages, and interactions:

• I glossed over the distinction between input/output streams. In practice, the shell has no trouble disambiguating the two, but a given program may wish to consider the distinction between words.in and words.out; to this end, we could rename the existing streams std.in and std.out and err.out (it being an error to read from err.in in most cases.¹)

• This is obviously not POSIX-compliant, but could be made to work with the existing UNIX process model by e.g. having a standard environment variable for stream-aware programs to look at which maps stream names to file descriptors. That way, programs which don't expect these special streams still use fds 0, 1, and 2 as expected, while programs that do handle these can read STREAM_DESC to find out which 'streams' correspond to which file descriptors. In that case, you can almost use these commands with an existing shell by doing something like

  sh$ echo foo >&3 | STREAM_DESC='foo.in:3' fmt "foo is {foo}\n"
  

  where STREAM_DESC takes the form of streamname:fd pairs separated by spaces.

• If we do use an existing UNIX system to write this, then we also should integrate libraries for this, and the API for it is unknown. Presumably a C interface would have int get_stream(char* stream_name) that could return -1 on failure. get_stream would look through the STREAM_DESC environment variable to find the relevant stream and return the fd mentioned there, otherwise failing. This does mean that you have to create your streams before you use them, which I think is reasonable.

• This would interact really interestingly with a semi-graphical shell² that could visualize the stream relationships between commands as well as a shell with a higher-level understanding of data types.³

So those are some ideas that have been drifting around in my head for a while. No idea if I'll ever implement any of them, or if they'd even be worth implementing, but I might get around to it at some point. We'll see.