std/stream/bytes

A chunk of bytes. An alias, not a type of its own: it is a List<UInt8> and every list operation works on it. The name exists because "bytes" is what the signatures mean, and List<UInt8> in twenty places reads like arithmetic.

A chunk belongs to whoever produced it. A source hands one out and does not keep it, so a reader may hold on to it - there is no borrowed buffer that has to be copied before the next await, because a list is a value.

• Sink.addText - writing text into a byte sink.

Bytes that are not UTF-8, with the offset of the byte that broke it. A String is never anything else.

Bytes to lines, split at \n, with a \r before it dropped so that CRLF files read the same as LF files. The line break itself is not part of the line, and a last line without one is still a line.

Fallible, so the items are Results; source.through(lines()).checked() turns them into the stream's own failure. The same stage works on a plain Iterate<Bytes>, with no Source and no Task involved.

const chunks = ["one\ntw".bytes().toList(), "o\nthree".bytes().toList()]
for line in chunks.through(lines()) {
  print line
}

• Utf8Error for a line whose bytes are not UTF-8, with the offset counted from the start of that line. One broken line fails once: every line around it arrives as it is, because each one is decoded on its own.

Bytes to text, one chunk at a time and cut at character borders: what arrives is whatever was complete, and a character split across two chunks is held back until the rest is there.

const chunks: List<Bytes> = [[0xE2, 0x82], [0xAC]]
print chunks.through(decodedText()).toList()

• Utf8Error for bytes that are not UTF-8, with the offset counted from the start of the stream: every byte the stage has been handed since it started counts, not only the bytes of the chunk the failure was found in. A chunk border is nobody's choice, so an offset inside a chunk would name a byte the caller cannot point at.
• At the end of the stream an incomplete sequence is such a failure too, and its offset is the byte that sequence starts on.

• A sequence that is still incomplete when the stream ends is reported as invalid UTF-8, not delivered as a partial character: there is no more input coming to complete it.

A whole chunk of bytes as text - the short form for everything that has all of its bytes already (a file that was read in one piece, a body below its limit). Bytes that are not UTF-8 are a failure, never a replacement character.

const bytes: Bytes = [104, 105]
print textOf(bytes)

• Utf8Error with the offset counted from the start of bytes: the byte that cannot be decoded, or the byte an incomplete sequence at the end starts on. Nothing is held back here - there is no next chunk to complete it with.

Text to bytes. UTF-8 needs no decision here: a String already is UTF-8.

const words = ["hi", "there"]
print words.through(encodedText()).toList()

• Sink.addText - the same step for one text, written straight into a byte sink.

Writing text into a sink of bytes. An extension of the instantiated trait, so it is there for every byte sink - a file, standard output, a socket, the input of a child process - and nowhere else.