Reference

std/encoding/lib

std/encoding/src/lib.trb

Encoding and decoding: what other languages need reflection for (serialization, config mapping, database rows, schemas, --help texts).

const json = Json()
const text = json.encode user
const back = json.decode<User>(text)?

A value is its constructor call. The compiler knows every type's constructor and offers it in three forms: written (Encode), read (Decode) and described without a value (Describe). A format (Json, a database driver, a command line parser) implements Encoder, Decoder or Describer and never sees a type. There is no tree in between: a value is written while the type describes itself, and nothing is lost on the way.

All three forms are generated for a type whose constructor is usable from outside, over exactly the constructor's parameters: a private field with a default is not one of them, so a cache stays out without an annotation. What the compiler generates is ordinary code over the vocabulary, and could be written by hand:

extend User with Encode {
  fn encode<Target: Encoder>(var target: Target) {
    target.record "app/User"
    target.field "name"
    name.encode target
    target.field "tags"
    tags.encode target
    target.finish()
  }
}

The three traits take their target as a bound, not as a trait type: a format hands its own encoder over as a var path and reads the result out of it afterwards, and the derived code is a direct call chain with no table. EncodedValue is what a program holds where it has left the type behind, and Structure is a type's description as a value. docs/design/ENCODING.md is the design.

trait Encode

trait Encode

A value writes itself into any format.

Generated for a type whose constructor is usable from outside and whose passable fields are all Encode, and for a capsule through its conversion pair. The target is a bound, so the call is monomorphized per format.

Examples

type Point {
  x: Int
  y: Int
}

print Json().encode(Point(1, 2))

Related

  • Decode - the way back, derived under the same condition.
  • Encoder - what a format implements.

fn encode

fn encode<Target: Encoder>(var target: Target)

Not documented.

trait Decode

trait Decode

A value reads itself back. Derived under the same condition as Encode, so what is written can always be read.

A field with a default may be missing from the input, and the default is only evaluated then.

Related

  • DecodeError - what a failure says, with the field chain it happened in.

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Self, DecodeError>

Not documented.

trait Describe

trait Describe

A type describes its structure without a value: what schema-driven formats need (a DDL statement, a .proto file, a JSON Schema, a --help text). It is the same constructor again, with every field's doc comment and whether the field has a default.

Related

fn describe

static fn describe<Target: Describer>(var target: Target)

Not documented.

type DecodeError

type DecodeError with Show, Error

What went wrong reading a value, with the whole chain of fields and positions it went wrong in: items[2].price.currency: a value is needed. A field is written after a ., a position of a sequence or a tuple in brackets, and an entry of a map after a . with its key.

Examples

const problem = DecodeError("a value is needed").inside("currency").inside("price").at(2).inside("items")
print problem

field message

message: String

What was wrong, without the place.

field path

path: List<String> = []

The fields and positions from the outermost value down to where it went wrong.

fn inside

fn inside(segment: String): DecodeError

error.inside("email"): an error collects its path on the way out, outermost segment first.

fn at

fn at(position: Int): DecodeError

error.at(2): the position of a sequence or a tuple the error happened in, shown as [2].

fn show

fn show(): String

Not documented.

trait Encoder

trait Encoder

The writing half of a format. The scalar methods are named after the types of the language, so a format learns no second vocabulary; narrow numbers are widened on the way in and checked on the way back.

sequence, map, record and variant open, and finish closes the innermost of them. A record's field is announced with field and its value is written right after. A record that announces no field at all is a wrapper around the one value it holds, and a text format writes that value without anything around it.

Related

fn nothing

var fn nothing()

An absent value: None. A field written as nothing may be left out, and its default applies on the way back.

fn bool

var fn bool(value: Bool)

Not documented.

fn int

var fn int(value: Int64)

Not documented.

fn unsigned

var fn unsigned(value: UInt64)

Not documented.

fn float

var fn float(value: Float64)

Not documented.

fn decimal

var fn decimal(value: Decimal)

Not documented.

fn string

var fn string(value: String)

Not documented.

fn bytes

var fn bytes(value: List<UInt8>)

Not documented.

fn sequence

var fn sequence(length: Int?)

length is known for a collection and unknown for a pipeline: a binary format writes it in front.

fn map

var fn map(length: Int?)

A map of length entries, each written as its key and then its value.

fn record

var fn record(typeName: String)

typeName is the qualified name of the declaration ("app/orders/Price"): the key a format finds a mapping by.

fn variant

var fn variant(typeName: String, name: String)

One case of a type with cases. Its fields follow as a record's do.

fn field

var fn field(name: String)

The name of the field whose value comes next.

fn finish

var fn finish()

Closes the innermost sequence, map, record or variant.

fn writesPrivateFields

fn writesPrivateFields(): Bool

Whether a derived encode writes the fields that are no parameter of the constructor from outside, too: a private field with a default. A format writes a type's data and answers false; a copy of the whole value answers true (Values(privateFields: true)), such as the value a native binary hands a script in its VM (std/sandbox, docs/design/SCRIPTS.md section 5). A field whose type is not Encode is left out either way.

trait Decoder

trait Decoder

The reading half of a format. Reading is driven by the type: it asks for what it expects, and gets a DecodeError when something else is there. A format with unordered fields buffers one record at a time; a sequence streams.

Pitfalls

hasNext both tests and positions: it answers true while one more item of the open sequence (or one more key or value of the open map) follows, and the next read takes it.

fn nothing

var fn nothing(): Bool

Consumes a "nothing" if one is there. This is how Option decides between None and Some.

fn bool

var fn bool(): Result<Bool, DecodeError>

Not documented.

fn int

var fn int(): Result<Int64, DecodeError>

Not documented.

fn unsigned

var fn unsigned(): Result<UInt64, DecodeError>

Not documented.

fn float

var fn float(): Result<Float64, DecodeError>

Not documented.

fn decimal

var fn decimal(): Result<Decimal, DecodeError>

Not documented.

fn string

var fn string(): Result<String, DecodeError>

Not documented.

fn bytes

var fn bytes(): Result<List<UInt8>, DecodeError>

Not documented.

fn sequence

var fn sequence(): Result<Int?, DecodeError>

Opens a sequence and answers its length where the format writes one in front.

fn map

var fn map(): Result<Int?, DecodeError>

Opens a map. Its entries are read as a key, then a value, while hasNext answers true.

fn hasNext

var fn hasNext(): Result<Bool, DecodeError>

true while one more item of the open sequence or one more key or value of the open map follows.

fn record

var fn record(typeName: String): Result<Void, DecodeError>

Opens a record. A wrapper's record holds one value and no field.

fn variant

var fn variant(typeName: String): Result<String, DecodeError>

Opens a variant and answers which case is there.

fn field

var fn field(name: String): Result<Bool, DecodeError>

Positions on the field name. false when the input has no such field: the field's default applies.

fn finish

var fn finish(): Result<Void, DecodeError>

Closes the innermost sequence, map, record or variant.

fn readsPrivateFields

fn readsPrivateFields(): Bool

Whether a derived decode reads the fields that are no parameter of the constructor from outside, too, where the input has them: the other half of Encoder.writesPrivateFields. A field the input does not have takes its default.

trait Describer

trait Describer

The describing half: the same method names as Encoder, without values. record and variant answer a Bool: false means "this type is already known here", which is what makes a type that contains itself terminate - the name has been handed over before the answer, so the target can refer to the type it declined.

Related

fn bool

var fn bool()

Not documented.

fn int

var fn int()

Not documented.

fn unsigned

var fn unsigned()

Not documented.

fn float

var fn float()

Not documented.

fn decimal

var fn decimal()

Not documented.

fn string

var fn string()

Not documented.

fn bytes

var fn bytes()

Not documented.

fn optional

var fn optional()

The description that follows is the one that may be absent. finish closes it.

fn sequence

var fn sequence()

A sequence: the one description that follows is its item's. finish closes it.

fn map

var fn map()

A map: the two descriptions that follow are its key's and its value's. finish closes it.

fn record

var fn record(typeName: String): Bool

Opens a record. false: this target knows the type already and does not want the body again.

fn variant

var fn variant(typeName: String): Bool

Opens a type with cases, as record does.

fn variantCase

var fn variantCase(name: String)

One case of the open variant. Its fields follow, finish closes it.

fn field

var fn field(description: FieldDescription)

The next description belongs to this field.

fn finish

var fn finish()

Closes the innermost open description.

trait Format

trait Format<Failure>

What a whole format is, once: bytes in both directions, and the two Stages that make it work on a stream.

Encode/Decode and Encoder/Decoder stay synchronous, and that is the point. A decoder that could wait would have to be written differently for every source, every derived implementation would have to change with it, and the whole standard library would be coloured by it. Streaming happens one level up, at the level it happens in practice: the element.

body.through(Json.items<User>()).checked()          // bytes in, one `User` at a time out, memory = one user
Source.from(users).through(Json.encoded<User>())    // and back

items is a resumable framer: it finds where one element ends in whatever bytes have arrived - across chunk borders - and decodes that element synchronously, through the ordinary Decode.

Failure is the type parameter because the language has no associated types: Json is a Format<JsonError>, and a function that works for any format says fn load<Chosen: Format<Problem>, Problem>(...). The members are static and use the format's default options; a format value with options has its own encode and decode.

fn encodeAll

static fn encodeAll<Value: Encode>(value: Value): Bytes

The whole value, as the bytes that go into a file or onto a wire.

fn decodeAll

static fn decodeAll<Value: Decode>(bytes: Bytes): Result<Value, Failure>

One whole value out of all of its bytes.

fn items

static fn items<Item: Decode>(): Stage<Bytes, Result<Item, Failure>>

Bytes to items. Fallible, so the items are Results - a Stage is synchronous and knows nothing about the failure of the stream around it; source.checked() is where they become one.

fn encoded

static fn encoded<Item: Encode>(): Stage<Item, Bytes>

Items to bytes: the other direction, and the one that never fails.

extend Bool with Encode, Decode, Describe

extend Bool with Encode, Decode, Describe

Bool is the vocabulary's own bool, in all three forms.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Bool, DecodeError>

fn describe

static fn describe<Target: Describer>(var target: Target)

extend Int64 with Encode, Decode, Describe

extend Int64 with Encode, Decode, Describe

Int64 is the vocabulary's own int, in all three forms.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Int64, DecodeError>

fn describe

static fn describe<Target: Describer>(var target: Target)

extend Int8 with Encode, Decode, Describe

extend Int8 with Encode, Decode, Describe

A narrow integer travels as an Int64 and is checked on the way back: a value that does not fit is an error.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Int8, DecodeError>

fn describe

static fn describe<Target: Describer>(var target: Target)

extend Int16 with Encode, Decode, Describe

extend Int16 with Encode, Decode, Describe

A narrow integer travels as an Int64 and is checked on the way back: a value that does not fit is an error.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Int16, DecodeError>

fn describe

static fn describe<Target: Describer>(var target: Target)

extend Int32 with Encode, Decode, Describe

extend Int32 with Encode, Decode, Describe

A narrow integer travels as an Int64 and is checked on the way back: a value that does not fit is an error.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Int32, DecodeError>

fn describe

static fn describe<Target: Describer>(var target: Target)

extend UInt8 with Encode, Decode, Describe

extend UInt8 with Encode, Decode, Describe

A narrow unsigned integer travels as an Int64, which holds every one of its values, and is checked on the way back.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<UInt8, DecodeError>

fn describe

static fn describe<Target: Describer>(var target: Target)

extend UInt16 with Encode, Decode, Describe

extend UInt16 with Encode, Decode, Describe

A narrow unsigned integer travels as an Int64, which holds every one of its values, and is checked on the way back.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<UInt16, DecodeError>

fn describe

static fn describe<Target: Describer>(var target: Target)

extend UInt32 with Encode, Decode, Describe

extend UInt32 with Encode, Decode, Describe

A narrow unsigned integer travels as an Int64, which holds every one of its values, and is checked on the way back.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<UInt32, DecodeError>

fn describe

static fn describe<Target: Describer>(var target: Target)

extend UInt64 with Encode, Decode, Describe

extend UInt64 with Encode, Decode, Describe

UInt64 is the vocabulary's own unsigned: its upper half does not fit into an Int64.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<UInt64, DecodeError>

fn describe

static fn describe<Target: Describer>(var target: Target)

extend Float32 with Encode, Describe

extend Float32 with Encode, Describe

A Float32 travels as a Float64. It has no Decode: nothing narrows a Float64 back into a Float32 yet.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn describe

static fn describe<Target: Describer>(var target: Target)

extend Float64 with Encode, Decode, Describe

extend Float64 with Encode, Decode, Describe

Float64 is the vocabulary's own float, in all three forms.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Float64, DecodeError>

fn describe

static fn describe<Target: Describer>(var target: Target)

extend Decimal with Encode, Decode, Describe

extend Decimal with Encode, Decode, Describe

Decimal is the vocabulary's own decimal: an exact number stays exact in a format that can say so.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Decimal, DecodeError>

fn describe

static fn describe<Target: Describer>(var target: Target)

extend Char with Encode, Decode, Describe

extend Char with Encode, Decode, Describe

A Char travels as a text of one character: the formats of the language have no character of their own.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Char, DecodeError>

fn describe

static fn describe<Target: Describer>(var target: Target)

extend String with Encode, Decode, Describe

extend String with Encode, Decode, Describe

String is the vocabulary's own string, in all three forms.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<String, DecodeError>

fn describe

static fn describe<Target: Describer>(var target: Target)

extend Option<Value> with Encode

extend<Value: Encode> Option<Value> with Encode

Some is its value, and None is nothing() - which a field may leave out, so its default applies.

fn encode

fn encode<Target: Encoder>(var target: Target)

extend Option<Value> with Decode

extend<Value: Decode> Option<Value> with Decode

A "nothing" is None, and anything else is the value's own decode.

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Value?, DecodeError>

extend Option<Value> with Describe

extend<Value: Describe> Option<Value> with Describe

An optional value describes as optional() around the value's own structure.

fn describe

static fn describe<Target: Describer>(var target: Target)

extend List<Item> with Encode

extend<Item: Encode> List<Item> with Encode

A list is a sequence: its length in front, then every item with its own encode.

fn encode

fn encode<Target: Encoder>(var target: Target)

extend List<Item> with Decode

extend<Item: Decode> List<Item> with Decode

A sequence, item by item, each through the item's own decode. An error names the position it happened at.

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<List<Item>, DecodeError>

extend List<Item> with Describe

extend<Item: Describe> List<Item> with Describe

A list describes as a sequence around its item's structure: the shape is one item, repeated.

fn describe

static fn describe<Target: Describer>(var target: Target)

extend ArrayList<Item> with Decode

extend<Item: Decode> ArrayList<Item> with Decode

An ArrayList reads back as the list it is written as - a sequence, item by item - so a field of one does not keep the type around it from having a derived decode (Chunk of the bytecode, whose code is one). It writes through the encode of List.

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<ArrayList<Item>, DecodeError>

extend ArrayList<Item> with Describe

extend<Item: Describe> ArrayList<Item> with Describe

An ArrayList describes as the sequence of List.

fn describe

static fn describe<Target: Describer>(var target: Target)

extend Set<Item> with Encode

extend<Item: Encode> Set<Item> with Encode

A set is a sequence of its items.

fn encode

fn encode<Target: Encoder>(var target: Target)

extend Set<Item> with Decode

extend<Item: Decode & Hash> Set<Item> with Decode

A sequence read into a set: an item that is there twice is there once.

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Set<Item>, DecodeError>

extend Set<Item> with Describe

extend<Item: Describe> Set<Item> with Describe

A set describes as a sequence of its item.

fn describe

static fn describe<Target: Describer>(var target: Target)

extend Map<Key, Value> with Encode

extend<Key: Encode, Value: Encode> Map<Key, Value> with Encode

A map is a map: its length in front, then every entry as its key and its value.

fn encode

fn encode<Target: Encoder>(var target: Target)

extend Map<Key, Value> with Decode

extend<Key: Decode & Hash & Show, Value: Decode> Map<Key, Value> with Decode

A map, entry by entry: a key, then its value. An error in a value names the key it belongs to.

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Map<Key, Value>, DecodeError>

extend Map<Key, Value> with Describe

extend<Key: Describe, Value: Describe> Map<Key, Value> with Describe

A map describes as a map around its key's and its value's structure.

fn describe

static fn describe<Target: Describer>(var target: Target)