Pipes
Pipes are uni-directional conduits for reliably transmitting data of a particular type.
Pipes can transmit any data type that RIDDL can specify. There is only one data type that flows in a pipe. The transmission type is often used with an alternation of messages such as the commands and queries that an entity might receive.
Pipes have two ends, from and to, and each must be connected to a
processor. Data elements of the transmission
type flow from from to end. Such end connections may be closed or
open. A closed end is specified in the definition of the pipe by
connecting it to an inlet (for to ends) or
outlet (for from ends).
An open end is not specified in the pipe definition. Instead, it is
implied by use of thepublish and subscribe
statementss. This permits multiple
publishers and multiple subscribers.
Pipes may play a large role in the resiliency of a reactive system, so we permit a variety of options to be specified on them. These options are only intended as advice to the translators converting the pipe into something else. For example, a pipe may or may not need to be persistent. If a pipe has the burden of persistence removed, it is likely much more performant because the latency of storage is not involved.
The messages flowing through the pipe are persisted to stable, durable storage, so they cannot be lost even in the event of system failure or shutdown. This arranges for a kind of bulkhead in the system that retains published data despite failures on either end of the pipe
With this option, pipes support the notion of being commitable. This means the consuming processors of a pipe’s data may commit to the pipe that they have completed their work on one or more data items. The pipe then guarantees that it will never transmit those data items to that processor again. This is helpful when the processor is starting up to know where it left off from its previous incarnation.
For scale purposes, a pipe must be able to partition the data by some data
value that is in each data item (a key) and assign the consumption of the
data to corresponding members of a consumer group. This permits multiple
instances of a consuming processor to handle the data in parallel. The n
value is the minimum recommended number of partitions which defaults to 5
if not specified
By default, pipes provide the guarantee that they will deliver each data item
at least once. The implementation must then arrange for data items to be
idempotent so that the effect of running the event two or more times is the
same as running it once. To counteract this assumption a pipe can be use the
lossy option which reduces the guarantee to merely best reasonable effort,
which could mean loss of data. This may increase throughput and lower overhead
and is warranted in situations where data loss is not catastrophic to the
system. For example, some IoT systems permit this kind of data loss because
the next transmission is imminent.
Attached to the ends of pipes are producers and consumers. These are processors of data and may originate, terminate or flow data through them, connecting two pipes together. Producers provide the data, consumers consume the data. Sometimes we call producers sources because they originate the data. Sometimes we call consumers sinks because they terminate the data.
graph LR;
Producers --> P{{Pipe}} --> Consumers
Source --> P1{{Pipe 1}} --> Flow --> P2{{Pipe 2}} --> Sink
Pipes may have multiple publishers (writers of data to the pipe) and multiple consumers (readers of data from the pipe). In fact, because of the partitioned consumption principle, there can be multiple groups of consumers, each group getting each data item from the pipe.
Not ImplementedThis feature is not implemented as of 0.16.1
When a pipe has multiple consumers, they are organized into subscriptions. Each subscription gets every datum the pipe carries. Consumers attach to a subscription and there is generally one consumer per partition of the subscription. Sometimes subscriptions are known as consumer groups as is the case for Kafka.
Nothing