Vector#

Vector is lightweight and ultra-fast tool for building observability pipelines. In the Aspects project, Vector can optionally be used as a replacement for Ralph to capture xAPI learner statements in the ClickHouse database, and/or as a way to store raw tracking log statements. It can be used as a general purpose log collector and forwarder.

Vector Components#

Vector consists of the following components:

  • Sources - Collects data from a source and sends it to Vector.

  • Transforms - Modifies events as they pass through Vector.

  • Sinks - Sends events to a destination.

Vector can be deployed in two roles:

  • Agent - Collects, transforms, and sends data to a destination.

  • Aggregator - Receives data from Vector agents and sends it to a destination.

We use the role Agent to collect tracking logs and xAPI events from the lms and send it to the Clickhouse sink.

For instance, Aspects has Sources configured for xAPI logging statements (optionally generated by event-routing-backends), and for tracking log statements. Each of these have their own Transforms, which validate that the log statements are JSON and forward them to Sinks which store them in ClickHouse tables.

Those tables are controlled by the variables:

ASPECTS_VECTOR_DATABASE: "openedx"
ASPECTS_VECTOR_RAW_TRACKING_LOGS_TABLE: "_tracking"
ASPECTS_VECTOR_RAW_XAPI_TABLE: "xapi_events_all"

To learn more about Vector, see the Vector documentation.