Conduit

Conduit is an open-source (Apache 2.0) Go-native data streaming platform maintained by Meroxa and the ConduitIO community. Each deployment on Guara Cloud runs a single /app/conduit binary executing exactly one pipeline: a source connector, optional in-line processors, and a destination connector, all described by a single pipeline.yml file.

Compared with the JVM-based alternative, Conduit trades some CDC depth for a much smaller footprint and a broader connector surface: NATS subjects, Kafka topics, and HTTP endpoints are first-class sources and sinks alongside the usual databases. State lives in a local BadgerDB at /app/conduit.db on a Longhorn PVC, so restarts resume from the last committed position.

When to use Conduit

Conduit is the right choice when:

• You want a lightweight Go runtime — no JVM heap to size, no Quarkus warm-up. Conduit comfortably runs on the Hobby tier.
• Your source or sink is something other than a database: a NATS subject, an HTTP endpoint, or a second database (Postgres-to-MySQL, Postgres-to-Postgres).
• You want in-line PII scrubbing via the built-in field.exclude processor — strip sensitive columns before they reach the sink, declaratively, without writing code.
• You like a declarative YAML pipeline you can read end-to-end in one file.

Pick Debezium instead when you need battle-tested CDC against a production OLTP database, or when the source is MongoDB (Conduit v1 does not ship a MongoDB source — see the troubleshooting section below).

Deploy walkthrough

Insights tab

The Insights tab is fed by Conduit’s native Prometheus exporter on the HTTP port — no sidecar, no extra scrape adapter. Panels are grouped into two sections:

Pipeline Throughput

• Throughput — area, records per second across all connectors (rate(conduit_connector_execution_duration_seconds_count[1m])). Your baseline depends on source volume; the shape is what you read.
• Active Connectors — gauge, the number of connectors currently live (usually 2: source + destination). A drop to 1 or 0 means a connector is down.
• Processor p95 Duration — sparkline, p95 latency added by in-line processors. field.exclude should stay well under a millisecond; spikes suggest payload bloat.

Pipeline Health

• DLQ Bytes — stacked area, total bytes written to the dead-letter queue (conduit_dlq_bytes_sum). Any growth means records are failing to write to the sink and are being logged instead.
• Connector p95 Duration — area, p95 latency of connector reads and writes. Sustained spikes on the source connector point to a slow source; spikes on the destination mean the sink is back-pressuring.
• Pipeline Status — status indicator, binary health of the pipeline (1 = running, 0 = stopped or degraded).

Troubleshooting

Pipeline stuck in DLQ

Conduit routes records that fail to write to a built-in log DLQ. The DLQ Bytes panel on the Insights tab shows whether growth is happening; the pod logs (filter by log.format=json) include the full reason for each failure.

Common causes:

• The sink’s schema doesn’t match the source (e.g. an expected column is missing on a destination Postgres table). Fix the sink schema and restart the pipeline — records already DLQ’d are not retried automatically.
• An external sink rejects auth mid-stream because the credentials rotated. Update the external credentials via the catalog wizard; Conduit re-reads them on pod restart.

Standalone connector fails to start

MySQL, NATS JetStream, and HTTP are pulled by an initContainer on first boot. If the init container fails, kubectl describe pod (or the Guara Cloud pod logs tab) shows a clear download failed or permission denied error. Three causes:

• arm64 availability. Guara Cloud runs on arm64 (OCI Ampere A1). Any pinned connector release must publish an arm64 binary; all supported versions do.
• Network policy. The pod needs egress to github.com on 443 for the download. Private-network projects that deny external egress have to switch to built-in connectors (PostgreSQL or Kafka).
• Image mirror drift. If the connector release is yanked upstream, the initContainer returns a 404. Re-deploy with an older Conduit version, or open a support ticket.

Postgres replication slot not cleaned up after delete

Deleting a Conduit service with a PostgreSQL source tears down the pod and its PVC; in the happy path the replication slot and publication on the source PostgreSQL are dropped automatically during shutdown. If the pod was terminated before cleanup finished, the slot and publication can survive. Retained slots hold WAL files forever, which will fill the source’s disk.

Conduit names its slot guara_slot_<service-slug> and its publication guara_pub_<service-slug>. Because Postgres identifiers cannot contain hyphens, the service slug’s - are replaced with _ at pipeline load time. Run, against the source database:

-- Replace hyphens in your service slug with underscores (e.g. my-pipe → my_pipe).
SELECT pg_drop_replication_slot('guara_slot_YOUR_SERVICE_SLUG');
DROP PUBLICATION IF EXISTS guara_pub_YOUR_SERVICE_SLUG;

List live slots with SELECT slot_name, active FROM pg_replication_slots; if you need to confirm the exact name.

MongoDB as a source is unavailable

Conduit v1 on Guara intentionally does not offer a MongoDB source. The upstream conduit-connector-mongo configuration shape differs enough from the other connectors that we decided the risk of misconfiguration at pipeline load was too high for v1. If you need MongoDB CDC, deploy a Debezium service — it ships a battle-tested MongoDB connector today.

External NATS URLs must embed credentials inline

Conduit’s NATS connector reads authentication from the URL itself rather than separate username/password fields:

nats://user:password@host:4222

Sibling NATS catalog services already inject an auth-embedded URL via the credential bridge. When pointing at an external NATS cluster, pre-encode the username and password into the URL you paste into sourceExternalUrl or sinkExternalUrl. The separate sourceExternalUsername / sourceExternalPassword fields are ignored by the NATS connector and can be left blank.

Processor silently drops the wrong field

The field.exclude processor strips record fields by an exact dot-path reference. Paths must start with a leading dot and address the target inside the OpenCDC record envelope — most often .Payload.After.<column> for database sources. Example:

.Payload.After.email,.Payload.After.cpf

Common pitfalls:

• The path always starts with a leading .; Payload.After.email (no leading dot) silently matches nothing.
• Nested paths use the same dot syntax (.Payload.After.metadata.pii), not JSON pointer.
• Field names are case-sensitive.
• Postgres column names are lower-cased by default; Email in your schema is email by the time it reaches the processor.

Verify from the pod logs with log.level=debug (temporary, via a redeploy) to see each record’s schema before and after processing.

Upstream resources

• Conduit documentation — pipeline YAML reference, connector configuration, processor list.
• Conduit on GitHub — source code, releases, issue tracker.
• Connector catalogue — built-in and standalone connectors with their configuration fields.

Next steps