You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
*Description:* This option can be used to enable or disable the chained hostname format. For more information, see the chain-hostnames() global option.
*Description:* This parameter assigns an emergency level to the messages
7
-
received from the {{ page.src | default: 'file' }} source if the message does not specify one. For
8
-
example, default-priority(warning).
10
+
*Description:* This option defines the default level value if the `PRIORITY` entry does not exist in the msg received from the {{ page.src | default: 'file' }} source.
Copy file name to clipboardExpand all lines: doc/_admin-guide/060_Sources/038_Kafka/001_Kafka_options.md
+29Lines changed: 29 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,7 @@
1
1
---
2
2
title: "Options of the kafka() source"
3
3
id: adm-src-kafka-opt
4
+
src: kafka
4
5
description: >-
5
6
This section describes the options of the kafka() source in {{ site.product.short_name }}.
6
7
---
@@ -13,18 +14,32 @@ To use the kafka() source, the following two options are required: bootstrap-ser
13
14
14
15
{% include doc/admin-guide/options/bootstrap-servers.md %}
15
16
17
+
{% include doc/admin-guide/options/chain-hostname.md %}
18
+
16
19
{% include doc/admin-guide/options/config-kafka.md kafka_type='consumer' type='source' protected_options='`bootstrap.servers``metadata.broker.list``enable.auto.offset.store``auto.offset.reset``enable.auto.commit``auto.commit.enable`' %}
17
20
21
+
{% include doc/admin-guide/options/default-facility.md %}
22
+
23
+
{% include doc/admin-guide/options/default-priority.md %}
24
+
18
25
{% include doc/admin-guide/options/disable-bookmarks.md %}
19
26
See Bookmarking in the kafka() source for more details.
20
27
28
+
{% include doc/admin-guide/options/dns-cache.md %}
29
+
21
30
{% include doc/admin-guide/options/hook.md %}
22
31
32
+
{% include doc/admin-guide/options/host-override.md %}
33
+
23
34
{% include doc/admin-guide/options/ignore-saved-bookmarks.md %} (depending on the setting of the read-old-records() option.\
24
35
See Bookmarking in the kafka() source for more details.
25
36
26
37
{% include doc/admin-guide/options/kafka-logging.md %}
27
38
39
+
{% include doc/admin-guide/options/keep-hostname.md %}
40
+
41
+
{% include doc/admin-guide/options/keep-timestamp.md %}
42
+
28
43
## log-fetch-limit()
29
44
30
45
| Type: | integer |
@@ -60,6 +75,8 @@ For example, with `workers(3)` and `fetch-limit(100000)`, the 2 processor worker
60
75
61
76
*Description:* When the main worker reaches the queued message limit defined by fetch-limit(), the kafka() source temporarily stops retrieving messages from the broker. It then waits for the duration specified by `fetch-queue-full-delay()` before attempting to fetch additional messages.
62
77
78
+
{% include doc/admin-guide/options/normalize-hostnames.md %}
79
+
63
80
{% include doc/admin-guide/options/persist-name.md %}
64
81
65
82
## persist-store()
@@ -78,9 +95,13 @@ For more details, see Bookmarking in the kafka() source.
78
95
79
96
*Description:* Specifies the maximum amount of time {{ site.product.short_name }} waits during a Kafka broker poll request for new messages to become available.
80
97
98
+
{% include doc/admin-guide/options/program-override.md %}\
99
+
81
100
{% include doc/admin-guide/options/read-old-records.md %}\
82
101
See Bookmarking in the kafka() source for more details.
83
102
103
+
<!-- {% include doc/admin-guide/options/sdata-prefix.md %} -->
104
+
84
105
## single-worker-queue()
85
106
86
107
| Type: | yes \| no |
@@ -121,13 +142,17 @@ Why is it worth using dual consumer strategies? describes the differences betwee
121
142
122
143
For details on how the resulting topic names, partitions, and Kafka assign/subscribe strategies are determined in different scenarios, see the Basic strategy usage cross-reference of the different topic configuration cases ; for information on how the resulting strategy participates in offset storing and bookmarking, refer to Bookmarking in the kafka() source.
123
144
145
+
{% include doc/admin-guide/options/tags.md %}
146
+
124
147
## time-reopen()
125
148
126
149
| Type: | integer in seconds |
127
150
| Default:| 60 |
128
151
129
152
*Description:* The time {{ site.product.short_name }} waits between attempts to recover from errors that require re-initialization of the full kafka connection and its internally used data structures.
130
153
154
+
{% include doc/admin-guide/options/time-zone.md %}
155
+
131
156
## topic()
132
157
133
158
| Type: | key-value pairs |
@@ -158,6 +183,10 @@ The partition number must be:
158
183
159
184
For details about how the resulting topic names, partitions, and Kafka assign/subscribe strategies are determined in different scenarios, see Basic startegy usage cross-reference of the different topic configuration cases and Why is it worth using dual consumer strategies?
| topic( "^topic-name-[13]$" => "-1" } | topic-name-1, topic-name-3 | all, all | assign | subscribe |
44
44
| topic( "^topic-name-[13]$" => "-1" } | topic-name-1, topic-name-3 | all, all | subscribe | subscribe |
45
45
@@ -60,9 +60,24 @@ Using both consumer strategies — `assign` and `subscribe` — provides the fle
60
60
61
61
By supporting both approaches, {{ site.product.short_name }} can be used effectively in a variety of Kafka consumption models — from tightly controlled, partition-specific pipelines to dynamic and scalable consumer setups that evolve with the broker configuration.
62
62
63
+
For more details about Kafka consumer types and rebalancing, refer to the librdkafka documentation.
64
+
63
65
## Bookmarking in the kafka() source
64
66
65
-
By default, {{ site.product.short_name }} stores the offset of the last read message of each topic it consumes in its own persist file. This can be disabled using the disable-bookmarks() option. Automatic offset restoration takes effect at startup or reload, based on the saved offset value and the ignore-saved-bookmarks() and read-old-record() settings. If ignore-saved-bookmarks() is set to `yes`, it will not use the saved offset. Instead, if read-old-record() is set to `yes`, it will start fetching from the oldest available message, otherwise it will start from the newest one.
67
+
Where to store the information about the latest consumed Kafka partition offsets can be controlled via the persist-store() option. By default, {{ site.product.short_name }} stores the offset of the last read message of each topic and partition it consumes in its own persist file. Storing this information altogether can be disabled using the disable-bookmarks() option. Automatic offset restoration takes effect at startup or reload, based on the saved offset value and the ignore-saved-bookmarks() and read-old-records() settings. If ignore-saved-bookmarks() is set to `yes`, it will not use the saved offset. Instead, if read-old-records() is set to `yes`, it will start fetching from the oldest available message; otherwise, it will start from the newest one.\
68
+
69
+
In most cases, you will want to use the `local` storage option, but you can also store the bookmarks remotely on the Kafka side partitions by setting the persist-store() option to `remote`. The only case where using remote-side storage is worth it is when you have multiple kafka() clients consuming the same partitions and using the same consumer group `group.id` in your config(). In this case, you **must use**`remote`; otherwise, you risk data loss and significant message duplication. For all other cases, use the `local` storage option instead.
| assign | remote | used for remote Kafka offset restoration, auto-generated if empty | even if clients use the same "group.id", they are isolated and will NOT participate in Kafka rebalancing |
77
+
| subscribe | local | not used and will be overridden if not empty | because of the `local` offset storage option, the user-provided "group.id" will be replaced by a self-generated one to prevent clients with the same "group.id" from participating in Kafka rebalancing; otherwise, inconsistent offsets might be provided by the Kafka broker |
78
+
| subscribe | remote | used for remote Kafka offset restoration and rebalancing, auto-generated if empty | clients with the same "group.id" will participate in Kafka rebalancing |
79
+
80
+
For more details about Kafka consumer types and rebalancing, refer to the librdkafka documentation.
*Description:* String for native macOS log message filtering using predicates. For example, the following predicate selects AirDrop logs: `subsystem=="com.apple.sharing" and category=="AirDrop"`
19
29
20
-
{% include doc/admin-guide/options/disable-bookmarks.md %}
21
-
22
30
## go-reverse()
23
31
24
32
|Type:| boolean|
@@ -28,10 +36,18 @@ The `darwin-oslog()` source has the following options.
28
36
29
37
{% include doc/admin-guide/options/hook.md %}
30
38
39
+
{% include doc/admin-guide/options/host-override.md %}
40
+
41
+
{% include doc/admin-guide/options/hook.md %}
42
+
31
43
{% include doc/admin-guide/options/ignore-saved-bookmarks.md %} (depending on the setting of the go-reverse() and the read-old-records() options.
32
44
33
45
{% include doc/admin-guide/options/deprecated-options.md old='do-not-use-bookmark()' new='ignore-saved-bookmarks()' %}
34
46
47
+
{% include doc/admin-guide/options/keep-hostname.md %}
48
+
49
+
{% include doc/admin-guide/options/keep-timestamp.md %}
50
+
35
51
{% include doc/admin-guide/options/log-fetch-delays.md %}
36
52
37
53
{% include doc/admin-guide/options/log-fetch-limit.md %}
@@ -43,4 +59,15 @@ The `darwin-oslog()` source has the following options.
43
59
44
60
*Description:* The maximum distance in seconds that a bookmark can point backward. That is, if {{ site.product.short_name }} is stopped for 10 minutes (600 seconds) and `max-bookmark-distance()` is set to `60`, then {{ site.product.short_name }} starts reading the logs from 60 seconds before the startup, missing 9 minutes (540 seconds) worth of logs.
45
61
62
+
{% include doc/admin-guide/options/program-override.md %}
63
+
46
64
{% include doc/admin-guide/options/read-old-records.md %}
65
+
66
+
{% include doc/admin-guide/options/tags.md %}
67
+
68
+
{% include doc/admin-guide/options/time-zone.md %}
0 commit comments