The parser is running

Summary (all instances)

Status OK
Per instance in plain text: /health/processing
Messages completed per minute (last 1 / 5 / 15 min) 684 / 781 / 870
Like a load average; an instance that started recently counts over its uptime.
Backlog 1 records waiting, oldest waiting for 0 s
Records waiting: consumer lag (not committed yet). Oldest: polled but not completed.
Problems in the last 15 min 0 retries, 0 parked, 3631 unknown devices
Unknown devices include messages without device id and ids found in several accounts.
Instances1

Instances

Instance Status Uptime Version Assigned partitions Messages per minute (last 5 min) In progress CPU cores (avg last min) Heap used / max (MB) Live data after the last full collection (MB) Memory of the machine (MB)
cle-usa-pro-doc-doc-fc0885b40bab (this one) OK 54 min 31 s 1.20.2.1 12 781 1 0.51 483 / 1024 370 2048

Heap used rises until Java cleans up. Live data after the last full collection is what's really in use: if it keeps rising from one collection to the next, memory leaks.

How messages ended

Outcome Last 15 min Since start Top networks (last 15 min) Top networks (since start)
Unknown device 3631 16481 senet: 2657, simple_generic: 673, digital_matter_oem: 301 senet: 11510, simple_generic: 3916, digital_matter_oem: 1053, wmw_bot: 2
Invalid metadata 4 11 senet: 4 senet: 11
Exact duplicate 0 2 senet: 2
Old message 10 749 inactivity: 10 inactivity: 551, senet: 198
Fully processed 9411 47661 senet: 8537, inactivity: 822, digital_matter_oem: 30, cart_retrieval_status: 11, cart_presumed_lost: 6 senet: 43432, inactivity: 3296, wmw_bot: 714, digital_matter_oem: 155, cart_retrieval_status: 40

"Since start" adds up the counts of each instance since its own start.

Message processing time

Messages (last 5 min) Avg (ms) p95 (ms) p99 (ms) Messages since start
3906 320.7 818 1211 64904

Includes messages dropped early (unknown device...). Percentiles are accurate within about 4%.

HTTP calls by URL (worst first)

HTTP time in the last 5 min: 983680 ms (78.5% of message processing time)
Only calls made while processing a message. Calls in parallel can add up to more than 100%.

URL template Share of processing time (last 5 min) Calls (last 5 min) Avg (ms, last 5 min) p95 (ms, last 5 min) Calls since start Total since start (ms)
https://127.0.0.1:8449/api/v1/assets/{id}/rules 23.2% 2730 106.3 216 47642 3893565
https://127.0.0.1:8451/active-alarms/by-asset-id/{id} 15.2% 2546 74.9 171 44633 2946471
https://127.0.0.1:8449/api/v1/asset-in-zones 13.4% 1751 96.2 200 28998 2242197
https://127.0.0.1:8449/api/v1/asset-at-locations 13.3% 1871 88.8 200 30948 2160283
https://127.0.0.1:8449/api/v1/devices/by-device-id/{token} 5.1% 1264 50.7 152 17615 900742
https://127.0.0.1:8451/account-id/{id}/history-items/last-unclosed-geofence-zone-dwell-time/asset-id/{id}/zone-id/{id}/before/{id} 2.3% 381 75.9 171 5990 523995
/api/v1/asset-in-zones 1.8% 169 130.1 253 2731 314331
https://127.0.0.1:8451/account-id/{id}/history-items 1.4% 328 54.2 146 4863 306373
https://127.0.0.1:8451/account-id/{id}/history-items/{id} 1.4% 192 91.0 200 3157 257492
https://127.0.0.1:8451/account-id/{id}/history-items/last-unclosed-location-dwell-time/asset-id/{id}/location-id/{id}/before/{id} 1.1% 221 60.9 192 3356 481751
https://127.0.0.1:8451/states/{uuid} 0.2% 91 25.7 91 1160 64223
/api/v1/asset-at-locations 0.1% 16 102.5 225 414 34195
https://127.0.0.1:8449/api/v1/shopping-cart-retrievals/{token} 0.0% 8 43.8 91 51 1972
https://127.0.0.1:8449/api/v1/assets/{id} 0.0% 1 16.0 16 20 780
https://127.0.0.1:8451/account-id/{id}/history-items/last-unfinished/asset-id/{id}/type/{id}/before/{id} 0.0% 0 0.0 0 20 1901
https://127.0.0.1:8451/active-alarms 0.0% 0 0.0 0 30 1786
https://127.0.0.1:8449/api/v1/zones/flat 0.0% 0 0.0 0 9 867
https://127.0.0.1:8449/api/v1/locations 0.0% 0 0.0 0 5 511
https://127.0.0.1:8449/api/v1/beacons/for-zoning 0.0% 0 0.0 0 4 459
https://127.0.0.1:8452/downlink-by-enterprise 0.0% 0 0.0 0 3 90
https://127.0.0.1:8451/account-id/{id}/messages/device-uuid/{uuid}/timestamp/{id} 0.0% 0 0.0 0 1 58
https://127.0.0.1:8449/api/v1/devices/by-device-id/{token}/FASDFSDF 0.0% 0 0.0 0 2 55

Kafka consumer (all instances)

Assigned partitions12
Paused partitions none
Consumer lag (records not committed yet)1
Lag growing for 10 min on none
Buffered records (not completed)1
Buffered MB (estimated memory)0.00
Waiting to start / in progress / waiting for a retry 0 / 1 / 0
Keys waiting for an earlier message of the same device0
Largest per-key backlog (top 5) none
Oldest waiting record (s)0
Circuit breakerclosed
JVM per instance

cle-usa-pro-doc-doc-fc0885b40bab

Sampled every second, last 60 s Average Max
Parser CPU (cores, 1.0 = one full core) 0.51 1.29
System CPU as seen by the JVM (%) 50.5 99.0
Busy message processing threads 4.1 25

Garbage collection: 26 collections, 366 ms in total (0.6% of the time), longest 38 ms.

Processors available to the JVM: 2. Memory: 2048 MB. Heap: 483 MB used, 524 MB committed, 1024 MB max. Garbage collectors: G1 Young Generation, G1 Concurrent GC, G1 Old Generation.

JVM arguments: -XX:-UseBASE64Intrinsics --enable-native-access=ALL-UNNAMED -XX:MaxRAM=2048m -XX:MaxRAMPercentage=50.0 -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=30 -Djava.security.egd=file:/dev/./urandom

Rarely needed counters (all instances, since start)
Skipped exact duplicates2
Skipped offsets already completed before a restart0
Consumer restarts by the watchdog0
Messages completed64904