Enhance Network Monitoring and Alerting for Key Events in Autonomys Network

[0xjeremyfrank]

Description

Implementing comprehensive monitoring and alerting is essential for detecting anomalies, ensuring security, and maintaining performance. This issue proposes extending our current systems to include specific alerts for critical events and network activity.

Proposed Monitors and Alerts

Potential monitors and alerts with configurable thresholds and notifications (e.g., via email, Slack, etc.):

Top Priority for now

Network Health

• Alert: Long observed re-orgs & Network split #12
• Alert: network split
  • Same mechanism as re-org or farmer count, see those tickets
• Alert: Farmer count & Network split #13
• Analyse distribution of block times and number of votes per block, from genesis onwards #31
• Alert: Realized consensus slot times #9

Transactions

• Alert: XDM activity #8
• Domains null account balance is greater than zero #30
• Large change in total issuance #29

Lower Priority

Loss of funds, double-minting, etc.

• Calls to "Sudo" extrinsics
• Alert: Important wallet activity #7
  • Large balance transfers
• Alert: Operators slashed #11
• Block production stalled

[teor2345]

We might find it useful to add these metrics to telemetry, because they can vary across instances (or the network ignores them in some way at the consensus level):

Long observed re-orgs
Realized consensus slot times

Farmer telemetry (what?)

These metrics should be available and globally consistent to any node, but some redundancy would still be useful:

Calls to "Sudo" extrinsics
XDM activity
Farmer count (unique signing keys over a given period of time)
Operators slashed

[teor2345]

What's the biggest risk here?

1. loss of funds, double-minting, etc

• sudo
• XDM
• operator slashed
• possibly: long re-orgs, network splits, frauds (and the rest of this list)

3. frauds or malicious operator behaviour
4. network downtime
5. network splits
6. network participation/redundancy degradation

[teor2345]

For these high-priority events, some possible designs are:

1. loss of funds, double-minting, etc

• sudo
• XDM
• operator slashed

Backend

• Node logging
• Node RPCs
• Custom chain monitoring tool that takes in consensus blocks

Frontend

• Grafana dashboard
• Extra page on Astral etc
• Lightweight dedicated web page
• Custom indexing/monitoring display service

[0xjeremyfrank]

Backend

• Node logging
• Node RPCs
• Custom chain monitoring tool that takes in consensus blocks

Frontend

• Grafana dashboard
• Extra page on Astral etc
• Lightweight dedicated web page
• Custom indexing/monitoring display service

For frontend I'm thinking a lightweight dedicated web page, as this might be easier than trying to create a grafana dashboard form lots of varying types of data. That said, I'm not a grafana expert so maybe it is easier than I am envisioning.

[teor2345]

For frontend I'm thinking a lightweight dedicated web page, as this might be easier than trying to create a grafana dashboard form lots of varying types of data. That said, I'm not a grafana expert so maybe it is easier than I am envisioning.

It depends exactly what we want to show about the data. Grafana is good at graphs with labels, and tables, and allowing users to change the scale as part of investigating the data.

We could start with grafana, and then supplement it with a web page for anything we can't get it to do?
(And show them together using an embedded web page.)

[teor2345]

As a first step, let's add a grafana dashboard that shows logs when:

• a sudo call is made
• an operator is slashed

This will require a node PR to log those things, and a grafana dashboard on one of our infra mainnet nodes (do we have those?)

[teor2345]

Thinking about this a bit more: for sudo calls and operator slashing, do we want Slack/email alerts as a first step instead?

And if we do, can I test the alerts on a devnet/testnet to the #taurus channel (or similar)?

[0xjeremyfrank]

I think that makes sense. We can create a dedicated channel for the alerts.

[teor2345]

Here's how we can set alerts up from Grafana to Slack:
https://grafana.com/docs/grafana/latest/alerting/configure-notifications/manage-contact-points/integrations/configure-slack/

So the initial architecture would be:

• info logs for sudo calls on node (PR Log sudo and domain sudo calls in the runtime subspace#3656)
• (existing?) node on our infra logging to prometheus, must have runtime (can't be a bootstrap node)
• grafana dashboard showing logs, linked from alerts channel
• grafana alert to slack (and maybe email later on)

[teor2345]

Important wallet activity

@jfrank-summit what does this mean specifically? Wallets for important accounts, or something else?

As a first step, let's add a grafana dashboard that shows logs when:

• a sudo call is made
• an operator is slashed

For sudo calls, do we want to log as soon as they hit the transaction pool?
For operator slashes, when do we want to log them? Are they performed as a transaction, or just an internal runtime operation?

[teor2345]

Jeremy and I were talking about whether to put the monitoring binary in its own repo, or in the subspace monorepo.

Things from subspace we might want to depend on:

• subspace-process for process utils
• Static definitions of runtime types, like this example
  • this makes sure the monitoring system keeps up to date with changed pallets or type names
  • dynamic definitions of runtime types can silently get out of sync and miss important events
• Existing clippy/build/test CI definitions
  • We can easily test the monitoring system against the current node version

Reasons we might want a separate repo:

• CI would be a lot faster
• Branch protection rules could be relaxed to increase development speed
• we could test the monitoring system against the latest subspace commit, but there will be a delay between subspace merging a PR, and finding any bugs
  • we could schedule a daily CI run and failure notifications to make sure it doesn't get out of sync

We can move a separate repository into subspace later if needed, splitting it out is a bit harder.

Dependencies already in subspace:

• subxt
  • can use blocks, events, extrinsics, extensions, RPCs, runtime calls, and storage
  • example
  • supports a reconnecting RPC client

Dependencies not in subspace:

• slack-api

[teor2345]

Some notes on Slack message posting:

• I've created a Subspace Chain Alerts app and added it to the production and test channels
• anyone on the engineering team can log in to Slack in their browser and edit the app settings

Next step:

• we'll want to have multiple instances for redundancy
• ideally each instance will have a different name, image/emoji, and IP address, and use those in its messages
• each instance can list messages, then check their metadata for notifications with the same block and event type, and emoji react or post in the thread to avoid duplicates
  • emoji names list
  • there is a race condition here, but occasional duplicate messages don't matter