More automation tasks the system performs, more device data will be retrieved and more task data will be generated, such as route tables, configuration files, execution logs and so on.

To proactively prevent the system database from data loss or even corruption, you can customize MongoDB, Front Server, and Elasticsearch disk alert rules with progressive quotas assigned. When the usage reaches the predefined threshold, specified users can be notified by both email alerts and system alerts. Each rule defines different actions with two conditions of the disk usage threshold. The corresponding action will be triggered when either condition is met. The thresholds are all configurable.

Notes:

1) To enable this function, the email server settings must be configured first. The default frequency for email alerting is 1 hour, indicating the system will not send duplicate emails within 1 hour. It is configurable and other positive integers are also allowed.

2) The deleted historical data includes configuration files, data tables, CLI commands and parser data for all domains in this MongoDB instance.

3) If Rule C is met, the MongoDB serving this domain will become read-only. The disabled writing permissions include: saving maps, saving data as current baseline, and so on. To enable the writing permissions, you can modify the Rule C.

4) The default thresholds for Rule C are recommended values based on Elasticsearch configurations since it shares the disk with MongoDB. When the disk usage reaches 95%, Elasticsearch will also become read-only. Refer to https://www.elastic.co/guide/en/elasticsearch/reference/6.7/disk-allocator.html for more details.

 
For example, when MongoDB usage reaches a predefined threshold, users will receive both email alerts and in-place warnings in the system. Here is an example of system warning:

Tip: When you encounter the unexpected disablement of written permissions to MongoDB, you can Reset Root Role for recovery.

Customizing Email Alerting Rules

To customize alert rules for MongoDB disk usage or any service anomaly, do the following:

1.In the System Management page, click Operations > Service Monitor from the quick access toolbar.

2.In the Service Monitor home page, click Alert Rules at the upper-right corner. The default settings are as follows. When a Front Server disk or Elasticsearch disk usage is high, the system will also push alerts by email

3.Change the settings based on your needs.

1)Specify the disk usage threshold for different levels.

Note: To email alerts when a server is disconnected or a service is stopped, select the corresponding check box.

2)Enter the email address in the Send Email To or CC fields.

Note: Email alerts are enabled only when email addresses are added at least in one field. Use a colon or semicolon to separate multiple items.

3)Specify the frequency to send emails.

4)Click OK to save the configuration.

Resetting Root Role for MongoDB

When you encounter the unexpected disablement of written permissions to MongoDB, do the following:

1.Run the following command to connect MongoDB.

▪If SSL is not enabled:

/bin/mongo 127.0.0.1:27017/admin -u <username> -p <password>  --authenticationDatabase admin  --authenticationMechanism SCRAM-SHA-256

▪If SSL is enabled:

/bin/mongo 127.0.0.1:27017/admin -u <username> -p <password> --authenticationDatabase admin --ssl --sslAllowInvalidCertificates --authenticationMechanism SCRAM-SHA-256

2.Run the use admin command and show users command to check the status of the admin role.

> use admin
> show users
{
        "_id" : "admin.DBUsername",
        "user" : "DBUsername",
        "db" : "admin",
        "roles" : [
                {
                "role" : "roleLimitwrite",
                "db" : "admin"
                }
        ],
        "mechanisms" : [
                "SCRAM-SHA-256"
        ]
}

3.If the role displays as "roleLimitwrite", run the db.grantRolesToUser("DBUsername", [{role:"root", db:"admin"}]) command to reset the role to "root".

 

See also:

▪Cleaning Domain Data