Maintenance Windows - Splunk

Before you Start

• Ensure you have access to Splunk through secure
• If you do not, with your primary add omni_splunk_mltk, SSMOSplunk_gvp_Nonprod, SSMOSplunk_gvp_Prod, SSMOSplunk_mlift_ogs_Prod

Why are Maintenance Windows Important

They are important because they allow you to keep track of the alerts you get within the patching windows time. Canceling all alerts wouldnt help because you still want to see them. The maintenance window will allow you to mute the alerts during that patching window time.

Procedure

1. Open the Splunk to GVP IVR application.
2. Go Alerts to see any alerts you have.
3. That alert logic will remodeled to fit with the maintenance window concept. If there is an alert that needs immediate attention, please escalate it immediately

Macro: AvoidServerPatchingWindow

This macro has been developed to be versatile and can be used in multiple ways within your SPL query. It works by manipulating the 'datetimeField' parameter to adjust for server patching windows.

Dependencies

• Lookup: Relies on 'omni_host.csv'

Arguments

• Server name Field: The name of the field to corrrelate with 'vmname' and 'fqdn' in the 'omni_hosts.csv' lookup.

• Datetime Field: The name of the field to correlate with 'p_beg_epoch' and 'p_end_epoch' in the 'omni_hosts.csv' lookup.

Permissions

• Read/Write: gvp_power
• Read Only: gvp_user

---

Use Examples

Example 1: Exclude logs Pre-Alert Logic

Exclude logs generated during their respective host's Server Patching Window BEFORE applying alert logic.

| tstats count where index=cba_omni sourcetype=omni:ors:voice host IN(apsrp02323.uhc.com,rp000026906,rp000109169,apsrp09836.uhc.com,apsrp10162 ) earliest=-24h@h latest=@h by _time host span=30m
| `AvoidServerPatchingWindow(host,_time)`
/// Alert Logic Goes Here, AFTER suppressing logs we want to ignore ///
| timechart sum(count) as count span=30m by host

Example 2: Suppress Alerts Post-Alert Logic

After applying alert logic, suppress alerting on servers if the current time is within their Patching Window.

| tstats count where index=cba_omni sourcetype=omni:ors:voice host earliest=-15m latest=now by host
 /// Alert Logic Goes Here ///
| `AvoidServerPatchingWindow(host,_time)`
| table host count

Work Flow example

Below is an example of a current work query and updating it to fit the example above.

Example

index="cba_omni" sourcetype="omni:gws"  "JettySettings  Jetty settings initialized" |`strip_domain(host)`
|lookup omni_hosts vmname as host OUTPUT genesys_name, data_center,env
|table _time,env,host,genesys_name

text

Completed example

|tstats count where index=cba_omni sourcetype=omni:ors:voice host IN(apsrp02323.uhc.com,rp000026906,rp000109169,apsrp09836.uhc.com,apsrp10162 ) earliest=-24h@h latest=@h by _time host span=30m
| `AvoidServerPatchingWindow(host,_time)`
| table _time, env, host, genesys_name, count

How to view your Maintenance Windows

In the Visualization tab you can see the graph the new alert created. With that and the Statistics you can see the alert suppressing alerts during the patching times.