Ansible Variable Management Strategy

This document outlines our strategy for managing Ansible variables in an environment using Red Hat Insights for dynamic inventory.

Overview

In our Ansible setup, we categorize variables into two primary categories:

1. Build Variables: These variables are required for server provisioning and are defined upfront based on the resource needs of different application roles.

2. App Variables: These variables are specific to application roles and are primarily used for post-build activities.

We utilize Red Hat Insights for dynamic inventory, which allows us to assign variables to inventory items (hosts, groups, etc.) and define groups based on facts collected by Insights. This enables us to define variables in Insights and have them automatically applied to the appropriate hosts, while still allowing for group and host-specific variables defined within playbooks and roles.

Best Practices for Managing Variables in Ansible

When it comes to managing variables in Ansible, the answer to the question of where to define a variable depends on various factors. However, we follow these best practices to ensure scalability, maintainability, consistency, and clarity in our environment.

Scalability and Maintainability

As our environment grows, the number of variables we need to manage can rapidly increase. To keep everything scalable, we organize variables using the following structure:

• group_vars/cl1/all.yml: This file contains default values for variables.
• group_vars/cl1/dev/all.yml: This file contains environment-specific overrides.

This structure allows us to easily add new environments or subgroups without having to search through a massive, single directory or file. Each environment or subgroup has its own dedicated space, making it clear which variables apply where. Scaling up our infrastructure becomes smoother and more manageable.

Consistency and Clarity

Adopting a standard structure for variables promotes consistency across our project. It makes it easier for anyone working on the project to understand how variables are organized, where to find them, and where to add new ones. This is particularly beneficial in a team setting where multiple people are involved.

Avoiding Conflicts

Variables in Ansible follow a specific order of precedence, with variables defined in host_vars overriding those in group_vars. By organizing our variables accordingly, we can fully utilize this feature to manage our configuration. We define default settings in group_vars/cl1/all.yml and then override those defaults for specific environments or hosts as needed. This allows us to manage the unique needs of different parts of our infrastructure without conflicts.

Simplicity

Although this structure may initially seem more complex, it simplifies the management of variables in the long run. Instead of dealing with one massive file or directory full of variables, we have a neatly organized structure where each file or directory serves a clear purpose.

Flexibility

Our approach provides flexibility, allowing us to easily adjust our configuration to meet the changing needs of our infrastructure. Adding a new app, environment, or host can be done without disrupting our existing setup.

For a visual guide to the order of precedence and layering of Ansible variables, refer to the following documentation:

• Ansible Documentation
• Red Hat Insights Documentation.

graph TB
    subgraph playbook [Run an Ansible Playbook]
    A[(Insights Dynamic Inventory)]
    B(["ansible_playbooks/pb_some_playbook.yml"])
    end
    subgraph role_defaults [Role Defaults]
    C[["ansible_role_name/defaults/main.yml"]]
    end
    subgraph group_vars [Group Vars]
    D[[inventory group_vars/all]]
    E[[playbook group_vars/all]]
    F[[inventory group_vars/*]]
    G[[playbook group_vars/*]]
    D --> E
    E --> F
    F --> G
    end
    subgraph host_vars [Host Vars]
    I[[inventory host_vars/*]]
    J[[playbook host_vars/*]]
    K[[host facts / cached set_facts]]
    I --> J
    J --> K
    end
    subgraph play_vars [Play Vars]
    L[[play vars]]
    M[[play vars_prompt]]
    N[[play vars_files]]
    L --> M
    M --> N
    end
    subgraph role_block_task_vars [Block & Task Vars]
    O[["role role/vars/main.yml"]]
    P[["block vars"]]
    Q[["task vars"]]
    O --> P
    P --> Q
    end
    subgraph misc_vars [Miscellaneous Vars]
    R[[include_vars]]
    S[[set_facts]]
    T[[role / include_role params]]
    U[[include params]]
    R --> S
    S --> T
    T --> U
    end
    V["`Extra vars (-e always win precedence)`"]
    playbook --> role_defaults
    role_defaults --> group_vars
    group_vars --> host_vars
    host_vars --> play_vars
    play_vars --> role_block_task_vars
    role_block_task_vars --> misc_vars
    misc_vars --> V

Ansible Variables Precedence

Variables in Ansible are organized according to their precedence. Here's a breakdown:

1. Role Defaults: These are defined in the defaults directory of each role. For example, in ansible_playbooks/roles/ansible_role_name/defaults/main.yml. More Info

2. Global and Group Variables: These are suitable for build variables and application variables that are common across all application roles. They can be defined in two places:

   • Inventory - Global Vars: Defined in the Insights Inventory. More Info

   • Playbooks - Global Vars: Defined in the group_vars directory of the playbook. For example, in ansible_playbooks/group_vars/all.yml. More Info

     ---
     env: prod
     dc: elr|ctc
     az: aza|azb|azt
     

3. Host Variables: These can be defined in two places:

   • Inventory - Host Variables: Defined in the Insights Inventory. More Info
   • Playbook - Host Variables: Defined in the host_vars directory of the playbook. For example, in ansible_playbooks/host_vars/host_name.yml. More Info
   • Host Facts / cached set_facts

4. Play Variables: These are defined in the playbook itself. They can be of three types:

   • Play Vars: Defined in the vars section of a playbook.

     ---
     - hosts: all
        vars:
          play_var1: value1
         play_var2: value2
     

   • Play Vars Prompt: Defined in the vars_prompt section of a playbook.

     ---
     - hosts: all
       vars_prompt:
         - name: "play_var1"
           prompt: "Enter a value for play_var1"
           private: no
         - name: "play_var2"
           prompt: "Enter a value for play_var2"
           private: no
     

   • Play Vars Files: Defined in the vars_files section of a playbook.

     ---
     - hosts: all
       vars_files:
         - some_file.yml
     

5. Role Variables: These are defined in the role itself. They can be of three types:

   • Role vars: Defined in ansible_role_name/vars/main.yml.

     app_var1: value1
     app_var2: value2
     

   • Block vars: Defined in the block section of a role.

     - name: "Some Block"
       block:
         - name: "Some Task"
           debug:
             msg: "Some Message"
       vars:
         block_var1: value1
         block_var2: value2
     

   • Task vars: Defined in the task section of a role.

     - name: "Some Task"
       debug:
         msg: "Some Message"
       vars:
         task_var1: value1
         task_var2: value2
     

6. Miscellaneous Variables: These include:

   • include_vars: Defined in the include_vars section of a playbook.
   • set_facts: Defined in the set_facts section of a playbook.
   • role / include_role params: Defined in the role / include_role params section of a playbook.
   • include params: Defined in the include params section of a playbook.

7. Extra vars: These are defined using the -e flag when running a playbook. For example, ansible-playbook -e "extra_var1=value1". These always win precedence.

Variable Definition Examples and guidelines

Global Variables - Build Variables & Application Variables

Global variables should be defined in a group_vars/all.yml file. This file is a good place to put any variables that are needed by all hosts. For example:

aide: <askid>

Group Variables - Build Variables & Application Variables

For variables that are specific to a group of hosts, you should use the group_vars directory. For example, you might have a file group_vars/cl1.yml with:

cpu: 8
memory: 16GB

And a group_vars/cl2.yml file with:

cpu: 16
memory: 32GB

With a dynamic inventory, the groups would be defined based on the facts collected by Insights and can be used to group your hosts.

Examples

Example 1: Group Variables - Build Variables

The recommended Ansible variable strategy is as follows:

1. Create a group_vars directory in your ansible_role, or ansible_playbooks directory. This is where you'll store the variables for each of your clusters.

   group_vars/
     cl1/
       prd/
         gws/
           gws_api.yml  # Variables specific to the gws_api sub-app in production on cluster cl1
         gws.yml        # Variables specific to the gws app in production on cluster cl1
       prd.yml          # Variables specific to the production environment on cluster cl1
     cl1.yml            # Variables specific to cluster cl1
     cl3/
       dev/
         cassandra/
           ors.yml     # Variables specific to the ors sub-app in development on cluster cl3
           urs.yml     # Variables specific to the urs sub-app in development on cluster cl3
           gws.yml     # Variables specific to the gws sub-app in development on cluster cl3
         cassandra.yml  # Variables specific to the cassandra app in development on cluster cl3
       dev.yml          # Variables specific to the development environment on cluster cl3
     cl3.yml            # Variables specific to cluster cl3
     all.yml            # General variables that apply to all hosts  ...
   

2. Define variables in a nested format within each cluster file. Key variables (cluster, env, availability_zone, dc, app, subapp) should be nested according to their hierarchy, along with other conditional defaults according to type. These act as the defaults for each cluster, and can be overridden by defining the same variable at a lower level. For example, a variable defined in cl1.yml can be overridden by the same variable defined in cl1/dev.yml, which can be further overridden by the same variable defined in cl1/dev/gws.yml, and so on.

For example, in ansible_playbooks/group_vars/cl1.yml:

---
cluster: cl1
env:
  dev:
    availability_zone: aza
    dc: ctc # Lower environment default is CTC
  test:
    availability_zone: aza
    dc: ctc # Lower environment default is CTC
  stage:
    availability_zone:
      - aza
      - azb
    dc:
      - ctc
      - elr
  prod:
    availability_zone:
      - aza
      - azb
    dc:
      - ctc
      - elr
app: # define an application configuration for each app and optional subapp
  gws: # required app gws configuration
    cpu: 2
    memory: 4GB
    disk:
      path: /path/to/disk
      size: 50GB
    os: redhat8
    az: aza
    gws_api: # optional subapp gws_api configuration, this needs to be defined as a subkey of "app: gws" named gws_api
      cpu: 4
      memory: 8GB
      disk:
        path: /path/to/disk
        size: 100GB
      os: redhat9
      az: azb
    gws_web: # optional subapp gws_web configuration, this needs to be defined as a subkey of "app: gws" named gws_web
      cpu: 2
      memory: 4GB
      disk:
        path: /path/to/disk
        size: 50GB
      os: redhat8
      az: aza
  mcp: # required app mcp configuration
    cpu: 2
    memory: 4GB
    disk:
      path: /path/to/disk
      size: 50GB
    os: redhat8
    az: aza
      mcp_conf: # optional subapp mcp_conf configuration, this needs to be defined as a subkey of "app: mcp" named mcp_conf
        cpu: 4
        memory: 8GB
        disk:
          path: /path/to/disk
          size: 100GB
        os: redhat9
        az: azb

When a user submits a build request, they'll need to specify at least the cluster, env, app, and subapp. This can be done using the --extra-vars option in the ansible-playbook command:

ansible-playbook server_build.yml -e "cluster=cl1 env=prod app=gws subapp=gws_api"

This strategy allows you to manage all the variables for each cluster in a single file, while still providing the flexibility to define different settings for each environment, app, and subapp. It also makes it easy to add new clusters, environments, apps, and subapps as your infrastructure grows.

Remember, Ansible applies variables with a certain precedence. In this case, the --extra-vars option always wins, meaning it takes highest precedence, followed by group_vars, and so on.

Example 2: Role Variables & Default Variables - Application Variables

Role-specific variables should be defined in the vars/main.yml file within each role. These will override group_vars set in playbook vars like cl1.yml. Default variables that might need to be overridden should be defined in the defaults/main.yml file within each role.

For example, in roles/ansible_role_app_name/vars/main.yml:

cpu: 8
memory: 4GB
app_var1: value1
app_var2: value2

And in roles/ansible_role_app_name/defaults/main.yml:

app_default1: default_value1
app_default2: default_value2

This structure allows for a clear hierarchy of variable precedence, with role-specific variables taking precedence over group variables, and --extra-vars taking the highest precedence.

Example 3: group_vars sub-directories - Build Variables & Application Variables

Group specific overrides can be set using a sub-directory structure within the group_vars directory. For example, you might have a file group_vars/cl1/dev.yml with:

cpu: 2
memory: 2GB

And a group_vars/cl1/prod.yml file with:

cpu: 4
memory: 4GB

These settings will override group variables set in group_vars/cl1.yml for the dev and prod environments respectively.

With a dynamic inventory, the groups would be defined based on the facts collected by Insights and can be used to group your hosts.

Example 4: Host Variables - Build Variables & Application Variables

For host-specific variables, you should use Red Hat Insights to assign variables to hosts based on their facts or tags. You can define these variables in Insights itself, and they will be included in the inventory that Ansible pulls from Insights. You can also define these variables in the inventory file itself, but this is not recommended as it will not be included in the inventory that Ansible pulls from Insights.

Similar to group_vars/ folders and subfolders, host vars can also be individually defined in either ansible_playbooks/group_vars or ansible_playbooks/host_varsfolders, or directly in the ansible_role_example/host_vars folder.

For example, you might have a file ansible_playbooks/host_vars/host_name.yml with:

cpu: 2
memory: 16GB

Guidelines for where to define variables

Build Variables

Server build variables should be defined in the group_vars directory of the playbook. For example, in ansible_playbooks/group_vars/all.yml.

Application Variables

Application variables should be defined in the group_vars directory of the playbook. For example, in ansible_playbooks/group_vars/all.yml.

Role Variables

Role-specific variables should be defined in the vars/main.yml file within each role. These will override group_vars set in playbook vars like cl1.yml. Default variables that might need to be overridden should be defined in the defaults/main.yml file within each role.

Default Variables

Default variables should be defined in the defaults/main.yml file within each role. These will be overridden by role-specific variables defined in the vars/main.yml file within each role.

Variable Naming Conventions

Naming conventions should be followed for all variables. This will help to ensure consistency and make it easier to find variables when needed.

!!! Required Build variables are required for server provisioning and should be defined in the group_vars directory of the playbook. For example, in ansible_playbooks/group_vars/cl1.yml. More Info

Genesys Build Variables (required):

cluster:
  - cl1
  - cl2
  - cl3

env:
  - prod
  - stage
  - dev
  - test

dc:
  - elr
  - ctc

az:
  - aza
  - azb
  - azt # (optional) For TNT dev only

os:
  - w2019
  - w2016
  - rhel7
  - rhel8
  - rhel9

Privileged Access Management (PAM) shared access group is defined as follows (default - do no change):

server_admin_group: "sre_omni_admin_linux" # (string) We only use this group for shared access

Service Level Owner (SLO) is defined as follows:

slo:
  - mbobbe1
  - rgudmun
  - lsong100
  - rsmi144
  - awyszyn
  - kthornto
  - ygustafs
  - surso
  - sabedi1
  - abarne41

CPU and RAM are defined as follows:

cpu: 4 # (int) default 4 cores
ram: 4 # (int) default 4 GB

Disks are defined as follows:

mount_points:
  - apps: (int) 100 # default 100
  - logs: (int) 100 # default 100
  - monitor: (int) 10 # default 10

Genesys Application Components (GAC) are defined as follows:

app: # (string) required
  db_server: [] # (string)
  dms: []
  chat_server: []
  conf_server: []
  config_proxy: []
  classification: []
  email_server: []
  feature_server: []
  ga: []
  gax: []
  gim: []
  gms: []
  gplus: []
  gvp_rs: []
  icon: []
  lca: []
  lds: []
  msg_server: []
  nuance: []
  pulse: []
  rm: []
  scs: []
  scxml: []
  secure_email: []
  snmp: []
  stat_server: []
  survey: []
  ucs: []
  ucs_web: []
  utility: []
  vxml: []
  cassandra: # (string) (list of strings)
    - gms # (string) subapp name
    - ors
    - cb
    - fs
  elasticsearch:
    - gws
    - ors
  grat:
    - digital
    - rte
    - ivr
  gre:
    - digital
    - rte
    - ivr
  gws:
    - gws_cfg_proxy
    - gws_stat_server
    - gws_api_node
    - gws_api_only
    - gws_index
    - gws_stat_node
    - gws_sync
  ixn:
    - chat
    - email
    - ocs
    - digital
    - gws
  mcp:
    - treat
    - ar
    - conf
  ocs:
    - digital
    - voice
    - ocs
    - con_mgr
  ors:
    - email
    - chat
    - voice
    - digital
  sip:
    - ar
    - vq

app_group:
  - call recording
  - genesys
  - genesys utility
  - nuance
  - optum
  - sip transport
  - undefined

code_automation: false # (bool)
host_name_post_build: [] # (string)
alias: [] # (string)
backup: [] # (string)
static_pod: [] # (string)
pci: [] # (string)

AIDE Friendly Name (Application alias) is defined as follows:

aide: # (string) required, must be unique, optional value is the aide friendly name
  aide_0074720: "omnichannel automation platform"
  uhgwm110-014078: "omnichannel call recording"
  aide_0074027: "omnichannel blue eemaan provisioning application cl1"
  aide_0074028: "omnichannel orange eemaan provisioning application cl2"
  aide_0074572: "omnichannel context data store"
  aide_0074049: "omnichannel orange eemaan platform cl2"
  aide_0074048: "omnichannel blue eemaan platform cl1"
  uhgwm110-017149: "omnichannel cyara"
  uhgwm110-017607: "omnichannel sandbox"
  uhgwm110-017770: "omnichannel blue digital channel cl1"
  uhgwm110-017771: "omnichannel blue core cl1"
  uhgwm110-017772: "omnichannel blue data services cl1"
  uhgwm110-017773: "omnichannel blue outbound dialer cl1"
  uhgwm110-017774: "omnichannel support tools"
  uhgwm110-021394: "omnichannel sbcs and sms"
  uhgwm110-021450: "omnichannel call delivery"
  uhgwm110-021462: "omnichannel blue agent workspace platform cl1"
  uhgwm110-022253: "omnichannel blue routing rules cl1"
  uhgwm110-022353: "omnichannel blue voice portal platform cl1"
  uhgwm110-022404: "omnichannel blue persona routing platform cl1"
  uhgwm110-022428: "omnichannel blue callback cl1"
  uhgwm110-024202: "omnichannel pci"
  uhgwm110-026096: "omnichannel blue service innovation portal platform cl1"
  uhgwm110-026691: "omnichannel transformlayer"
  uhgwm110-027665: "omnichannel blue agent workspace applications cl1"
  uhgwm110-027888: "omnichannel orange data services cl2"
  uhgwm110-027889: "omnichannel orange persona routing platform cl2"
  uhgwm110-027890: "omnichannel orange outbound dialer cl2"
  uhgwm110-027891: "omnichannel orange callback cl2"
  uhgwm110-027892: "omnichannel orange voice portal platform cl2"
  uhgwm110-027893: "omnichannel orange routing rules cl2"
  uhgwm110-027894: "omnichannel orange core cl2"
  uhgwm110-027895: "omnichannel orange agent workspace applications cl2"
  uhgwm110-027896: "omnichannel orange digital channel cl2"
  uhgwm110-027897: "omnichannel orange service innovation portal platform cl2"
  uhgwm110-027901: "omnichannel orange agent workspace platform cl2"
  uhgwm110-028647: "omnichannel blue service innovation portal applications cl1"
  uhgwm110-028648: "omnichannel orange service innovation portal applications cl2"
  uhgwm110-028661: "omnichannel blue persona routing applications cl1"
  uhgwm110-028764: "omnichannel orange persona routing applications cl2"
  aide_0078441: "omnichannel green agent workspace platform cl3"
  aide_0078442: "omnichannel green callback cl"
  aide_0078444: "omnichannel green data services cl3"
  aide_0078443: "omnichannel green core cl3"
  aide_0078445: "omnichannel green digital channel cl3"
  aide_0078446: "omnichannel greem eemaan platform cl3"
  aide_0078447: "omnichannel green outbound dialer cl3"
  aide_0078448: "omnichannel green persona routing platform cl3"
  aide_0078449: "omnichannel green routing rules cl3"
  aide_0078587: "omnichannel green service innovation portal applications cl3"
  aide_0078450: "omnichannel green service innovation portal platform cl3"
  aide_0078451: "omnichannel green voice portal platform cl3"

Server Build Owner (SBO) is defined as follows:

owner:
  - sreadmin # (default)
  - c2sreadp
  - c2sredv
  - c2srestg
  - sreadmdv
  - sreadstg

owner_managed_by_group: "Omni_Server_Mgmt" # (default) (string)

Optional application components are defined as follows:

lca: (bool)
tomcat: (bool)
java_version:
  - 1.8.0_282
  - 1.8.0_341
  - JDK 11

Additional optional fields are defined as follows:

f5: optional (url)
test_f5: optional (test env url)
notes: optional (string)