Fluent Bit Log Collection

Overview

Fluent Bit is a lightweight, high-performance log processor and forwarder used to collect logs from Linux systems—including RHEL 9 and other supported distributions—and send them to Azure Event Hub for ingestion into Splunk. It provides efficient log collection with built-in buffering, filtering, and metadata enrichment capabilities.

Key Features

• Low resource footprint: Minimal CPU and memory usage
• Persistent buffering: Filesystem-backed storage prevents log loss during network issues
• Dynamic configuration: Automatically monitors configured log paths
• Metadata enrichment: Adds sourcetype and source fields for Splunk routing
• Multiple input sources: Collects from systemd journal and file-based logs

Architecture

Log Sources → Fluent Bit → Azure Event Hub → Splunk

Fluent Bit runs as a systemd service under the svc_eoa_fluentbit service account, which requires read access to monitored log directories and files via POSIX ACLs.

---

Metadata Mapping for Splunk

Fluent Bit enriches log events with metadata fields that map to Splunk index-time fields:

How Metadata is Added

The configuration template (99-eoa-fluentbit.conf.j2) uses FILTER blocks to add these fields:

[FILTER]
    Name              modify
    Match             {{ category_name }}
    Copy              data.log_file_path agg
    Add               type {{ category_config.type }}

• type field: Set from linux_fluentbit_log_access_paths.<category>.type variable
• agg field: Copied from the actual log file path (log_file_path) captured by Fluent Bit

These fields are nested under data.* in the JSON payload sent to Azure Event Hub, where they are extracted and mapped to Splunk sourcetype and source during indexing.

---

Log Collection Configuration

All log categories are defined in defaults/main/linux.yml under the linux_fluentbit_log_access_paths variable. Each category configures:

• Paths: Directories to monitor
• Patterns: Filename patterns to match
• Permissions: ACL permissions for directories and files
• Recursion: Whether to monitor subdirectories
• Type: Splunk sourcetype identifier

---

Adding New Log Sources

Scenario 1: Logs in a Single Parent Directory

Use Case: Monitor specific log files that exist only in the parent directory, not in subdirectories. This is useful when you want to monitor select files in a high-volume directory without monitoring all files or subdirectories.

Example: System logs in /var/log/ (messages and dnf.log only, not subdirectories)

linux_fluentbit_log_access_paths:
  system:
    type: linux:system
    paths:
      - /var/log
    patterns:
      - "messages"
      - "dnf.log"
    permissions: rx
    file_permissions: r
    recursive: false # Do NOT search subdirectories
    default_acl: false # Don't set default ACL on /var/log itself
    apply_to_existing_files: true

Generated Fluent Bit Paths:

• /var/log/messages
• /var/log/dnf.log

Key Settings:

• recursive: false - Only matches files directly in /var/log/, not in subdirectories
• patterns: ["messages", "dnf.log"] - Exact filename matches
• default_acl: false - Prevents setting default ACL on the high-volume /var/log directory

Matches:

• ✅ /var/log/messages
• ✅ /var/log/dnf.log
• ❌ /var/log/audit/audit.log (in subdirectory)
• ❌ /var/log/boot.log (not in patterns list)

---

Scenario 2: Logs in All Subdirectories (Excluding Parent)

Use Case: Monitor logs in subdirectories at any depth, but NOT in the parent directory itself. This is ideal when all log files are organized in subdirectories (e.g., one subdirectory per extension or service).

Example: Azure extension logs at /var/log/azure/Microsoft.*/*/logs/*.log

linux_fluentbit_log_access_paths:
  azure:
    type: linux:azure
    paths:
      - /var/log/azure
    patterns:
      - "*.log"
    permissions: rx
    file_permissions: r
    recursive: true # Search ALL subdirectories recursively
    default_acl: true
    apply_to_existing_files: true

Generated Fluent Bit Path: /var/log/azure/**/*.log

Key Settings:

• recursive: true - Generates /** wildcard for recursive matching at any depth
• patterns: ["*.log"] - Matches any .log file
• default_acl: true - Ensures new subdirectories (new extensions) automatically inherit ACLs

Important: The /** pattern requires at least one subdirectory level. It will NOT match files directly in /var/log/azure/.

Matches:

• ✅ /var/log/azure/Microsoft.CPlat.Core.LinuxPatchExtension/Events/2024-11-07.log
• ✅ /var/log/azure/Microsoft.Azure.Monitor.AzureMonitorLinuxAgent/1.29.5/handler.log
• ✅ /var/log/azure/Microsoft.Azure.Extensions.CustomScript/status/0.status
• ❌ /var/log/azure/direct.log (no subdirectory between parent and file)

---

Scenario 3: Logs in All Subdirectories AND Parent Directory

Use Case: Monitor logs at any depth, including the parent directory itself. This is necessary when log files may appear both in the parent directory AND in subdirectories.

Example: Azure logs that may exist in /var/log/azure/extension.log AND /var/log/azure/Microsoft.*/handler.log

Solution: Define two separate categories - one for the parent, one for subdirectories.

linux_fluentbit_log_access_paths:
  # Category 1: Parent directory only
  azure_parent:
    type: linux:azure
    paths:
      - /var/log/azure
    patterns:
      - "*.log"
    permissions: rx
    file_permissions: r
    recursive: false # Parent directory only
    default_acl: true # Ensures new subdirectories inherit ACLs
    apply_to_existing_files: true

  # Category 2: All subdirectories recursively
  azure_subdirs:
    type: linux:azure
    paths:
      - /var/log/azure
    patterns:
      - "*.log"
    permissions: rx
    file_permissions: r
    recursive: true # All subdirectories
    default_acl: false # Already handled by parent category
    apply_to_existing_files: true

Generated Fluent Bit Paths:

• /var/log/azure/*.log (from azure_parent)
• /var/log/azure/**/*.log (from azure_subdirs)

Why Two Categories?:

• /** pattern does NOT match direct children - requires at least one subdirectory level
• Separate non-recursive category ensures parent-level files are monitored
• default_acl: true on parent ensures new subdirectories (new extensions) inherit permissions

Matches:

• ✅ /var/log/azure/extension.log (from azure_parent)
• ✅ /var/log/azure/Microsoft.CPlat.Core.LinuxPatchExtension/handler.log (from azure_subdirs)
• ✅ /var/log/azure/Microsoft.Azure.Monitor/1.29.5/error.log (from azure_subdirs)

---

Scenario 4: Logs in Select Subdirectories

Use Case: Monitor logs only in specific named subdirectories, not all subdirectories. This gives you fine-grained control when you only want to monitor certain extensions or services.

Example: Only monitor Azure Monitor and Custom Script extensions, but not other Azure extensions.

Solution: Include a parent directory category with a non-matching pattern to set ACLs for directory traversal, then list each subdirectory explicitly.

linux_fluentbit_log_access_paths:
  # Parent directory - ACL traversal only, no actual log monitoring
  azure_parent_acl:
    type: linux:azure
    paths:
      - /var/log/azure
    patterns:
      - "" # Empty pattern - will never match any files
    permissions: rx # Allows svc_eoa_fluentbit to traverse into subdirectories
    file_permissions: r
    recursive: false
    default_acl: true # Future subdirectories inherit ACL
    apply_to_existing_files: false # No existing files to apply ACL to

  azure_monitor:
    type: linux:azure:monitor
    paths:
      - /var/log/azure/Microsoft.Azure.Monitor.AzureMonitorLinuxAgent
    patterns:
      - "*.log"
    permissions: rx
    file_permissions: r
    recursive: true # Monitor all subdirectories within this extension
    default_acl: true
    apply_to_existing_files: true

  azure_customscript:
    type: linux:azure:customscript
    paths:
      - /var/log/azure/Microsoft.Azure.Extensions.CustomScript
    patterns:
      - "*.log"
    permissions: rx
    file_permissions: r
    recursive: true # Monitor all subdirectories within this extension
    default_acl: true
    apply_to_existing_files: true

Generated Fluent Bit Paths:

• /var/log/azure/ (empty pattern generates path without glob - never matches)
• /var/log/azure/Microsoft.Azure.Monitor.AzureMonitorLinuxAgent/**/*.log
• /var/log/azure/Microsoft.Azure.Extensions.CustomScript/**/*.log

Key Settings:

• Parent category with empty pattern: Sets ACL on /var/log/azure for directory traversal without monitoring files
• Empty pattern ("") - Won't match any files, safe for Fluent Bit glob parser
• apply_to_existing_files: false - Skips unnecessary file ACL application on parent (no files match)
• Each subdirectory is a separate category with its own type (allows different Splunk sourcetypes)
• recursive: true on subdirectories - Monitors all subdirectories within each extension (e.g., version folders)

Why the Parent Category is Required:

Without ACL on /var/log/azure, the svc_eoa_fluentbit user cannot traverse into subdirectories like Microsoft.Azure.Monitor.AzureMonitorLinuxAgent, even if those subdirectories have correct ACLs. The rx permission on the parent directory is mandatory for filesystem traversal.

Matches:

• ✅ /var/log/azure/Microsoft.Azure.Monitor.AzureMonitorLinuxAgent/1.29.5/handler.log
• ✅ /var/log/azure/Microsoft.Azure.Extensions.CustomScript/status/0.status
• ❌ /var/log/azure/extension.log (parent directory, blocked by non-matching pattern)
• ❌ /var/log/azure/Microsoft.CPlat.Core.LinuxPatchExtension/handler.log (not in monitored extensions)

---

Scenario 5: Logs in Select Subdirectories AND Parent Directory

Use Case: Monitor specific subdirectories AND the parent directory, but not all subdirectories.

Example: Monitor /var/log/app/, /var/log/app/frontend/, and /var/log/app/backend/, but not /var/log/app/other/.

Solution: Define separate categories for parent and each subdirectory.

linux_fluentbit_log_access_paths:
  # Parent directory
  app_root:
    type: linux:app
    paths:
      - /var/log/app
    patterns:
      - "app.log"
      - "error.log"
    permissions: rx
    file_permissions: r
    recursive: false # Only parent directory
    default_acl: true # Ensures subdirectories inherit ACL
    apply_to_existing_files: true

  # Frontend subdirectory
  app_frontend:
    type: linux:app:frontend
    paths:
      - /var/log/app/frontend
    patterns:
      - "*.log"
    permissions: rx
    file_permissions: r
    recursive: false # Only this specific subdirectory
    default_acl: true
    apply_to_existing_files: true

  # Backend subdirectory
  app_backend:
    type: linux:app:backend
    paths:
      - /var/log/app/backend
    patterns:
      - "*.log"
    permissions: rx
    file_permissions: r
    recursive: false # Only this specific subdirectory
    default_acl: true
    apply_to_existing_files: true

Generated Fluent Bit Paths:

• /var/log/app/app.log (from app_root)
• /var/log/app/error.log (from app_root)
• /var/log/app/frontend/*.log (from app_frontend)
• /var/log/app/backend/*.log (from app_backend)

Benefits:

• Granular control over which subdirectories are monitored
• Different sourcetypes per subdirectory for Splunk routing
• Parent directory ACL inheritance ensures new subdirectories can be added easily

---

Configuration Reference

Variable Structure

linux_fluentbit_log_access_paths:
  <category_name>:
    type: <string> # Splunk sourcetype (e.g., linux:audit)
    paths: [<string>, ...] # List of directories to monitor
    patterns: [<string>, ...] # Filename patterns (glob syntax)
    permissions: <string> # Directory ACL permissions (e.g., rx)
    file_permissions: <string> # File ACL permissions (e.g., r)
    recursive: <boolean> # true = /**/, false = /
    default_acl: <boolean> # Set default ACL for inheritance
    apply_to_existing_files: <boolean> # Apply ACL to existing files

Field Descriptions

Pattern Syntax

Fluent Bit supports standard glob patterns:

• * - Matches any characters (zero or more)
• ? - Matches exactly one character
• [abc] - Matches any character in the set
• ** - Recursive wildcard (matches subdirectories at any depth)

Examples:

• audit.log - Exact filename match
• *.log - Any file ending in .log
• app_*.log - Files starting with app_ and ending in .log
• sssd_[a-z]*.log - SSSD logs starting with lowercase letters

---

ACL Permissions Explained

Why ACLs are Required

Fluent Bit runs as the svc_eoa_fluentbit user, which needs read access to log files. Since log files are typically owned by root or other service accounts, POSIX ACLs grant read-only access without changing file ownership.

Permission Types

1. Directory Access ACL (permissions: rx):

   • Read (r): List directory contents
   • Execute (x): Traverse the directory (required to access files inside)

2. File ACL (file_permissions: r):

   • Read (r): Read file contents (execute bit NOT needed for files)

3. Default ACL (default_acl: true):

   • Applied to directories only
   • New files/subdirectories inherit this ACL automatically
   • Uses file_permissions (not permissions) to avoid execute bit on files

ACL Inheritance

When default_acl: true:

/var/log/audit/          # Access ACL: rx, Default ACL: r
├── audit.log            # Inherits: r (from parent default ACL)
└── subdir/              # Inherits: r (from parent default ACL)
    └── file.log         # Inherits: r (from subdir's default ACL)

---

Troubleshooting

Logs Not Appearing in Splunk

1. Check Fluent Bit service status:

   sudo systemctl status fluent-bit
   sudo journalctl -u fluent-bit -n 50
   

2. Verify path existence:

   ls -la /var/log/audit/audit.log
   

3. Check ACL permissions:

   getfacl /var/log/audit/audit.log
   # Should show: user:svc_eoa_fluentbit:r--
   

4. Test pattern matching:

   # For recursive: false
   find /var/log/audit -maxdepth 1 -name "audit.log"
   
   # For recursive: true
   find /var/log/azure -type f -name "*.log"
   

5. Check Fluent Bit configuration:

   sudo cat /etc/fluent-bit/99-eoa-fluentbit.conf
   # Verify INPUT blocks have correct Path directives
   

Permission Denied Errors

Symptom: journalctl -u fluent-bit shows "Permission denied" errors

Solution:

1. Verify ACL is set:

   getfacl /var/log/audit/audit.log
   

2. If missing, re-run Ansible playbook in AWX via the following template:

   Install Fluentbit Logger
   

Recursive Pattern Not Matching

Symptom: recursive: true not matching expected files

Common Issue: /** pattern requires at least one subdirectory level.

Example:

• /var/log/azure/**/file.log matches /var/log/azure/subdir/file.log ✅
• /var/log/azure/**/file.log does NOT match /var/log/azure/file.log ❌

Solution: Use two categories (one with recursive: false for parent, one with recursive: true for subdirectories).

---

Best Practices

1. Use descriptive category names: audit, sssd, app_frontend (not logs1, logs2)
2. Set default_acl: true on parent directories to ensure new subdirectories inherit ACLs
3. Use recursive: false for parent directories to avoid missing direct children
4. Group related logs by sourcetype: One category per Splunk sourcetype for easier routing
5. Test patterns locally with find commands before deploying
6. Monitor Fluent Bit logs after deployment to verify successful ingestion

---

Examples from Production

Audit Logs (Direct Children Only)

audit:
  type: linux:audit
  paths: [/var/log/audit]
  patterns: ["audit.log"]
  permissions: rx
  file_permissions: r
  recursive: false # audit.log is a direct child
  default_acl: true
  apply_to_existing_files: true

SSSD Logs (Multiple Patterns, Direct Children)

sssd:
  type: linux:sssd
  paths: [/var/log/sssd]
  patterns: ["sssd.log", "sssd_*.log"]
  permissions: rx
  file_permissions: r
  recursive: false # All SSSD logs are direct children
  default_acl: true
  apply_to_existing_files: true

System Logs (Specific Files in High-Volume Directory)

system:
  type: linux:system
  paths: [/var/log]
  patterns: ["messages", "dnf.log"]
  permissions: rx
  file_permissions: r
  recursive: false # Only these specific files, not all /var/log
  default_acl: false # Don't set default ACL on /var/log
  apply_to_existing_files: true

---

Related Documentation

• Fluent Bit Official Documentation
• Azure Event Hub Integration
• POSIX ACLs on RHEL 9