Administrator
/
Splunk_Docker
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'main'
${ noResults }
Splunk_Docker/New/splunk-ansible/docs/ADVANCED.md

25 KiB
Raw Permalink Blame History Escape

Navigation

Inventory Script

Splunk-Ansible ships with an inventory script in inventory/environ.py. The script gathers user configurations from a local YAML file and/or OS environment variables and converts them into Ansible variables accessible in Ansible tasks.

Supported environment variables

Environment Variable Name Description Required for Standalone Required for Search Head Clustering Required for Index Clustering
SPLUNK_BUILD_URL URL of Splunk build to be installed no no no
SPLUNK_DEFAULTS_URL URL of default.yml no no no
SPLUNK_ALLOW_UPGRADE Perform upgrade if target build version doesn't match installed version no no no
SPLUNK_ROLE The container’s current Splunk Enterprise role. Supported Roles: splunk_standalone, splunk_indexer, splunk_deployer, splunk_search_head, etc. See Roles. no yes yes
SPLUNK_HOSTNAME Specify the instance's hostname. no no no
DEBUG Print Ansible vars to stdout (supports Docker logging) no no no
SPLUNK_START_ARGS Accept the license with —accept-license in this variable. The container will not start without it. yes yes yes
SPLUNK_LICENSE_URI URI we can fetch a Splunk Enterprise license. This can be a local path, a remote URL, or "Free". no no no
SPLUNK_IGNORE_LICENSE Flag to disable any plays that would add a license. Set this env var to "true" if you do not want to license your installation. no no no
SPLUNK_LICENSE_INSTALL_PATH Path on filesystem where Splunk license will be moved/downloaded to no no no
SPLUNK_STANDALONE_URL Comma-separated list of all Splunk Enterprise standalone hosts (network alias) no no no
SPLUNK_SEARCH_HEAD_URL Comma-separated list of all Splunk Enterprise search head hosts (network alias) no yes yes
SPLUNK_INDEXER_URL Comma-separated list of all Splunk Enterprise indexer hosts (network alias) no yes yes
SPLUNK_HEAVY_FORWARDER_URL Comma-separated list of all Splunk Enterprise heavy forwarder hosts (network alias) no no no
SPLUNK_DEPLOYER_URL One Splunk Enterprise deployer host (network alias) no yes no
SPLUNK_CLUSTER_MASTER_URL One Splunk Enterprise cluster master host (network alias) no no yes
SPLUNK_SEARCH_HEAD_CAPTAIN_URL One Splunk Enterprise search head host (network alias). Passing this ENV variable will enable search head clustering. no yes no
SPLUNK_LICENSE_MASTER_URL One Splunk Enterprise license master host (network alias). Passing this ENV variable will enable license master setup. no no no
SPLUNK_DEPLOYMENT_SERVER One Splunk host (network alias) that we use as a deployment server. no no no
SPLUNK_ADD Comma-separated list of items to add to monitoring. Example: SPLUNK_ADD=udp 1514,monitor /var/log/\*. This will monitor the UDP port 1514 and /var/log/\* files. no no no
SPLUNK_BEFORE_START_CMD Comma-separated list of commands to run before Splunk starts. Ansible will run {{splunk.exec}} {{item}}. no no no
SPLUNK_S2S_PORT Splunk forwarding/receiving port. Default: 9997 no no no
SPLUNK_SVC_PORT Splunk management/administration port. Default: 8089 no no no
SPLUNK_CERT_PREFIX HTTP scheme used when making API requests to Splunk management endpoint. Default: https no no no
SPLUNK_ROOT_ENDPOINT Allow SplunkWeb to be accessed behind a given route (ex. reverse proxy usage) no no no
SPLUNK_PASSWORD* Default password of the admin user yes yes yes
SPLUNK_DECLARATVE_ADMIN_PASSWORD When true, admin password will be fixed to the value defined through Ansible no no no
SPLUNK_PASS4SYMMKEY Used to overwrite default pass4SymmKey for Splunk secrets no no no
SPLUNK_HEC_TOKEN HEC (HTTP Event Collector) token when enabled no no no
SPLUNK_SHC_SECRET Search Head Clustering shared secret (deprecated in favor of SPLUNK_SHC_PASS4SYMMKEY) no no no
SPLUNK_SHC_PASS4SYMMKEY Password for the Search Head Clustering shared secret no yes no
SPLUNK_SHC_LABEL Search head clustering label no yes no
SPLUNK_SHC_REPLICATION_FACTOR Configure search head clustering replication factor no no no
SPLUNK_PREFERRED_CAPTAINCY Set up search head clustering with preferred captaincy, typically pinned to the instance designated as splunk_search_head_captain no no no
SPLUNK_IDXC_SECRET Indexer Clustering shared Secret (deprecated in favor of SPLUNK_SHC_PASS4SYMMKEY) no no no
SPLUNK_IDXC_PASS4SYMMKEY Password for the Indexer Clustering shared Secret no no yes
SPLUNK_IDXC_DISCOVERYPASS4SYMMKEY Password for the indexer discovery shared secret no no yes
SPLUNK_IDXC_LABEL Indexer clustering label no no yes
SPLUNK_IDXC_REPLICATION_FACTOR Configure indexer clustering data replication factor no no no
SPLUNK_IDXC_SEARCH_FACTOR Configure indexer clustering search factor no no no
SPLUNK_OPT Location where Splunk Enterprise will be installed (not recommended to change). Default: /opt no no no
SPLUNK_HOME Location of Splunk Enterprise home directory (not recommended to change). Default: /opt/splunk no no no
SPLUNK_EXEC Location of Splunk Enterprise executable (not recommended to change). Default: /opt/splunk/bin/splunk no no no
SPLUNK_PID Location of Splunk Enterprise process ID file (not recommended to change). Default: /opt/splunk/var/run/splunk/splunkd.pid no no no
SPLUNK_USER User which owns the Splunk home directory and runs the splunkd process (not recommended to change). Default: splunk no no no
SPLUNK_GROUP Group which owns the Splunk home directory and runs the splunkd process (not recommended to change). Default: splunk no no no
SPLUNK_PASS4SYMMKEY Set user-defined pass4SymmKey in the general stanza of server.conf instead of using the Splunk default no no no
SPLUNK_SECRET Set user-defined ${SPLUNK_HOME}/etc/splunk.secret instead of using the Splunk default no no no
SPLUNK_ENABLE_SERVICE Enable Splunk to start as a system service (enable boot-start) no no no
SPLUNK_SERVICE_NAME Define Splunk service name when set to start as a system service no no no
SPLUNK_LAUNCH_CONF Pass in a comma-separated list of "key=value" pairs that will get written to ${SPLUNK_HOME}/etc/splunk-launch.conf no no no
SPLUNK_APPS_URL Pass in a comma-separated list of local paths or remote URLs to Splunk apps that will get installed no no no
SPLUNKBASE_USERNAME Splunkbase username used for authentication when installing an app from Splunkbase no no no
SPLUNKBASE_PASSWORD Splunkbase password used for authentication when installing an app from Splunkbase no no no
SPLUNK_HTTP_PORT Port to run SplunkWeb on. To disable SplunkWeb, set to 0. Default: 8000 no no no
SPLUNK_HTTP_ENABLESSL Enable HTTPS on SplunkWeb no no no
SPLUNK_HTTP_ENABLESSL_CERT Path to SSL certificate used for SplunkWeb, if HTTPS is enabled no no no
SPLUNK_HTTP_ENABLESSL_PRIVKEY Path to SSL private key used for SplunkWeb, if HTTPS is enabled no no no
SPLUNK_HTTP_ENABLESSL_PRIVKEY_PASSWORD SSL certificate private key password used with SplunkWeb, if HTTPS is enabled no no no
SPLUNKD_SSL_ENABLE Enable HTTPS on Splunkd. By default, this is enabled out-of-the-box. To disable, set this to "false" no no no
SPLUNKD_SSL_CERT Path to custom SSL certificate used for Splunkd when HTTPS is enabled no no no
SPLUNKD_SSL_CA Path to custom CA certificate used for Splunkd when HTTPS is enabled no no no
SPLUNKD_SSL_PASSWORD Custom SSL password used with Splunkd when HTTPS is enabled no no no
SPLUNK_KVSTORE_PORT Port to run Splunk KVStore. Default: 8191 no no no
SPLUNK_APPSERVER_PORT Port to run Splunk appserver. Default: 8065 no no no
SPLUNK_SET_SEARCH_PEERS Boolean to configure whether search heads should connect to search peers. Default: True. Not recommended to change no no no
SPLUNK_SITE For multisite topologies, define the site of this particular Splunk Enterprise instance no no no
SPLUNK_ALL_SITES For multisite topologies, define all sites of the topology no no no
SPLUNK_MULTISITE_MASTER For multisite topologies, define location of the multisite cluster master no no no
SPLUNK_MULTISITE_MASTER_PORT For multisite topologies, define the Splunk management port of the multisite cluster master no no no
SPLUNK_MULTISITE_REPLICATION_FACTOR_ORIGIN For multisite topologies, define the origin replication factor no no no
SPLUNK_MULTISITE_REPLICATION_FACTOR_TOTAL For multisite topologies, define the total replication factor no no no
SPLUNK_MULTISITE_SEARCH_FACTOR_ORIGIN For multisite topologies, define the origin search factor no no no
SPLUNK_MULTISITE_SEARCH_FACTOR_TOTAL For multisite topologies, define the total search factor no no no
NO_HEALTHCHECK Disable the Splunk health check script no no yes
STEPDOWN_ANSIBLE_USER Removes Ansible user from the sudo group when set to true. This means that no other users than root will have root access. no no no
SPLUNK_HOME_OWNERSHIP_ENFORCEMENT Recursively enforces ${SPLUNK_HOME} to be owned by the user "splunk". Default: True no no no
SPLUNK_DISABLE_POPUPS Disable pop-ups from login on home page and search app. Default: False no no no
HIDE_PASSWORD Hide all Ansible task logs containing Splunk password to secure output to stdout. no no no
JAVA_VERSION Supply "oracle:8", "openjdk:8", or "openjdk:11" to install a respective Java distribution. no no no
JAVA_DOWNLOAD_URL Provide a custom URL where the Java installation will be fetched no no no
SPLUNK_TAIL_FILE Determine which file gets written to the container's stdout (default: splunkd_stderr.log) no no no
SPLUNK_ES_SSL_ENABLEMENT When running Enterprise Security version >= 6.3.0, define ssl_enablement installation option no no no
SPLUNK_DSP_ENABLE Enable DSP forwarding. Default: false no no no
SPLUNK_DSP_SERVER DSP S2S forwarding endpoint no no no
SPLUNK_DSP_CERT DSP certificate used when forwarding no no no
SPLUNK_DSP_VERIFY Enable cert verification when forwarding to DSP. Default: false no no no
SPLUNK_DSP_PIPELINE_NAME Name of DSP pipeline to create/update no no no
SPLUNK_DSP_PIPELINE_DESC Description of DSP pipeline to create/update no no no
SPLUNK_DSP_PIPELINE_SPEC SPL2 specification of DSP pipeline to create/update no no no
SPLUNK_ENABLE_DFS Enable Data Fabric Search (DFS) no no no
SPLUNK_DFW_NUM_SLOTS Maximum number of concurrent DFS searches that run on a search head cluster no no no
SPLUNK_DFC_NUM_SLOTS Maximum number of concurrent DFS searches that run on each search head no no no
SPLUNK_DFW_NUM_SLOTS_ENABLED Enables you to set the value of the field dfw_num_slots. no no no
SPARK_MASTER_HOST This setting identifies the Spark master. no no no
SPARK_MASTER_WEBUI_PORT Identifies the port for the Spark master web UI. no no no
DMC_FORWARDER_MONITORING Enables forwarder monitoring on all standalone and search head instances. Default: False no no no
DMC_ASSET_INTERVAL Cron schedule that determines how often forwarder assets are re-built. Default: "3,18,33,48 * * * *" = every 15 minutes) no no no
SPLUNK_ENABLE_ASAN Internally used trigger to handle special provisioning of debug builds of Splunk Enterprise no no no
SPLUNK_DEFAULTS_URL URL to a remote default.yml file - when fetched, this will get merged into a consolidated mapping of variables no no no
SPLUNK_DEFAULTS_HTTP_MAX_TIMEOUT When fetching a remote default.yml, specify the timeout of the request no no no
SPLUNK_DEFAULTS_HTTP_MAX_RETRIES When fetching a remote default.yml, the number of retries to make. For unlimited retries, use -1 no no no
SPLUNK_DEFAULTS_HTTP_MAX_DELAY When fetching a remote default.yml, specify the delay between each retry no no no
SPLUNK_DEFAULTS_HTTP_AUTH_HEADER When fetching a remote default.yml, set the Authorization header no no no
SPLUNK_ANSIBLE_PRE_TASKS Pass in a comma-separated list of local paths or remote URLs to Ansible playbooks that will be executed before site.yml. Must include the protocol, i.e. it must match the regex ^(http|https|file)://.* no no no
SPLUNK_ANSIBLE_POST_TASKS Pass in a comma-separated list of local paths or remote URLs to Ansible playbooks that will be executed after site.yml. Must include the protocol, i.e. it must match the regex ^(http|https|file)://.* no no no
SPLUNK_ANSIBLE_ENV Pass in a comma-separated list of "key=value" pairs that will be mapped to environment variables used during site.yml execution. These variables are also available in ansible pre/post playbooks and can be referenced as hostvars['localhost'].ansible_environment['key'] no no no
SPLUNK_CONNECTION_TIMEOUT Configures splunkdConnectionTimeout in web.conf with passed integer value (in seconds) no no no
SPLUNK_ES_SSL_ENABLEMENT Set the ssl-enablement flag in ES. Valid values are 'auto', 'strict', and 'ignore'. Defaults to auto when present. no no no

* Password must be set either in default.yml or as the environment variable SPLUNK_PASSWORD

Additional Splunk Universal Forwarder variables

The splunk/universalforwarder image accepts the majority* environment variables as the splunk/splunk image above. However, there are some additional ones that are specific to the Universal Forwarder.

* Note: Specifically for the splunk/universalforwarder image, the environment variable SPLUNK_ROLE will by default be set to splunk_universal_forwarder. This image cannot accept any other role, and should not be changed (unlike its splunk/splunk image counterpart).

Environment Variable Name Description Required for Standalone Required for Search Head Clustering Required for Index Clustering
SPLUNK_CMD Comma-separated list of commands to run after Splunk starts. Ansible will run {{splunk.exec}} {{item}}. no no no

Suggestions for environment variables for Splunk search head cluster

For Splunk search cluster configuration, we suggest passing in the environment variables SPLUNK_HOSTNAME and SPLUNK_SEARCH_HEAD_URL with fully qualified domain names.

The dynamic inventory script will assign the value of SPLUNK_HOSTNAME if defined or socket.getfqdn() to the {{ splunk.hostname }} Ansible variable, which will be used to init search head cluster member. SPLUNK_SEARCH_HEAD_URL will be used as the --server_list argument of search cluster captain bootstrap command, and it requires that each member in --server_list must be exactly the same as the {{ splunk.hostname }} specified earlier.

To be consistent, we suggest passing in the environment variables SPLUNK_SEARCH_HEAD_CAPTAIN_URL, SPLUNK_INDEXER_URL and SPLUNK_DEPLOYER_URL with fully qualified domain names as well.

Defaults

For security purposes, we do not ship with a standard default.yml. However, it is a required component when running these Ansible playbooks in this codebase. This file can be created manually, but for a quick shortcut you can run:

$ docker run -it --rm -e SPLUNK_PASSWORD=helloworld splunk/splunk create-defaults > default.yml

NOTE: The default.yml generated above may require additional, manual modifications.

For distributed Splunk topologies, there are certain configuration settings that are required to be consistent across all members of the deployment. These are settings such as administrator password, clustering labels, keys, etc. To achieve this, it is possible to source all the settings from a single default.yml and distribute it universally in the following manners.

Loading defaults through file

A static default.yml can be dropped on to each instance utilizing the playbooks in this repository. If you're bundling this code in a Docker image or AWS AMI, note that this will be a static configuration, such that Splunk should always spin up in the same way.

This default.yml file must be placed at /tmp/defaults/default.yml, and it must be readable to the user executing the ansible-playbook command. Additionally, it is also possible for certain environment variables to dynamically override settings in this default.yml - for more information on those environment variables, please see the Splunk Docker image project.

In order for the file in /tmp/defaults/default.yml to be read and interpreted, it is required that the Ansible command is executed using the dynamic inventory script (environ.py) as so:

$ ansible-playbook -i inventory/environ.py ...

Loading defaults through URL

If the default.yml file is hosted as a static asset on some webserver, it can be retrieved using an HTTP GET request. The URL must point to a file in a proper YAML format in order for it to be used correctly. Currently, no parameters can be passed through the request. However, redirects are permitted.

To specify a URL to pull a given default.yml, a dummy default.yml can be baked into each instance (as instructed above) to force Ansible to dynamically pull a new one. This placeholder will simply modify the url parameter shown below:

config:
  baked: default.yml
  defaults_dir: /tmp/defaults
  env:
    headers: null
    var: https://webserver/path/to/default.yml
    verify: false
  host:
    headers: null
    url: null
    verify: true
  max_delay: 60
  max_retries: 3
  max_timeout: 1200

This will try to download the default.yml located at https://webserver/path/to/default.yml using the given headers and verify key-word arguments for each request. The download request will timeout after max_timeout seconds, but it will retry a maximum of max_retries attempts with a delay of max_delay in between each attempt.

To use the URL-based loading of a default.yml, it is required that the Ansible command is executed using the dynamic inventory script (environ.py) as so:

$ ansible-playbook -i inventory/environ.py ...

Schema

For more information on the format and options exposed in a default.yml, please see the full spec.

Apps

One of the more versatile features of the Splunk platform is the ability to extend its functionality through the use of apps and add-ons. Apps and add-ons are used heavily by customers by providing out-of-the-box solutions for targeted needs. Many of these products are internally developed and supported by Splunk itself - but an even larger offering is developed by members of the community, partners, and even open-source contributors. Many of these apps can be found on SplunkBase.

For more information on apps and add-ons, please see Splunk's featured apps.

The Ansible playbooks provided in this repository offer the ability to perform fully-vetted app installation. To enable this, modify this section of your default.yml to include a list of URLs or files local to the host:

splunkbase_username: ...
splunkbase_password: ...
splunk:
  ...
  apps_location:
    - /tmp/app.tgz
    - http://webserver.com/path/to/splunkApp.spl
    - https://splunkbase.splunk.com/app/978/release/1.1/download
  ...

Note that in the above, a direct SplunkBase download link was provided. To install an app or add-on directly from SplunkBase, values for splunkbase_username and splunkbase_password must be specified in order to process licensing agreements and authentication. Additionally, the Ansible provisioning must be done using the dynamic inventory script (environ.py) in order to perform the authentication as so:

$ ansible-playbook -i inventory/environ.py ...

If SplunkBase apps are not specified or needed, the splunkbase_username and splunkbase_password variables should be omitted entirely.

When deploying distributed Splunk Enterprise environments, apps should be installed on the deployer, cluster master, and deployment server instances. Each of these roles will take care of bundling and pushing the apps to their respective downstream peers. Note that any configuration files in any custom app's local directory will not be sent to peers - this is in alignment with Splunk best practices around configuration management.

To install an app from elsewhere, provide a path to a compressed splunkApp.spl file (either through a filesystem or URL) as seen above. For proper installation, apps should be compressed using tar in a GNU/Linux environment, as apps compressed on OSX or other BSD-variant operating systems have been known to cause issues.

For roles that manage clusters, such as Cluster Master and SH Deployer, the apps specified in apps_location will be pushed to the cluster and disabled on the local CM or Deployer instance. For apps that need to be installed locally on these instances, use the apps_location_local variable. Apps specified here will not be pushed to the cluster but installed locally in the same way that apps are installed on a standalone instance. Both apps_location and apps_location_local can be used concurrently to allow for different sets of apps to be installed locally vs. pushed to the cluster for CM and Deployer instance. The syntax is the same as apps_location:

splunk:
  ...
  apps_location:
    - /tmp/cluster_app.tgz
  apps_location_local:
    - /tmp/local_app.tgz
    - http://webserver.com/path/to/local_auth_app.spl
  ...

Splunk Apps and Add-on archive files can also be extracted and installed using the app_paths_install: option. This configuration will install a list of apps directly to the configure app_paths directory, bypassing any local install then copy to bundle directory that apps_location uses for cluster managing roles such as CM or Deployer. The suported app_paths are default (local apps), shc, idxc, and deployment. An example of a CM config with one local app and two cluster apps is:

splunk:
  ...
  app_paths_install:
    default:
      - /tmp/local_app.tgz
    idxc:
      - /tmp/cluster_app1.tgz
      - /tmp/cluster_app2.spl
  ...

The shc and idxc apps specified in app_paths_install are not installed locally so Apps that require a local installation prior to a cluster-wide implementation (such as Enterprise Security Suite) would not be support. Those apps would still need to use the apps_location field for proper installation.

SmartStore

SmartStore utilizes S3-compliant object storage in order to store indexed data. This is a capability only available if you're using an indexer cluster (cluster_master + indexers). For more information, please see the blog post as well as technical overview.

Here's an overview of what this looks like if you want to persist all your indexes (default) with a SmartStore backend using the default.yml:

---
splunk:
  smartstore:
    index:
      - indexName: default
        remoteName: remote_store
        scheme: s3
        remoteLocation: <bucket-name>
        s3:
          access_key: <access_key>
          secret_key: <secret_key>
          endpoint: http://s3-us-west-2.amazonaws.com

Custom splunk-launch.conf

splunk-launch.conf is a configuration file that exists in ${SPLUNK_HOME}/etc/ that has some global environment variables that are using by the splunkd process. You can add new variables to this file using either the default.yml or via environment variables.

For instance, if you want to add OPTIMISTIC_ABOUT_FILE_LOCKING=1 to the splunk-launch.conf, you can use this default.yml as reference:

---
splunk:
  launch:
    OPTIMISTIC_ABOUT_FILE_LOCKING: 1
  ...

Multi-cluster Search

See the documentation on how multi-cluster search can be configured.

Powered by BW's shoe-string budget.