Test Playbooks

We use Test Playbooks to test our integrations and automation scripts. The Test Playbooks provide full End to End testing. For testing small units of code, use Unit Testing.

Tests are run using our CI framework. They are run both as part of the build process and on a nightly basis. If you are a contributor, during the initial PR, Test Playbooks will not run but they will be used as part of the review process. Once the PR is merged into a contrib/* branch by one of the team members and credentials (if needed) are provided to Demisto, the test playbooks will be run as part of our CI framework.

Note: Test Playbooks are required as part of the PR acceptance review process. For simple Scripts that have unit tests, a test playbook is optional.

Create a Test Playbook

A Test Playbook consists of several steps, including testing commands, verifying the results, and closing the investigation.

We use a standard naming convention for our playbook tests which follows the format below:

Integration_Name-Test

Create a playbook

  1. Navigate to Playbooks and click New Playbook.
  2. Define a Playbook name.

Add DeleteContext

When creating a Test Playbook, it is often recommended for the first step to be DeleteContext. "Delete Context" does just that, it deletes all of the context data. While not always vital, it ensures that a test playbook has a clean beginning to test from without conflicting data. Especially, useful while developing and you wish to re-run the playbook. This allows for a test to be "sterile" and can help us to eliminate unrelated issues from the test.

  1. In the search field, type deletecontext and click Utilities.
  2. In the DeleteContext task, click Add.
  3. From the dropdown menu in the all field, select yes.
  4. Click OK and connect the Playbook Triggered task to the DeleteContext task.

Testing a Command, Verifying the Results, and Closing an Investigation

It's important to test as many commands of the integration as possible as tasks, and each command should have a task. For this example we will look at the integration IPInfo, which accepts only one command called !ip.

Test a Command

  1. Navigate to Playbooks and click New Playbook.
  2. In the search field, type ipinfo and click ip.
  3. In the ip task, click Add to edit the configuration options.
  4. Select an entity that will produce the most consistent results in the ip field, such as 8.8.8.8, the Google DNS server.
  5. Click OK to save your changes.
  6. Finally, connect the starting DeleteContext task to the ip task.

connect delete and ip

Verify the Command Results

After you build the command, verify that you have received the results that you expect:

  1. Open the Task Library and select Create Task.
  2. Configure the new task.
ConfigurationAction
ConditionalSelect the Conditional button to display the condition options.
Task NameType a task name.
From previous tasksClick {} to display the Select source for tool. The Select source for tool displays the #2 ip task that you created.
2 ipClick to display the ip task configurations.
IPClick Address and click Close. The IP.Address is displayed in the From previous tasks field. This is the Context Path.
From previous tasksWrap the Context Path using this format ${IP.Address}. Wrapping the Context Path tells Demisto to retrieve the value located in the curly brackets.
As valueType 8.8.8.8 and click ✅.

Note: If you need to edit the value in a field, you can click on the value and edit it. For example, click on the value in the From previous tasks field and edit the ${IP.Address} value.

  1. Optional: If you need to filter or format the result, click Filters and Operations located in the Select source for dialog box.
  2. Click Save.
  3. Connect the ip task to the Verify Command Results task.

Close the Investigation

  1. Navigate to Playbooks and click New Playbook.
  2. In the search field, type closeinvestigation and click BuiltIn Commands.
  3. For closeInvestigation, click Add.
  4. Click the {} in the id field.
  5. Click Incident details and find ID. ${incident.id} is inserted into the id field.
  6. Click Close and click OK.
  7. Connect the Verify Command Results task to the closeInvestigation task.
  8. Choose the yes label name for the condition and click Save.

Naming and Exporting the Playbook

Demisto uses a standard naming convention for playbook tests that follows this format: Integration_Name-Test.

  1. Click Save Version.
  2. Exit the playbook editor.
  3. Export the playbook by clicking download button.

Adding the Playbook to your Project

  1. In the YAML file that you created, edit the id so that it is identical to the name field.
  2. Modify the value in the version field to -1 to prevent user changes.
  3. Using the example above, the top of your YAML should look like this:
id: IPInfo-Test
version: -1
name: IPInfo-Test

Adding Tests to conf.json

In order to associate integrations with a test plabyook we mange a conf.json file (at the root of the repository). The conf.json file is located in the Tests directory.

The following is an example of a correct conf.json entry for an integration:

{
"integrations": "Forcepoint",
"playbookID": "forcepoint test",
"timeout": 500,
"nightly": true
},

The following table describes the fields:

NameDescription
integrationsThe ID of the integration that you are testing.
playbookIDThe ID of the test playbook that you are running.
timeoutThe time in seconds to extend the timeout to (optional).
nightlyBoolean that indicates if the test should be part of only the nightly tests (optional).

Resources

Last updated on