Merge pull request #7630 from netbox-community/6238-conditional-webhooks

Closes #6238: Implement conditional webhooks
This commit is contained in:
Jeremy Stretch 2021-10-25 10:42:17 -04:00 committed by GitHub
commit 8276933dbb
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
13 changed files with 515 additions and 18 deletions

View File

@ -17,6 +17,7 @@ A webhook is a mechanism for conveying to some external system a change that too
* **Additional headers** - Any additional headers to include with the request (optional). Add one header per line in the format `Name: Value`. Jinja2 templating is supported for this field (see below).
* **Body template** - The content of the request being sent (optional). Jinja2 templating is supported for this field (see below). If blank, NetBox will populate the request body with a raw dump of the webhook context. (If the HTTP cotent type is set to `application/json`, this will be formatted as a JSON object.)
* **Secret** - A secret string used to prove authenticity of the request (optional). This will append a `X-Hook-Signature` header to the request, consisting of a HMAC (SHA-512) hex digest of the request body using the secret as the key.
* **Conditions** - An optional set of conditions evaluated to determine whether the webhook fires for a given object.
* **SSL verification** - Uncheck this option to disable validation of the receiver's SSL certificate. (Disable with caution!)
* **CA file path** - The file path to a particular certificate authority (CA) file to use when validating the receiver's SSL certificate (optional).
@ -80,3 +81,16 @@ If no body template is specified, the request body will be populated with a JSON
}
}
```
## Conditional Webhooks
A webhook may include a set of conditional logic expressed in JSON used to control whether a webhook triggers for a specific object. For example, you may wish to trigger a webhook for devices only when the `status` field of an object is "active":
```json
{
"attr": "status",
"value": "active"
}
```
For more detail, see the reference documentation for NetBox's [conditional logic](../reference/conditions.md).

View File

@ -0,0 +1,89 @@
# Conditions
Conditions are NetBox's mechanism for evaluating whether a set data meets a prescribed set of conditions. It allows the author to convey simple logic by declaring an arbitrary number of attribute-value-operation tuples nested within a hierarchy of logical AND and OR statements.
## Conditions
A condition is expressed as a JSON object with the following keys:
| Key name | Required | Default | Description |
|----------|----------|---------|-------------|
| attr | Yes | - | Name of the key within the data being evaluated |
| value | Yes | - | The reference value to which the given data will be compared |
| op | No | `eq` | The logical operation to be performed |
| negate | No | False | Negate (invert) the result of the condition's evaluation |
### Available Operations
* `eq`: Equals
* `gt`: Greater than
* `gte`: Greater than or equal to
* `lt`: Less than
* `lte`: Less than or equal to
* `in`: Is present within a list of values
* `contains`: Contains the specified value
### Examples
`name` equals "foobar":
```json
{
"attr": "name",
"value": "foobar"
}
```
`asn` is greater than 65000:
```json
{
"attr": "asn",
"value": 65000,
"op": "gt"
}
```
`status` is not "planned" or "staging":
```json
{
"attr": "status",
"value": ["planned", "staging"],
"op": "in",
"negate": true
}
```
## Condition Sets
Multiple conditions can be combined into nested sets using AND or OR logic. This is done by declaring a JSON object with a single key (`and` or `or`) containing a list of condition objects and/or child condition sets.
### Examples
`status` is "active" and `primary_ip` is defined _or_ the "exempt" tag is applied.
```json
{
"or": [
{
"and": [
{
"attr": "status",
"value": "active"
},
{
"attr": "primary_ip",
"value": "",
"negate": true
}
]
},
{
"attr": "tags",
"value": "exempt",
"op": "contains"
}
]
}
```

View File

@ -28,6 +28,20 @@ Both types of connection include SSID and authentication attributes. Additionall
* Channel - A predefined channel within a standardized band
* Channel frequency & width - Customizable channel attributes (e.g. for licensed bands)
#### Conditional Webhooks ([#6238](https://github.com/netbox-community/netbox/issues/6238))
Webhooks now include a `conditions` field, which may be used to specify conditions under which a webhook triggers. For example, you may wish to generate outgoing requests for a device webhook only when its status is "active" or "staged". This can be done by declaring conditional logic in JSON:
```json
{
"attr": "status",
"op": "in",
"value": ["active", "staged"]
}
```
Multiple conditions may be nested using AND/OR logic as well. For more information, please see the [conditional logic documentation](../reference/conditions.md).
#### Interface Bridging ([#6346](https://github.com/netbox-community/netbox/issues/6346))
A `bridge` field has been added to the interface model for devices and virtual machines. This can be set to reference another interface on the same parent device/VM to indicate a direct layer two bridging adjacency.
@ -85,5 +99,7 @@ Multiple interfaces can be bridged to a single virtual interface to effect a bri
* Added `wwn` field
* dcim.Location
* Added `tenant` field
* extras.Webhook
* Added the `conditions` field
* virtualization.VMInterface
* Added `bridge` field

View File

@ -93,6 +93,8 @@ nav:
- Authentication: 'rest-api/authentication.md'
- GraphQL API:
- Overview: 'graphql-api/overview.md'
- Reference:
- Conditions: 'reference/conditions.md'
- Development:
- Introduction: 'development/index.md'
- Getting Started: 'development/getting-started.md'

View File

@ -61,7 +61,7 @@ class WebhookSerializer(ValidatedModelSerializer):
fields = [
'id', 'url', 'display', 'content_types', 'name', 'type_create', 'type_update', 'type_delete', 'payload_url',
'enabled', 'http_method', 'http_content_type', 'additional_headers', 'body_template', 'secret',
'ssl_verification', 'ca_file_path',
'conditions', 'ssl_verification', 'ca_file_path',
]

144
netbox/extras/conditions.py Normal file
View File

@ -0,0 +1,144 @@
import functools
import re
__all__ = (
'Condition',
'ConditionSet',
)
AND = 'and'
OR = 'or'
def is_ruleset(data):
"""
Determine whether the given dictionary looks like a rule set.
"""
return type(data) is dict and len(data) == 1 and list(data.keys())[0] in (AND, OR)
class Condition:
"""
An individual conditional rule that evaluates a single attribute and its value.
:param attr: The name of the attribute being evaluated
:param value: The value being compared
:param op: The logical operation to use when evaluating the value (default: 'eq')
"""
EQ = 'eq'
GT = 'gt'
GTE = 'gte'
LT = 'lt'
LTE = 'lte'
IN = 'in'
CONTAINS = 'contains'
REGEX = 'regex'
OPERATORS = (
EQ, GT, GTE, LT, LTE, IN, CONTAINS, REGEX
)
TYPES = {
str: (EQ, CONTAINS, REGEX),
bool: (EQ, CONTAINS),
int: (EQ, GT, GTE, LT, LTE, CONTAINS),
float: (EQ, GT, GTE, LT, LTE, CONTAINS),
list: (EQ, IN, CONTAINS)
}
def __init__(self, attr, value, op=EQ, negate=False):
if op not in self.OPERATORS:
raise ValueError(f"Unknown operator: {op}. Must be one of: {', '.join(self.OPERATORS)}")
if type(value) not in self.TYPES:
raise ValueError(f"Unsupported value type: {type(value)}")
if op not in self.TYPES[type(value)]:
raise ValueError(f"Invalid type for {op} operation: {type(value)}")
self.attr = attr
self.value = value
self.eval_func = getattr(self, f'eval_{op}')
self.negate = negate
def eval(self, data):
"""
Evaluate the provided data to determine whether it matches the condition.
"""
value = functools.reduce(dict.get, self.attr.split('.'), data)
result = self.eval_func(value)
if self.negate:
return not result
return result
# Equivalency
def eval_eq(self, value):
return value == self.value
def eval_neq(self, value):
return value != self.value
# Numeric comparisons
def eval_gt(self, value):
return value > self.value
def eval_gte(self, value):
return value >= self.value
def eval_lt(self, value):
return value < self.value
def eval_lte(self, value):
return value <= self.value
# Membership
def eval_in(self, value):
return value in self.value
def eval_contains(self, value):
return self.value in value
# Regular expressions
def eval_regex(self, value):
return re.match(self.value, value) is not None
class ConditionSet:
"""
A set of one or more Condition to be evaluated per the prescribed logic (AND or OR). Example:
{"and": [
{"attr": "foo", "op": "eq", "value": 1},
{"attr": "bar", "op": "eq", "value": 2, "negate": true}
]}
:param ruleset: A dictionary mapping a logical operator to a list of conditional rules
"""
def __init__(self, ruleset):
if type(ruleset) is not dict:
raise ValueError(f"Ruleset must be a dictionary, not {type(ruleset)}.")
if len(ruleset) != 1:
raise ValueError(f"Ruleset must have exactly one logical operator (found {len(ruleset)})")
# Determine the logic type
logic = list(ruleset.keys())[0]
if type(logic) is not str or logic.lower() not in (AND, OR):
raise ValueError(f"Invalid logic type: {logic} (must be '{AND}' or '{OR}')")
self.logic = logic.lower()
# Compile the set of Conditions
self.conditions = [
ConditionSet(rule) if is_ruleset(rule) else Condition(**rule)
for rule in ruleset[self.logic]
]
def eval(self, data):
"""
Evaluate the provided data to determine whether it matches this set of conditions.
"""
func = any if self.logic == 'or' else all
return func(d.eval(data) for d in self.conditions)

View File

@ -137,7 +137,7 @@ class WebhookBulkEditForm(BootstrapMixin, BulkEditForm):
)
class Meta:
nullable_fields = ['secret', 'ca_file_path']
nullable_fields = ['secret', 'conditions', 'ca_file_path']
class TagBulkEditForm(BootstrapMixin, BulkEditForm):

View File

@ -102,6 +102,7 @@ class WebhookForm(BootstrapMixin, forms.ModelForm):
('HTTP Request', (
'payload_url', 'http_method', 'http_content_type', 'additional_headers', 'body_template', 'secret',
)),
('Conditions', ('conditions',)),
('SSL', ('ssl_verification', 'ca_file_path')),
)
widgets = {

View File

@ -0,0 +1,18 @@
# Generated by Django 3.2.8 on 2021-10-22 20:37
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('extras', '0062_clear_secrets_changelog'),
]
operations = [
migrations.AddField(
model_name='webhook',
name='conditions',
field=models.JSONField(blank=True, null=True),
),
]

View File

@ -9,11 +9,12 @@ from django.db import models
from django.http import HttpResponse
from django.urls import reverse
from django.utils import timezone
from django.utils.formats import date_format, time_format
from django.utils.formats import date_format
from rest_framework.utils.encoders import JSONEncoder
from extras.choices import *
from extras.constants import *
from extras.conditions import ConditionSet
from extras.utils import extras_features, FeatureQuery, image_upload
from netbox.models import BigIDModel, ChangeLoggedModel
from utilities.querysets import RestrictedQuerySet
@ -107,6 +108,11 @@ class Webhook(ChangeLoggedModel):
"the secret as the key. The secret is not transmitted in "
"the request."
)
conditions = models.JSONField(
blank=True,
null=True,
help_text="A set of conditions which determine whether the webhook will be generated."
)
ssl_verification = models.BooleanField(
default=True,
verbose_name='SSL verification',
@ -138,9 +144,13 @@ class Webhook(ChangeLoggedModel):
# At least one action type must be selected
if not self.type_create and not self.type_delete and not self.type_update:
raise ValidationError(
"You must select at least one type: create, update, and/or delete."
)
raise ValidationError("At least one type must be selected: create, update, and/or delete.")
if self.conditions:
try:
ConditionSet(self.conditions)
except ValueError as e:
raise ValidationError({'conditions': e})
# CA file path requires SSL verification enabled
if not self.ssl_verification and self.ca_file_path:

View File

@ -0,0 +1,199 @@
from django.test import TestCase
from extras.conditions import Condition, ConditionSet
class ConditionTestCase(TestCase):
def test_dotted_path_access(self):
c = Condition('a.b.c', 1, 'eq')
self.assertTrue(c.eval({'a': {'b': {'c': 1}}}))
self.assertFalse(c.eval({'a': {'b': {'c': 2}}}))
self.assertFalse(c.eval({'a': {'b': {'x': 1}}}))
def test_undefined_attr(self):
c = Condition('x', 1, 'eq')
self.assertFalse(c.eval({}))
self.assertTrue(c.eval({'x': 1}))
#
# Validation tests
#
def test_invalid_op(self):
with self.assertRaises(ValueError):
# 'blah' is not a valid operator
Condition('x', 1, 'blah')
def test_invalid_type(self):
with self.assertRaises(ValueError):
# dict type is unsupported
Condition('x', 1, dict())
def test_invalid_op_type(self):
with self.assertRaises(ValueError):
# 'gt' supports only numeric values
Condition('x', 'foo', 'gt')
#
# Operator tests
#
def test_default_operator(self):
c = Condition('x', 1)
self.assertEqual(c.eval_func, c.eval_eq)
def test_eq(self):
c = Condition('x', 1, 'eq')
self.assertTrue(c.eval({'x': 1}))
self.assertFalse(c.eval({'x': 2}))
def test_eq_negated(self):
c = Condition('x', 1, 'eq', negate=True)
self.assertFalse(c.eval({'x': 1}))
self.assertTrue(c.eval({'x': 2}))
def test_gt(self):
c = Condition('x', 1, 'gt')
self.assertTrue(c.eval({'x': 2}))
self.assertFalse(c.eval({'x': 1}))
def test_gte(self):
c = Condition('x', 1, 'gte')
self.assertTrue(c.eval({'x': 2}))
self.assertTrue(c.eval({'x': 1}))
self.assertFalse(c.eval({'x': 0}))
def test_lt(self):
c = Condition('x', 2, 'lt')
self.assertTrue(c.eval({'x': 1}))
self.assertFalse(c.eval({'x': 2}))
def test_lte(self):
c = Condition('x', 2, 'lte')
self.assertTrue(c.eval({'x': 1}))
self.assertTrue(c.eval({'x': 2}))
self.assertFalse(c.eval({'x': 3}))
def test_in(self):
c = Condition('x', [1, 2, 3], 'in')
self.assertTrue(c.eval({'x': 1}))
self.assertFalse(c.eval({'x': 9}))
def test_in_negated(self):
c = Condition('x', [1, 2, 3], 'in', negate=True)
self.assertFalse(c.eval({'x': 1}))
self.assertTrue(c.eval({'x': 9}))
def test_contains(self):
c = Condition('x', 1, 'contains')
self.assertTrue(c.eval({'x': [1, 2, 3]}))
self.assertFalse(c.eval({'x': [2, 3, 4]}))
def test_contains_negated(self):
c = Condition('x', 1, 'contains', negate=True)
self.assertFalse(c.eval({'x': [1, 2, 3]}))
self.assertTrue(c.eval({'x': [2, 3, 4]}))
def test_regex(self):
c = Condition('x', '[a-z]+', 'regex')
self.assertTrue(c.eval({'x': 'abc'}))
self.assertFalse(c.eval({'x': '123'}))
def test_regex_negated(self):
c = Condition('x', '[a-z]+', 'regex', negate=True)
self.assertFalse(c.eval({'x': 'abc'}))
self.assertTrue(c.eval({'x': '123'}))
class ConditionSetTest(TestCase):
def test_empty(self):
with self.assertRaises(ValueError):
ConditionSet({})
def test_invalid_logic(self):
with self.assertRaises(ValueError):
ConditionSet({'foo': []})
def test_and_single_depth(self):
cs = ConditionSet({
'and': [
{'attr': 'a', 'value': 1, 'op': 'eq'},
{'attr': 'b', 'value': 1, 'op': 'eq', 'negate': True},
]
})
self.assertTrue(cs.eval({'a': 1, 'b': 2}))
self.assertFalse(cs.eval({'a': 1, 'b': 1}))
def test_or_single_depth(self):
cs = ConditionSet({
'or': [
{'attr': 'a', 'value': 1, 'op': 'eq'},
{'attr': 'b', 'value': 1, 'op': 'eq'},
]
})
self.assertTrue(cs.eval({'a': 1, 'b': 2}))
self.assertTrue(cs.eval({'a': 2, 'b': 1}))
self.assertFalse(cs.eval({'a': 2, 'b': 2}))
def test_and_multi_depth(self):
cs = ConditionSet({
'and': [
{'attr': 'a', 'value': 1, 'op': 'eq'},
{'and': [
{'attr': 'b', 'value': 2, 'op': 'eq'},
{'attr': 'c', 'value': 3, 'op': 'eq'},
]}
]
})
self.assertTrue(cs.eval({'a': 1, 'b': 2, 'c': 3}))
self.assertFalse(cs.eval({'a': 9, 'b': 2, 'c': 3}))
self.assertFalse(cs.eval({'a': 1, 'b': 9, 'c': 3}))
self.assertFalse(cs.eval({'a': 1, 'b': 2, 'c': 9}))
def test_or_multi_depth(self):
cs = ConditionSet({
'or': [
{'attr': 'a', 'value': 1, 'op': 'eq'},
{'or': [
{'attr': 'b', 'value': 2, 'op': 'eq'},
{'attr': 'c', 'value': 3, 'op': 'eq'},
]}
]
})
self.assertTrue(cs.eval({'a': 1, 'b': 9, 'c': 9}))
self.assertTrue(cs.eval({'a': 9, 'b': 2, 'c': 9}))
self.assertTrue(cs.eval({'a': 9, 'b': 9, 'c': 3}))
self.assertFalse(cs.eval({'a': 9, 'b': 9, 'c': 9}))
def test_mixed_and(self):
cs = ConditionSet({
'and': [
{'attr': 'a', 'value': 1, 'op': 'eq'},
{'or': [
{'attr': 'b', 'value': 2, 'op': 'eq'},
{'attr': 'c', 'value': 3, 'op': 'eq'},
]}
]
})
self.assertTrue(cs.eval({'a': 1, 'b': 2, 'c': 9}))
self.assertTrue(cs.eval({'a': 1, 'b': 9, 'c': 3}))
self.assertFalse(cs.eval({'a': 1, 'b': 9, 'c': 9}))
self.assertFalse(cs.eval({'a': 9, 'b': 2, 'c': 3}))
def test_mixed_or(self):
cs = ConditionSet({
'or': [
{'attr': 'a', 'value': 1, 'op': 'eq'},
{'and': [
{'attr': 'b', 'value': 2, 'op': 'eq'},
{'attr': 'c', 'value': 3, 'op': 'eq'},
]}
]
})
self.assertTrue(cs.eval({'a': 1, 'b': 9, 'c': 9}))
self.assertTrue(cs.eval({'a': 9, 'b': 2, 'c': 3}))
self.assertTrue(cs.eval({'a': 1, 'b': 2, 'c': 9}))
self.assertFalse(cs.eval({'a': 9, 'b': 2, 'c': 9}))
self.assertFalse(cs.eval({'a': 9, 'b': 9, 'c': 3}))

View File

@ -145,6 +145,7 @@ class WebhookTestCase(ViewTestCases.PrimaryObjectViewTestCase):
'payload_url': 'http://example.com/?x',
'http_method': 'GET',
'http_content_type': 'application/foo',
'conditions': None,
}
cls.csv_data = (

View File

@ -6,6 +6,7 @@ from django_rq import job
from jinja2.exceptions import TemplateError
from .choices import ObjectChangeActionChoices
from .conditions import ConditionSet
from .webhooks import generate_signature
logger = logging.getLogger('netbox.webhooks_worker')
@ -16,6 +17,12 @@ def process_webhook(webhook, model_name, event, data, snapshots, timestamp, user
"""
Make a POST request to the defined Webhook
"""
# Evaluate webhook conditions (if any)
if webhook.conditions:
if not ConditionSet(webhook.conditions).eval(data):
return
# Prepare context data for headers & body templates
context = {
'event': dict(ObjectChangeActionChoices)[event].lower(),
'timestamp': timestamp,
@ -33,14 +40,14 @@ def process_webhook(webhook, model_name, event, data, snapshots, timestamp, user
try:
headers.update(webhook.render_headers(context))
except (TemplateError, ValueError) as e:
logger.error("Error parsing HTTP headers for webhook {}: {}".format(webhook, e))
logger.error(f"Error parsing HTTP headers for webhook {webhook}: {e}")
raise e
# Render the request body
try:
body = webhook.render_body(context)
except TemplateError as e:
logger.error("Error rendering request body for webhook {}: {}".format(webhook, e))
logger.error(f"Error rendering request body for webhook {webhook}: {e}")
raise e
# Prepare the HTTP request
@ -51,15 +58,13 @@ def process_webhook(webhook, model_name, event, data, snapshots, timestamp, user
'data': body.encode('utf8'),
}
logger.info(
"Sending {} request to {} ({} {})".format(
params['method'], params['url'], context['model'], context['event']
)
f"Sending {params['method']} request to {params['url']} ({context['model']} {context['event']})"
)
logger.debug(params)
try:
prepared_request = requests.Request(**params).prepare()
except requests.exceptions.RequestException as e:
logger.error("Error forming HTTP request: {}".format(e))
logger.error(f"Error forming HTTP request: {e}")
raise e
# If a secret key is defined, sign the request with a hash of the key and its content
@ -74,12 +79,10 @@ def process_webhook(webhook, model_name, event, data, snapshots, timestamp, user
response = session.send(prepared_request, proxies=settings.HTTP_PROXIES)
if 200 <= response.status_code <= 299:
logger.info("Request succeeded; response status {}".format(response.status_code))
return 'Status {} returned, webhook successfully processed.'.format(response.status_code)
logger.info(f"Request succeeded; response status {response.status_code}")
return f"Status {response.status_code} returned, webhook successfully processed."
else:
logger.warning("Request failed; response status {}: {}".format(response.status_code, response.content))
logger.warning(f"Request failed; response status {response.status_code}: {response.content}")
raise requests.exceptions.RequestException(
"Status {} returned with content '{}', webhook FAILED to process.".format(
response.status_code, response.content
)
f"Status {response.status_code} returned with content '{response.content}', webhook FAILED to process."
)