Merge pull request #6576 from netbox-community/5963-custom-validation

Closes #5963: Custom model validation

docs/additional-features/custom-validation.md

@@ -0,0 +1,86 @@
+# Custom Validation
+
+NetBox validates every object prior to it being written to the database to ensure data integrity. This validation includes things like checking for proper formatting and that references to related objects are valid. However, you may wish to supplement this validation with some rules of your own. For example, perhaps you require that every site's name conforms to a specific pattern.  This can be done using NetBox's `CustomValidator` class.
+
+## CustomValidator
+
+### Validation Rules
+
+A custom validator can be instantiated by passing a mapping of attributes to a set of rules to which that attribute must conform. For example:
+
+```python
+from extras.validators import CustomValidator
+
+CustomValidator({
+    'name': {
+        'min_length': 5,
+        'max_length': 30,
+    }
+})
+```
+
+This defines a custom validator which checks that the length of the `name` attribute for an object is at least five characters long, and no longer than 30 characters. This validation is executed _after_ NetBox has performed its own internal validation.
+
+The `CustomValidator` class supports several validation types:
+
+* `min`: Minimum value
+* `max`: Maximum value
+* `min_length`: Minimum string length
+* `max_length`: Maximum string length
+* `regex`: Application of a [regular expression](https://en.wikipedia.org/wiki/Regular_expression)
+* `required`: A value must be specified
+* `prohibited`: A value must _not_ be specified
+
+The `min` and `max` types should be defined for numeric values, whereas `min_length`, `max_length`, and `regex` are suitable for character strings (text values). The `required` and `prohibited` validators may be used for any field, and should be passed a value of `True`.
+
+!!! warning
+    Bear in mind that these validators merely supplement NetBox's own validation: They will not override it. For example, if a certain model field is required by NetBox, setting a validator for it with `{'prohibited': True}` will not work.
+
+### Custom Validation Logic
+
+There may be instances where the provided validation types are insufficient. The `CustomValidator` class can be extended to enforce arbitrary validation logic by overriding its `validate()` method, and calling `fail()` when an unsatisfactory condition is detected.
+
+```python
+from extras.validators import CustomValidator
+
+class MyValidator(CustomValidator):
+    def validate(self, instance):
+        if instance.status == 'active' and not instance.description:
+            self.fail("Active sites must have a description set!", field='status')
+```
+
+The `fail()` method may optionally specify a field with which to associate the supplied error message. If specified, the error message will appear to the user as associated with this field. If omitted, the error message will not be associated with any field.
+
+## Assigning Custom Validators
+
+Custom validators are associated with specific NetBox models under the [CUSTOM_VALIDATORS](../configuration/optional-settings.md#custom_validators) configuration parameter, as such:
+
+```python
+CUSTOM_VALIDATORS = {
+    'dcim.site': (
+        Validator1,
+        Validator2,
+        Validator3
+    )
+}
+```
+
+!!! note
+    Even if defining only a single validator, it must be passed as an iterable.
+
+When it is not necessary to define a custom `validate()` method, you may opt to pass a `CustomValidator` instance directly:
+
+```python
+from extras.validators import CustomValidator
+
+CUSTOM_VALIDATORS = {
+    'dcim.site': (
+        CustomValidator({
+            'name': {
+                'min_length': 5,
+                'max_length': 30,
+            }
+        }),
+    )
+}
+```

docs/configuration/optional-settings.md

@@ -96,6 +96,12 @@ CORS_ORIGIN_WHITELIST = [
 
 ---
 
+## CUSTOM_VALIDATORS
+
+This is a mapping of models to [custom validators](../additional-features/custom-validation.md) that have been defined locally to enforce custom validation logic.
+
+---
+
 ## DEBUG
 
 Default: False
@@ -144,7 +150,7 @@ In order to send email, NetBox needs an email server configured. The following i
 !!! note
     The `USE_SSL` and `USE_TLS` parameters are mutually exclusive.
 
-Email is sent from NetBox only for critical events or if configured for [logging](#logging). If you would like to test the email server configuration, Django provides a convenient [send_mail()](https://docs.djangoproject.com/en/stable/topics/email/#send-mail) fuction accessible within the NetBox shell:
+Email is sent from NetBox only for critical events or if configured for [logging](#logging). If you would like to test the email server configuration, Django provides a convenient [send_mail()](https://docs.djangoproject.com/en/stable/topics/email/#send-mail) function accessible within the NetBox shell:
 
 ```no-highlight
 # python ./manage.py nbshell

docs/development/signals.md

@@ -0,0 +1,11 @@
+# Signals
+
+In addition to [Django's built-in signals](https://docs.djangoproject.com/en/stable/topics/signals/), NetBox defines some of its own, listed below.
+
+## post_clean
+
+This signal is sent by models which inherit from `CustomValidationMixin` at the end of their `clean()` method.
+
+### Receivers
+
+* `extras.signals.run_custom_validators()`

mkdocs.yml

@@ -64,6 +64,7 @@ nav:
         - Change Logging: 'additional-features/change-logging.md'
         - Context Data: 'models/extras/configcontext.md'
         - Custom Fields: 'additional-features/custom-fields.md'
+        - Custom Validation: 'additional-features/custom-validation.md'
         - Custom Links: 'additional-features/custom-links.md'
         - Custom Scripts: 'additional-features/custom-scripts.md'
         - Export Templates: 'additional-features/export-templates.md'
@@ -90,6 +91,7 @@ nav:
         - Style Guide: 'development/style-guide.md'
         - Models: 'development/models.md'
         - Extending Models: 'development/extending-models.md'
+        - Signals: 'development/signals.md'
         - Application Registry: 'development/application-registry.md'
         - User Preferences: 'development/user-preferences.md'
         - Release Checklist: 'development/release-checklist.md'

netbox/extras/signals.py

@@ -6,10 +6,12 @@ from django.conf import settings
 from django.contrib.contenttypes.models import ContentType
 from django.db import DEFAULT_DB_ALIAS
 from django.db.models.signals import m2m_changed, post_save, pre_delete
+from django.dispatch import receiver
 from django.utils import timezone
 from django_prometheus.models import model_deletes, model_inserts, model_updates
 from prometheus_client import Counter
 
+from netbox.signals import post_clean
 from .choices import ObjectChangeActionChoices
 from .models import CustomField, ObjectChange
 from .webhooks import enqueue_object, get_snapshots, serialize_for_webhook
@@ -136,6 +138,18 @@ post_save.connect(handle_cf_renamed, sender=CustomField)
 pre_delete.connect(handle_cf_deleted, sender=CustomField)
 
 
+#
+# Custom validation
+#
+
+@receiver(post_clean)
+def run_custom_validators(sender, instance, **kwargs):
+    model_name = f'{sender._meta.app_label}.{sender._meta.model_name}'
+    validators = settings.CUSTOM_VALIDATORS.get(model_name, [])
+    for validator in validators:
+        validator(instance)
+
+
 #
 # Caching
 #

netbox/extras/tests/test_customvalidator.py

@@ -0,0 +1,121 @@
+from django.conf import settings
+from django.core.exceptions import ValidationError
+from django.test import TestCase, override_settings
+
+from dcim.models import Site
+from extras.validators import CustomValidator
+
+
+class MyValidator(CustomValidator):
+
+    def validate(self, instance):
+        if instance.name != 'foo':
+            self.fail("Name must be foo!")
+
+
+min_validator = CustomValidator({
+    'asn': {
+        'min': 65000
+    }
+})
+
+
+max_validator = CustomValidator({
+    'asn': {
+        'max': 65100
+    }
+})
+
+
+min_length_validator = CustomValidator({
+    'name': {
+        'min_length': 5
+    }
+})
+
+
+max_length_validator = CustomValidator({
+    'name': {
+        'max_length': 10
+    }
+})
+
+
+regex_validator = CustomValidator({
+    'name': {
+        'regex': r'\d{3}$'  # Ends with three digits
+    }
+})
+
+
+required_validator = CustomValidator({
+    'description': {
+        'required': True
+    }
+})
+
+
+prohibited_validator = CustomValidator({
+    'description': {
+        'prohibited': True
+    }
+})
+
+custom_validator = MyValidator()
+
+
+class CustomValidatorTest(TestCase):
+
+    @override_settings(CUSTOM_VALIDATORS={'dcim.site': [min_validator]})
+    def test_configuration(self):
+        self.assertIn('dcim.site', settings.CUSTOM_VALIDATORS)
+        validator = settings.CUSTOM_VALIDATORS['dcim.site'][0]
+        self.assertIsInstance(validator, CustomValidator)
+
+    @override_settings(CUSTOM_VALIDATORS={'dcim.site': [min_validator]})
+    def test_min(self):
+        with self.assertRaises(ValidationError):
+            Site(name='abcdef123', slug='abcdefghijk', asn=1).clean()
+
+    @override_settings(CUSTOM_VALIDATORS={'dcim.site': [max_validator]})
+    def test_max(self):
+        with self.assertRaises(ValidationError):
+            Site(name='abcdef123', slug='abcdefghijk', asn=65535).clean()
+
+    @override_settings(CUSTOM_VALIDATORS={'dcim.site': [min_length_validator]})
+    def test_min_length(self):
+        with self.assertRaises(ValidationError):
+            Site(name='abc', slug='abc', asn=65000).clean()
+
+    @override_settings(CUSTOM_VALIDATORS={'dcim.site': [max_length_validator]})
+    def test_max_length(self):
+        with self.assertRaises(ValidationError):
+            Site(name='abcdefghijk', slug='abcdefghijk').clean()
+
+    @override_settings(CUSTOM_VALIDATORS={'dcim.site': [regex_validator]})
+    def test_regex(self):
+        with self.assertRaises(ValidationError):
+            Site(name='abcdefgh', slug='abcdefgh').clean()
+
+    @override_settings(CUSTOM_VALIDATORS={'dcim.site': [required_validator]})
+    def test_required(self):
+        with self.assertRaises(ValidationError):
+            Site(name='abcdefgh', slug='abcdefgh', description='').clean()
+
+    @override_settings(CUSTOM_VALIDATORS={'dcim.site': [prohibited_validator]})
+    def test_prohibited(self):
+        with self.assertRaises(ValidationError):
+            Site(name='abcdefgh', slug='abcdefgh', description='ABC').clean()
+
+    @override_settings(CUSTOM_VALIDATORS={'dcim.site': [min_length_validator]})
+    def test_valid(self):
+        Site(name='abcdef123', slug='abcdef123').clean()
+
+    @override_settings(CUSTOM_VALIDATORS={'dcim.site': [custom_validator]})
+    def test_custom_invalid(self):
+        with self.assertRaises(ValidationError):
+            Site(name='abc', slug='abc').clean()
+
+    @override_settings(CUSTOM_VALIDATORS={'dcim.site': [custom_validator]})
+    def test_custom_valid(self):
+        Site(name='foo', slug='foo').clean()

netbox/extras/validators.py

@@ -0,0 +1,107 @@
+from django.core.exceptions import ValidationError
+from django.core import validators
+
+# NOTE: As this module may be imported by configuration.py, we cannot import
+# anything from NetBox itself.
+
+
+class IsEmptyValidator:
+    """
+    Employed by CustomValidator to enforce required fields.
+    """
+    message = "This field must be empty."
+    code = 'is_empty'
+
+    def __init__(self, enforce=True):
+        self._enforce = enforce
+
+    def __call__(self, value):
+        if self._enforce and value not in validators.EMPTY_VALUES:
+            raise ValidationError(self.message, code=self.code)
+
+
+class IsNotEmptyValidator:
+    """
+    Employed by CustomValidator to enforce prohibited fields.
+    """
+    message = "This field must not be empty."
+    code = 'not_empty'
+
+    def __init__(self, enforce=True):
+        self._enforce = enforce
+
+    def __call__(self, value):
+        if self._enforce and value in validators.EMPTY_VALUES:
+            raise ValidationError(self.message, code=self.code)
+
+
+class CustomValidator:
+    """
+    This class enables the application of user-defined validation rules to NetBox models. It can be instantiated by
+    passing a dictionary of validation rules in the form {attribute: rules}, where 'rules' is a dictionary mapping
+    descriptors (e.g. min_length or regex) to values.
+
+    A CustomValidator instance is applied by calling it with the instance being validated:
+
+        validator = CustomValidator({'name': {'min_length: 10}})
+        site = Site(name='abcdef')
+        validator(site)  # Raises ValidationError
+
+    :param validation_rules: A dictionary mapping object attributes to validation rules
+    """
+    VALIDATORS = {
+        'min': validators.MinValueValidator,
+        'max': validators.MaxValueValidator,
+        'min_length': validators.MinLengthValidator,
+        'max_length': validators.MaxLengthValidator,
+        'regex': validators.RegexValidator,
+        'required': IsNotEmptyValidator,
+        'prohibited': IsEmptyValidator,
+    }
+
+    def __init__(self, validation_rules=None):
+        self.validation_rules = validation_rules or {}
+        assert type(self.validation_rules) is dict, "Validation rules must be passed as a dictionary"
+
+    def __call__(self, instance):
+        # Validate instance attributes per validation rules
+        for attr_name, rules in self.validation_rules.items():
+            assert hasattr(instance, attr_name), f"Invalid attribute '{attr_name}' for {instance.__class__.__name__}"
+            attr = getattr(instance, attr_name)
+            for descriptor, value in rules.items():
+                validator = self.get_validator(descriptor, value)
+                try:
+                    validator(attr)
+                except ValidationError as exc:
+                    # Re-package the raised ValidationError to associate it with the specific attr
+                    raise ValidationError({attr_name: exc})
+
+        # Execute custom validation logic (if any)
+        self.validate(instance)
+
+    def get_validator(self, descriptor, value):
+        """
+        Instantiate and return the appropriate validator based on the descriptor given. For
+        example, 'min' returns MinValueValidator(value).
+        """
+        if descriptor not in self.VALIDATORS:
+            raise NotImplementedError(
+                f"Unknown validation type for {self.__class__.__name__}: '{descriptor}'"
+            )
+        validator_cls = self.VALIDATORS.get(descriptor)
+        return validator_cls(value)
+
+    def validate(self, instance):
+        """
+        Custom validation method, to be overridden by the user. Validation failures should
+        raise a ValidationError exception.
+        """
+        return
+
+    def fail(self, message, field=None):
+        """
+        Raise a ValidationError exception. Associate the provided message with a form/serializer field if specified.
+        """
+        if field is not None:
+            raise ValidationError({field: message})
+        raise ValidationError(message)

netbox/netbox/configuration.example.py

@@ -106,6 +106,20 @@ CORS_ORIGIN_REGEX_WHITELIST = [
     # r'^(https?://)?(\w+\.)?example\.com$',
 ]
 
+# Specify any custom validators here, as a mapping of model to a list of validators classes. Validators should be
+# instances of or inherit from CustomValidator.
+# from extras.validators import CustomValidator
+CUSTOM_VALIDATORS = {
+    # 'dcim.site': [
+    #     CustomValidator({
+    #         'name': {
+    #             'min_length': 10,
+    #             'regex': r'\d{3}$',
+    #         }
+    #     })
+    # ],
+}
+
 # Set to True to enable server debugging. WARNING: Debugging introduces a substantial performance penalty and may reveal
 # sensitive information about your installation. Only enable debugging while performing testing. Never enable debugging
 # on a production system.

netbox/netbox/models.py

@@ -9,6 +9,7 @@ from mptt.models import MPTTModel, TreeForeignKey
 from taggit.managers import TaggableManager
 
 from extras.choices import ObjectChangeActionChoices
+from netbox.signals import post_clean
 from utilities.mptt import TreeManager
 from utilities.utils import serialize_object
 
@@ -123,6 +124,20 @@ class CustomFieldsMixin(models.Model):
                 raise ValidationError(f"Missing required custom field '{cf.name}'.")
 
 
+class CustomValidationMixin(models.Model):
+    """
+    Enables user-configured validation rules for built-in models by extending the clean() method.
+    """
+    class Meta:
+        abstract = True
+
+    def clean(self):
+        super().clean()
+
+        # Send the post_clean signal
+        post_clean.send(sender=self.__class__, instance=self)
+
+
 #
 # Base model classes
 
@@ -138,7 +153,7 @@ class BigIDModel(models.Model):
         abstract = True
 
 
-class ChangeLoggedModel(ChangeLoggingMixin, BigIDModel):
+class ChangeLoggedModel(ChangeLoggingMixin, CustomValidationMixin, BigIDModel):
     """
     Base model for all objects which support change logging.
     """
@@ -146,7 +161,7 @@ class ChangeLoggedModel(ChangeLoggingMixin, BigIDModel):
         abstract = True
 
 
-class PrimaryModel(ChangeLoggingMixin, CustomFieldsMixin, BigIDModel):
+class PrimaryModel(ChangeLoggingMixin, CustomFieldsMixin, CustomValidationMixin, BigIDModel):
     """
     Primary models represent real objects within the infrastructure being modeled.
     """
@@ -163,7 +178,7 @@ class PrimaryModel(ChangeLoggingMixin, CustomFieldsMixin, BigIDModel):
         abstract = True
 
 
-class NestedGroupModel(ChangeLoggingMixin, CustomFieldsMixin, BigIDModel, MPTTModel):
+class NestedGroupModel(ChangeLoggingMixin, CustomFieldsMixin, CustomValidationMixin, BigIDModel, MPTTModel):
     """
     Base model for objects which are used to form a hierarchy (regions, locations, etc.). These models nest
     recursively using MPTT. Within each parent, each child instance must have a unique name.
@@ -205,7 +220,7 @@ class NestedGroupModel(ChangeLoggingMixin, CustomFieldsMixin, BigIDModel, MPTTMo
             })
 
 
-class OrganizationalModel(ChangeLoggingMixin, CustomFieldsMixin, BigIDModel):
+class OrganizationalModel(ChangeLoggingMixin, CustomFieldsMixin, CustomValidationMixin, BigIDModel):
     """
     Organizational models are those which are used solely to categorize and qualify other objects, and do not convey
     any real information about the infrastructure being modeled (for example, functional device roles). Organizational

netbox/netbox/settings.py

@@ -74,6 +74,7 @@ CHANGELOG_RETENTION = getattr(configuration, 'CHANGELOG_RETENTION', 90)
 CORS_ORIGIN_ALLOW_ALL = getattr(configuration, 'CORS_ORIGIN_ALLOW_ALL', False)
 CORS_ORIGIN_REGEX_WHITELIST = getattr(configuration, 'CORS_ORIGIN_REGEX_WHITELIST', [])
 CORS_ORIGIN_WHITELIST = getattr(configuration, 'CORS_ORIGIN_WHITELIST', [])
+CUSTOM_VALIDATORS = getattr(configuration, 'CUSTOM_VALIDATORS', {})
 DATE_FORMAT = getattr(configuration, 'DATE_FORMAT', 'N j, Y')
 DATETIME_FORMAT = getattr(configuration, 'DATETIME_FORMAT', 'N j, Y g:i a')
 DEBUG = getattr(configuration, 'DEBUG', False)

netbox/netbox/signals.py

@@ -0,0 +1,5 @@
+from django.dispatch import Signal
+
+
+# Signals that a model has completed its clean() method
+post_clean = Signal()