diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 7f3d0935a..a026878bb 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -9,8 +9,7 @@ IF YOUR PULL REQUEST DOES NOT REFERENCE AN ACCEPTED BUG REPORT OR FEATURE REQUEST, IT WILL BE MARKED AS INVALID AND CLOSED. --> -### Fixes: - +### Fixes: diff --git a/.github/stale.yaml b/.github/stale.yaml new file mode 100644 index 000000000..7c8d03f12 --- /dev/null +++ b/.github/stale.yaml @@ -0,0 +1,23 @@ +# Number of days of inactivity before an issue becomes stale +daysUntilStale: 14 +# Number of days of inactivity before a stale issue is closed +daysUntilClose: 7 +# Issues with these labels will never be considered stale +exemptLabels: + - "status: accepted" + - "status: gathering feedback" + - "status: blocked" +# Label to use when marking an issue as stale +staleLabel: wontfix +# Comment to post when marking an issue as stale. Set to `false` to disable +markComment: > + This issue has been automatically marked as stale because it has not had + recent activity. It will be closed if no further activity occurs. NetBox + is governed by a small group of core maintainers which means not all opened + issues may receive direct feedback. Please see our [contributing guide](https://github.com/netbox-community/netbox/blob/develop/CONTRIBUTING.md). +# Comment to post when closing a stale issue. Set to `false` to disable +closeComment: > + This issue has been automatically closed due to lack of activity. In an + effort to reduce noise, please do not comment any further. Note that the + core maintainers may elect to reopen this issue at a later date if deemed + necessary. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a688be9b3..55a979eef 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -24,7 +24,7 @@ already been fixed. to see if the bug you've found has already been reported. If you think you may be experiencing a reported issue that hasn't already been resolved, please click "add a reaction" in the top right corner of the issue and add a thumbs -up (+1). You mightalso want to add a comment describing how it's affecting your +up (+1). You might also want to add a comment describing how it's affecting your installation. This will allow us to prioritize bugs based on how many users are affected. @@ -99,6 +99,8 @@ any work that's already in progress. * Any pull request which does _not_ relate to an accepted issue will be closed. +* All major new functionality must include relevant tests where applicable. + * When submitting a pull request, please be sure to work off of the `develop` branch, rather than `master`. The `develop` branch is used for ongoing development, while `master` is used for tagging new stable releases. @@ -118,6 +120,30 @@ feedback. **Do not** comment on an issue just to show your support (give the top post a :+1: instead) or ask for an ETA. These comments will be deleted to reduce noise in the discussion. +## Issue Lifecycle + +When a correctly formatted issue is submitted it is evaluated by a moderator +who may elect to immediately label the issue as accepted in addition to another +issue type label. In other cases, the issue may be labeled as "status: gathering feedback" +which will often be accompanied by a comment from a moderator asking for further dialog from the community. +If an issue is labeled as "status: revisions needed" a moderator has identified a problem with +the issue itself and is asking for the submitter himself to update the original post with +the requested information. If the original post is not updated in a reasonable amount of time, +the issue will be closed as invalid. + +The core maintainers group has chosen to make use of the GitHub Stale bot to aid in issue management. + +* Issues will be marked as stale after 14 days of no activity. +* Then after 7 more days of inactivity, the issue will be closed. +* Any issue bearing one of the following labels will be exempt from all Stale bot actions: + * `status: accepted` + * `status: gathering feedback` + * `status: blocked` + +It is natural that some new issues get more attention than others. Often this is a metric of an issues's +overall usefulness to the project. In other cases in which issues merely get lost in the shuffle, +notifications from Stale bot can bring renewed attention to potentially meaningful issues. + ## Maintainer Guidance * Maintainers are expected to contribute at least four hours per week to the diff --git a/README.md b/README.md index e915c8662..996f26332 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,8 @@ NetBox is an IP address management (IPAM) and data center infrastructure management (DCIM) tool. Initially conceived by the network engineering team at [DigitalOcean](https://www.digitalocean.com/), NetBox was developed specifically -to address the needs of network and infrastructure engineers. +to address the needs of network and infrastructure engineers. It is intended to +function as a domain-specific source of truth for network operations. NetBox runs as a web application atop the [Django](https://www.djangoproject.com/) Python framework with a [PostgreSQL](http://www.postgresql.org/) database. For a @@ -42,6 +43,15 @@ and run `upgrade.sh`. * [Ansible deployment](https://github.com/lae/ansible-role-netbox) (via [@lae](https://github.com/lae)) * [Kubernetes deployment](https://github.com/CENGN/netbox-kubernetes) (via [@CENGN](https://github.com/CENGN)) +# Providing Feedback + +Feature requests and bug reports must be submitted as GiHub issues. (Please be +sure to use the [appropriate template](https://github.com/netbox-community/netbox/issues/new/choose).) +For general discussion, please consider joining our [mailing list](https://groups.google.com/forum/#!forum/netbox-discuss). + +If you are interested in contributing to the development of NetBox, please read +our [contributing guide](CONTRIBUTING.md) prior to beginning any work. + # Related projects Please see [our wiki](https://github.com/netbox-community/netbox/wiki/Community-Contributions) for a list of relevant community projects. diff --git a/docs/additional-features/custom-scripts.md b/docs/additional-features/custom-scripts.md index c00f54d4c..8d453f668 100644 --- a/docs/additional-features/custom-scripts.md +++ b/docs/additional-features/custom-scripts.md @@ -119,6 +119,23 @@ Stored a numeric integer. Options include: A true/false flag. This field has no options beyond the defaults. +### ChoiceVar + +A set of choices from which the user can select one. + +* `choices` - A list of `(value, label)` tuples representing the available choices. For example: + +```python +CHOICES = ( + ('n', 'North'), + ('s', 'South'), + ('e', 'East'), + ('w', 'West') +) + +direction = ChoiceVar(choices=CHOICES) +``` + ### ObjectVar A NetBox object. The list of available objects is defined by the queryset parameter. Each instance of this variable is limited to a single object type. diff --git a/docs/additional-features/export-templates.md b/docs/additional-features/export-templates.md index 41bc11a6e..541858a88 100644 --- a/docs/additional-features/export-templates.md +++ b/docs/additional-features/export-templates.md @@ -4,7 +4,7 @@ NetBox allows users to define custom templates that can be used when exporting o Each export template is associated with a certain type of object. For instance, if you create an export template for VLANs, your custom template will appear under the "Export" button on the VLANs list. -Export templates are written in [Django's template language](https://docs.djangoproject.com/en/1.9/ref/templates/language/), which is very similar to Jinja2. The list of objects returned from the database is stored in the `queryset` variable, which you'll typically want to iterate through using a `for` loop. Object properties can be access by name. For example: +Export templates are written in [Django's template language](https://docs.djangoproject.com/en/stable/ref/templates/language/), which is very similar to Jinja2. The list of objects returned from the database is stored in the `queryset` variable, which you'll typically want to iterate through using a `for` loop. Object properties can be access by name. For example: ``` {% for rack in queryset %} diff --git a/docs/additional-features/webhooks.md b/docs/additional-features/webhooks.md index 0e74640fa..9a02449f8 100644 --- a/docs/additional-features/webhooks.md +++ b/docs/additional-features/webhooks.md @@ -11,8 +11,10 @@ The webhook POST request is structured as so (assuming `application/json` as the ```no-highlight { "event": "created", - "signal_received_timestamp": 1508769597, - "model": "Site" + "timestamp": "2019-10-12 12:51:29.746944", + "username": "admin", + "model": "site", + "request_id": "43d8e212-94c7-4f67-b544-0dcde4fc0f43", "data": { ... } @@ -24,8 +26,10 @@ The webhook POST request is structured as so (assuming `application/json` as the ```no-highlight { "event": "deleted", - "signal_received_timestamp": 1508781858.544069, - "model": "Site", + "timestamp": "2019-10-12 12:55:44.030750", + "username": "johnsmith", + "model": "site", + "request_id": "e9bb83b2-ebe4-4346-b13f-07144b1a00b4", "data": { "asn": None, "comments": "", diff --git a/docs/administration/netbox-shell.md b/docs/administration/netbox-shell.md index 2ebea5ce5..2d0b8f307 100644 --- a/docs/administration/netbox-shell.md +++ b/docs/administration/netbox-shell.md @@ -4,7 +4,7 @@ NetBox includes a Python shell within which objects can be directly queried, cre ./manage.py nbshell ``` -This will launch a customized version of [the built-in Django shell](https://docs.djangoproject.com/en/dev/ref/django-admin/#shell) with all relevant NetBox models pre-loaded. (If desired, the stock Django shell is also available by executing `./manage.py shell`.) +This will launch a customized version of [the built-in Django shell](https://docs.djangoproject.com/en/stable/ref/django-admin/#shell) with all relevant NetBox models pre-loaded. (If desired, the stock Django shell is also available by executing `./manage.py shell`.) ``` $ ./manage.py nbshell @@ -28,7 +28,7 @@ DCIM: ## Querying Objects -Objects are retrieved by forming a [Django queryset](https://docs.djangoproject.com/en/dev/topics/db/queries/#retrieving-objects). The base queryset for an object takes the form `.objects.all()`, which will return a (truncated) list of all objects of that type. +Objects are retrieved by forming a [Django queryset](https://docs.djangoproject.com/en/stable/topics/db/queries/#retrieving-objects). The base queryset for an object takes the form `.objects.all()`, which will return a (truncated) list of all objects of that type. ``` >>> Device.objects.all() @@ -99,7 +99,7 @@ This approach can span multiple levels of relations. For example, the following ``` !!! note - While the above query is functional, it is very inefficient. There are ways to optimize such requests, however they are out of the scope of this document. For more information, see the [Django queryset method reference](https://docs.djangoproject.com/en/dev/ref/models/querysets/) documentation. + While the above query is functional, it is very inefficient. There are ways to optimize such requests, however they are out of the scope of this document. For more information, see the [Django queryset method reference](https://docs.djangoproject.com/en/stable/ref/models/querysets/) documentation. Reverse relationships can be traversed as well. For example, the following will find all devices with an interface named "em0": @@ -137,7 +137,7 @@ To return the inverse of a filtered queryset, use `exclude()` instead of `filter ``` !!! info - The examples above are intended only to provide a cursory introduction to queryset filtering. For an exhaustive list of the available filters, please consult the [Django queryset API docs](https://docs.djangoproject.com/en/dev/ref/models/querysets/). + The examples above are intended only to provide a cursory introduction to queryset filtering. For an exhaustive list of the available filters, please consult the [Django queryset API docs](https://docs.djangoproject.com/en/stable/ref/models/querysets/). ## Creating and Updating Objects diff --git a/docs/administration/replicating-netbox.md b/docs/administration/replicating-netbox.md index 6dd686594..781b3d214 100644 --- a/docs/administration/replicating-netbox.md +++ b/docs/administration/replicating-netbox.md @@ -39,6 +39,11 @@ If you want to export only the database schema, and not the data itself (e.g. fo ```no-highlight pg_dump -s netbox > netbox_schema.sql ``` +If you are migrating your instance of NetBox to a different machine, please make sure you invalidate the cache by performing this command: + +```no-highlight +python3 manage.py invalidate all +``` --- diff --git a/docs/configuration/optional-settings.md b/docs/configuration/optional-settings.md index af00fe2d2..89532d4b7 100644 --- a/docs/configuration/optional-settings.md +++ b/docs/configuration/optional-settings.md @@ -139,7 +139,7 @@ Enforcement of unique IP space can be toggled on a per-VRF basis. To enforce uni By default, all messages of INFO severity or higher will be logged to the console. Additionally, if `DEBUG` is False and email access has been configured, ERROR and CRITICAL messages will be emailed to the users defined in `ADMINS`. -The Django framework on which NetBox runs allows for the customization of logging, e.g. to write logs to file. Please consult the [Django logging documentation](https://docs.djangoproject.com/en/1.11/topics/logging/) for more information on configuring this setting. Below is an example which will write all INFO and higher messages to a file: +The Django framework on which NetBox runs allows for the customization of logging, e.g. to write logs to file. Please consult the [Django logging documentation](https://docs.djangoproject.com/en/stable/topics/logging/) for more information on configuring this setting. Below is an example which will write all INFO and higher messages to a file: ``` LOGGING = { @@ -311,7 +311,7 @@ Enable this option to run the webhook backend. See the docs section on the webho ## Date and Time Formatting -You may define custom formatting for date and times. For detailed instructions on writing format strings, please see [the Django documentation](https://docs.djangoproject.com/en/dev/ref/templates/builtins/#date). +You may define custom formatting for date and times. For detailed instructions on writing format strings, please see [the Django documentation](https://docs.djangoproject.com/en/stable/ref/templates/builtins/#date). Defaults: diff --git a/docs/configuration/required-settings.md b/docs/configuration/required-settings.md index e4f2aed97..92b2fbfb8 100644 --- a/docs/configuration/required-settings.md +++ b/docs/configuration/required-settings.md @@ -2,7 +2,7 @@ ## ALLOWED_HOSTS -This is a list of valid fully-qualified domain names (FQDNs) that is used to reach the NetBox service. Usually this is the same as the hostname for the NetBox server, but can also be different (e.g. when using a reverse proxy serving the NetBox website under a different FQDN than the hostname of the NetBox server). NetBox will not permit access to the server via any other hostnames (or IPs). The value of this option is also used to set `CSRF_TRUSTED_ORIGINS`, which restricts `HTTP POST` to the same set of hosts (more about this [here](https://docs.djangoproject.com/en/1.9/ref/settings/#std:setting-CSRF_TRUSTED_ORIGINS)). Keep in mind that NetBox, by default, has `USE_X_FORWARDED_HOST = True` (in `netbox/netbox/settings.py`) which means that if you're using a reverse proxy, it's the FQDN used to reach that reverse proxy which needs to be in this list (more about this [here](https://docs.djangoproject.com/en/1.9/ref/settings/#allowed-hosts)). +This is a list of valid fully-qualified domain names (FQDNs) that is used to reach the NetBox service. Usually this is the same as the hostname for the NetBox server, but can also be different (e.g. when using a reverse proxy serving the NetBox website under a different FQDN than the hostname of the NetBox server). NetBox will not permit access to the server via any other hostnames (or IPs). The value of this option is also used to set `CSRF_TRUSTED_ORIGINS`, which restricts `HTTP POST` to the same set of hosts (more about this [here](https://docs.djangoproject.com/en/stable/ref/settings/#std:setting-CSRF_TRUSTED_ORIGINS)). Keep in mind that NetBox, by default, has `USE_X_FORWARDED_HOST = True` (in `netbox/netbox/settings.py`) which means that if you're using a reverse proxy, it's the FQDN used to reach that reverse proxy which needs to be in this list (more about this [here](https://docs.djangoproject.com/en/stable/ref/settings/#allowed-hosts)). Example: @@ -21,6 +21,7 @@ NetBox requires access to a PostgreSQL database service to store data. This serv * `PASSWORD` - PostgreSQL password * `HOST` - Name or IP address of the database server (use `localhost` if running locally) * `PORT` - TCP port of the PostgreSQL service; leave blank for default port (5432) +* `CONN_MAX_AGE` - Number in seconds for Netbox to keep database connections open. 150-300 seconds is typically a good starting point ([more info](https://docs.djangoproject.com/en/stable/ref/databases/#persistent-connections)). Example: @@ -31,6 +32,7 @@ DATABASE = { 'PASSWORD': 'J5brHrAXFLQSif0K', # PostgreSQL password 'HOST': 'localhost', # Database server 'PORT': '', # Database port (leave blank for default) + 'CONN_MAX_AGE': 300, # Max database connection age } ``` @@ -69,7 +71,7 @@ REDIS = { !!! note: If you were using these settings in a prior release with webhooks, the `DATABASE` setting remains the same but an additional `CACHE_DATABASE` setting has been added with a default value of 1 to support the caching backend. The - `DATABASE` setting will be renamed in a future release of NetBox to better relay the meaning of the setting. + `DATABASE` setting will be renamed in a future release of NetBox to better relay the meaning of the setting. !!! warning: It is highly recommended to keep the webhook and cache databases seperate. Using the same database number for both may result in webhook diff --git a/docs/development/style-guide.md b/docs/development/style-guide.md index 138d0e12d..4c490eebf 100644 --- a/docs/development/style-guide.md +++ b/docs/development/style-guide.md @@ -1,6 +1,6 @@ # Style Guide -NetBox generally follows the [Django style guide](https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/coding-style/), which is itself based on [PEP 8](https://www.python.org/dev/peps/pep-0008/). [Pycodestyle](https://github.com/pycqa/pycodestyle) is used to validate code formatting, ignoring certain violations. See `scripts/cibuild.sh`. +NetBox generally follows the [Django style guide](https://docs.djangoproject.com/en/stable/internals/contributing/writing-code/coding-style/), which is itself based on [PEP 8](https://www.python.org/dev/peps/pep-0008/). [Pycodestyle](https://github.com/pycqa/pycodestyle) is used to validate code formatting, ignoring certain violations. See `scripts/cibuild.sh`. ## PEP 8 Exceptions diff --git a/docs/installation/2-netbox.md b/docs/installation/2-netbox.md index cd7aea171..7bae23d77 100644 --- a/docs/installation/2-netbox.md +++ b/docs/installation/2-netbox.md @@ -129,6 +129,7 @@ DATABASE = { 'PASSWORD': 'J5brHrAXFLQSif0K', # PostgreSQL password 'HOST': 'localhost', # Database server 'PORT': '', # Database port (leave blank for default) + 'CONN_MAX_AGE': 300, # Max database connection age } ``` diff --git a/docs/installation/3-http-daemon.md b/docs/installation/3-http-daemon.md index dcf16101e..c1bcf7ca8 100644 --- a/docs/installation/3-http-daemon.md +++ b/docs/installation/3-http-daemon.md @@ -108,7 +108,7 @@ Install gunicorn: # pip3 install gunicorn ``` -Save the following configuration in the root netbox installation path as `gunicorn_config.py` (e.g. `/opt/netbox/gunicorn_config.py` per our example installation). Be sure to verify the location of the gunicorn executable on your server (e.g. `which gunicorn`) and to update the `pythonpath` variable if needed. If using CentOS/RHEL, change the username from `www-data` to `nginx` or `apache`. +Save the following configuration in the root netbox installation path as `gunicorn_config.py` (e.g. `/opt/netbox/gunicorn_config.py` per our example installation). Be sure to verify the location of the gunicorn executable on your server (e.g. `which gunicorn`) and to update the `pythonpath` variable if needed. If using CentOS/RHEL, change the username from `www-data` to `nginx` or `apache`. More info on `max_requests` can be found in the [gunicorn docs](https://docs.gunicorn.org/en/stable/settings.html#max-requests). ```no-highlight command = '/usr/bin/gunicorn' @@ -116,6 +116,8 @@ pythonpath = '/opt/netbox/netbox' bind = '127.0.0.1:8001' workers = 3 user = 'www-data' +max_requests = 5000 +max_requests_jitter = 500 ``` # supervisord Installation diff --git a/docs/release-notes/version-2.6.md b/docs/release-notes/version-2.6.md index 2b9b3b765..181d448ff 100644 --- a/docs/release-notes/version-2.6.md +++ b/docs/release-notes/version-2.6.md @@ -1,3 +1,27 @@ +# v2.6.7 (2019-11-01) + +## Enhancements + +* [#3445](https://github.com/netbox-community/netbox/issues/3445) - Add support for additional user defined headers to be added to webhook requests +* [#3499](https://github.com/netbox-community/netbox/issues/3499) - Add `ca_file_path` to Webhook model to support user supplied CA certificate verification of webhook requests +* [#3594](https://github.com/netbox-community/netbox/issues/3594) - Add ChoiceVar for custom scripts +* [#3619](https://github.com/netbox-community/netbox/issues/3619) - Add 400GE OSFP interface type +* [#3659](https://github.com/netbox-community/netbox/issues/3659) - Add filtering for objects in admin UI + +## Bug Fixes + +* [#3309](https://github.com/netbox-community/netbox/issues/3309) - Rewrite change logging middleware to resolve sporadic testing failures +* [#3340](https://github.com/netbox-community/netbox/issues/3340) - Add missing options to connect front ports to console ports +* [#3357](https://github.com/netbox-community/netbox/issues/3357) - Enable filter sites/devices/VMs by null region +* [#3460](https://github.com/netbox-community/netbox/issues/3460) - Extend upgrade script to validate Python dependencies +* [#3596](https://github.com/netbox-community/netbox/issues/3596) - Prevent server error when reassigning a device to a new device bay +* [#3629](https://github.com/netbox-community/netbox/issues/3629) - Use `get_lldp_neighors_detail` to validation LLDP neighbors +* [#3635](https://github.com/netbox-community/netbox/issues/3635) - Add missing cache support for the circuits app +* [#3636](https://github.com/netbox-community/netbox/issues/3636) - Add missing `rack_group` field to PowerFeed CSV export +* [#3652](https://github.com/netbox-community/netbox/issues/3652) - Limit next/previous rack by assigned rack group + +--- + # v2.6.6 (2019-10-10) ## Notes diff --git a/netbox/dcim/constants.py b/netbox/dcim/constants.py index 840d55d6b..034911a26 100644 --- a/netbox/dcim/constants.py +++ b/netbox/dcim/constants.py @@ -86,6 +86,7 @@ IFACE_TYPE_100GE_QSFP28 = 1600 IFACE_TYPE_200GE_CFP2 = 1650 IFACE_TYPE_200GE_QSFP56 = 1700 IFACE_TYPE_400GE_QSFP_DD = 1750 +IFACE_TYPE_400GE_OSFP = 1800 # Wireless IFACE_TYPE_80211A = 2600 IFACE_TYPE_80211G = 2610 @@ -180,6 +181,7 @@ IFACE_TYPE_CHOICES = [ [IFACE_TYPE_100GE_QSFP28, 'QSFP28 (100GE)'], [IFACE_TYPE_200GE_QSFP56, 'QSFP56 (200GE)'], [IFACE_TYPE_400GE_QSFP_DD, 'QSFP-DD (400GE)'], + [IFACE_TYPE_400GE_OSFP, 'OSFP (400GE)'], ] ], [ diff --git a/netbox/dcim/migrations/0075_cable_devices.py b/netbox/dcim/migrations/0075_cable_devices.py index 45a94389c..f022fe111 100644 --- a/netbox/dcim/migrations/0075_cable_devices.py +++ b/netbox/dcim/migrations/0075_cable_devices.py @@ -1,3 +1,5 @@ +import sys + from django.db import migrations, models import django.db.models.deletion @@ -5,14 +7,15 @@ import django.db.models.deletion def cache_cable_devices(apps, schema_editor): Cable = apps.get_model('dcim', 'Cable') - print("\nUpdatng cable device terminations...") + if 'test' not in sys.argv: + print("\nUpdating cable device terminations...") cable_count = Cable.objects.count() # Cache A/B termination devices on all existing Cables. Note that the custom save() method on Cable is not # available during a migration, so we replicate its logic here. for i, cable in enumerate(Cable.objects.all(), start=1): - if not i % 1000: + if not i % 1000 and 'test' not in sys.argv: print("[{}/{}]".format(i, cable_count)) termination_a_model = apps.get_model(cable.termination_a_type.app_label, cable.termination_a_type.model) diff --git a/netbox/dcim/models.py b/netbox/dcim/models.py index 03f560a59..45b241d42 100644 --- a/netbox/dcim/models.py +++ b/netbox/dcim/models.py @@ -2588,6 +2588,16 @@ class DeviceBay(ComponentModel): if self.device == self.installed_device: raise ValidationError("Cannot install a device into itself.") + # Check that the installed device is not already installed elsewhere + if self.installed_device: + current_bay = DeviceBay.objects.filter(installed_device=self.installed_device).first() + if current_bay: + raise ValidationError({ + 'installed_device': "Cannot install the specified device; device is already installed in {}".format( + current_bay + ) + }) + # # Inventory items @@ -3112,6 +3122,7 @@ class PowerFeed(ChangeLoggedModel, CableTermination, CustomFieldModel): return ( self.power_panel.site.name, self.power_panel.name, + self.rack.group.name if self.rack and self.rack.group else None, self.rack.name if self.rack else None, self.name, self.get_status_display(), diff --git a/netbox/dcim/views.py b/netbox/dcim/views.py index 58c759822..959e1043e 100644 --- a/netbox/dcim/views.py +++ b/netbox/dcim/views.py @@ -404,8 +404,12 @@ class RackView(PermissionRequiredMixin, View): position__isnull=True, parent_bay__isnull=True ).prefetch_related('device_type__manufacturer') - next_rack = Rack.objects.filter(site=rack.site, name__gt=rack.name).order_by('name').first() - prev_rack = Rack.objects.filter(site=rack.site, name__lt=rack.name).order_by('-name').first() + if rack.group: + peer_racks = Rack.objects.filter(site=rack.site, group=rack.group) + else: + peer_racks = Rack.objects.filter(site=rack.site, group__isnull=True) + next_rack = peer_racks.filter(name__gt=rack.name).order_by('name').first() + prev_rack = peer_racks.filter(name__lt=rack.name).order_by('-name').first() reservations = RackReservation.objects.filter(rack=rack) power_feeds = PowerFeed.objects.filter(rack=rack).prefetch_related('power_panel') diff --git a/netbox/extras/admin.py b/netbox/extras/admin.py index 1c35be600..f99848b1b 100644 --- a/netbox/extras/admin.py +++ b/netbox/extras/admin.py @@ -40,6 +40,9 @@ class WebhookAdmin(admin.ModelAdmin): 'name', 'models', 'payload_url', 'http_content_type', 'enabled', 'type_create', 'type_update', 'type_delete', 'ssl_verification', ] + list_filter = [ + 'enabled', 'type_create', 'type_update', 'type_delete', 'obj_type', + ] form = WebhookForm def models(self, obj): @@ -70,7 +73,12 @@ class CustomFieldChoiceAdmin(admin.TabularInline): @admin.register(CustomField, site=admin_site) class CustomFieldAdmin(admin.ModelAdmin): inlines = [CustomFieldChoiceAdmin] - list_display = ['name', 'models', 'type', 'required', 'filter_logic', 'default', 'weight', 'description'] + list_display = [ + 'name', 'models', 'type', 'required', 'filter_logic', 'default', 'weight', 'description', + ] + list_filter = [ + 'type', 'required', 'obj_type', + ] form = CustomFieldForm def models(self, obj): @@ -106,7 +114,12 @@ class CustomLinkForm(forms.ModelForm): @admin.register(CustomLink, site=admin_site) class CustomLinkAdmin(admin.ModelAdmin): - list_display = ['name', 'content_type', 'group_name', 'weight'] + list_display = [ + 'name', 'content_type', 'group_name', 'weight', + ] + list_filter = [ + 'content_type', + ] form = CustomLinkForm @@ -116,7 +129,12 @@ class CustomLinkAdmin(admin.ModelAdmin): @admin.register(Graph, site=admin_site) class GraphAdmin(admin.ModelAdmin): - list_display = ['name', 'type', 'weight', 'source'] + list_display = [ + 'name', 'type', 'weight', 'source', + ] + list_filter = [ + 'type', + ] # @@ -139,7 +157,12 @@ class ExportTemplateForm(forms.ModelForm): @admin.register(ExportTemplate, site=admin_site) class ExportTemplateAdmin(admin.ModelAdmin): - list_display = ['name', 'content_type', 'description', 'mime_type', 'file_extension'] + list_display = [ + 'name', 'content_type', 'description', 'mime_type', 'file_extension', + ] + list_filter = [ + 'content_type', + ] form = ExportTemplateForm diff --git a/netbox/extras/api/customfields.py b/netbox/extras/api/customfields.py index 2cdb79f8b..42dc486b8 100644 --- a/netbox/extras/api/customfields.py +++ b/netbox/extras/api/customfields.py @@ -97,13 +97,13 @@ class CustomFieldModelSerializer(ValidatedModelSerializer): def __init__(self, *args, **kwargs): def _populate_custom_fields(instance, fields): - custom_fields = {f.name: None for f in fields} - for cfv in instance.custom_field_values.all(): - if cfv.field.type == CF_TYPE_SELECT: - custom_fields[cfv.field.name] = CustomFieldChoiceSerializer(cfv.value).data + instance.custom_fields = {} + for field in fields: + value = instance.cf.get(field.name) + if field.type == CF_TYPE_SELECT and value is not None: + instance.custom_fields[field.name] = CustomFieldChoiceSerializer(value).data else: - custom_fields[cfv.field.name] = cfv.value - instance.custom_fields = custom_fields + instance.custom_fields[field.name] = value super().__init__(*args, **kwargs) diff --git a/netbox/extras/middleware.py b/netbox/extras/middleware.py index b0203309b..57f8f37d1 100644 --- a/netbox/extras/middleware.py +++ b/netbox/extras/middleware.py @@ -1,14 +1,15 @@ import random import threading import uuid +from copy import deepcopy from datetime import timedelta from django.conf import settings -from django.db.models.signals import post_delete, post_save +from django.db.models.signals import pre_delete, post_save from django.utils import timezone -from django.utils.functional import curry from django_prometheus.models import model_deletes, model_inserts, model_updates +from utilities.querysets import DummyQuerySet from .constants import * from .models import ObjectChange from .signals import purge_changelog @@ -19,33 +20,34 @@ _thread_locals = threading.local() def handle_changed_object(sender, instance, **kwargs): """ - Fires when an object is created or updated + Fires when an object is created or updated. """ - # Queue the object and a new ObjectChange for processing once the request completes - if hasattr(instance, 'to_objectchange'): - action = OBJECTCHANGE_ACTION_CREATE if kwargs['created'] else OBJECTCHANGE_ACTION_UPDATE - objectchange = instance.to_objectchange(action) - _thread_locals.changed_objects.append( - (instance, objectchange) - ) + # Queue the object for processing once the request completes + action = OBJECTCHANGE_ACTION_CREATE if kwargs['created'] else OBJECTCHANGE_ACTION_UPDATE + _thread_locals.changed_objects.append( + (instance, action) + ) -def _handle_deleted_object(request, sender, instance, **kwargs): +def handle_deleted_object(sender, instance, **kwargs): """ - Fires when an object is deleted + Fires when an object is deleted. """ - # Record an Object Change - if hasattr(instance, 'to_objectchange'): - objectchange = instance.to_objectchange(OBJECTCHANGE_ACTION_DELETE) - objectchange.user = request.user - objectchange.request_id = request.id - objectchange.save() + # Cache custom fields prior to copying the instance + if hasattr(instance, 'cache_custom_fields'): + instance.cache_custom_fields() - # Enqueue webhooks - enqueue_webhooks(instance, request.user, request.id, OBJECTCHANGE_ACTION_DELETE) + # Create a copy of the object being deleted + copy = deepcopy(instance) - # Increment metric counters - model_deletes.labels(instance._meta.model_name).inc() + # Preserve tags + if hasattr(instance, 'tags'): + copy.tags = DummyQuerySet(instance.tags.all()) + + # Queue the copy of the object for processing once the request completes + _thread_locals.changed_objects.append( + (copy, OBJECTCHANGE_ACTION_DELETE) + ) def purge_objectchange_cache(sender, **kwargs): @@ -81,12 +83,9 @@ class ObjectChangeMiddleware(object): # the same request. request.id = uuid.uuid4() - # Signals don't include the request context, so we're currying it into the post_delete function ahead of time. - handle_deleted_object = curry(_handle_deleted_object, request) - # Connect our receivers to the post_save and post_delete signals. - post_save.connect(handle_changed_object, dispatch_uid='cache_changed_object') - post_delete.connect(handle_deleted_object, dispatch_uid='cache_deleted_object') + post_save.connect(handle_changed_object, dispatch_uid='handle_changed_object') + pre_delete.connect(handle_deleted_object, dispatch_uid='handle_deleted_object') # Provide a hook for purging the change cache purge_changelog.connect(purge_objectchange_cache) @@ -98,22 +97,31 @@ class ObjectChangeMiddleware(object): if not _thread_locals.changed_objects: return response - # Create records for any cached objects that were created/updated. - for obj, objectchange in _thread_locals.changed_objects: + # Create records for any cached objects that were changed. + for instance, action in _thread_locals.changed_objects: - # Record the change - objectchange.user = request.user - objectchange.request_id = request.id - objectchange.save() + # Refresh cached custom field values + if action in [OBJECTCHANGE_ACTION_CREATE, OBJECTCHANGE_ACTION_UPDATE]: + if hasattr(instance, 'cache_custom_fields'): + instance.cache_custom_fields() + + # Record an ObjectChange if applicable + if hasattr(instance, 'to_objectchange'): + objectchange = instance.to_objectchange(action) + objectchange.user = request.user + objectchange.request_id = request.id + objectchange.save() # Enqueue webhooks - enqueue_webhooks(obj, request.user, request.id, objectchange.action) + enqueue_webhooks(instance, request.user, request.id, action) # Increment metric counters - if objectchange.action == OBJECTCHANGE_ACTION_CREATE: - model_inserts.labels(obj._meta.model_name).inc() - elif objectchange.action == OBJECTCHANGE_ACTION_UPDATE: - model_updates.labels(obj._meta.model_name).inc() + if action == OBJECTCHANGE_ACTION_CREATE: + model_inserts.labels(instance._meta.model_name).inc() + elif action == OBJECTCHANGE_ACTION_UPDATE: + model_updates.labels(instance._meta.model_name).inc() + elif action == OBJECTCHANGE_ACTION_DELETE: + model_deletes.labels(instance._meta.model_name).inc() # Housekeeping: 1% chance of clearing out expired ObjectChanges. This applies only to requests which result in # one or more changes being logged. diff --git a/netbox/extras/migrations/0026_webhook_ca_file_path.py b/netbox/extras/migrations/0026_webhook_ca_file_path.py new file mode 100644 index 000000000..07b5267f2 --- /dev/null +++ b/netbox/extras/migrations/0026_webhook_ca_file_path.py @@ -0,0 +1,18 @@ +# Generated by Django 2.2 on 2019-10-13 05:22 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('extras', '0025_objectchange_time_index'), + ] + + operations = [ + migrations.AddField( + model_name='webhook', + name='ca_file_path', + field=models.CharField(blank=True, max_length=4096, null=True), + ), + ] diff --git a/netbox/extras/migrations/0027_webhook_additional_headers.py b/netbox/extras/migrations/0027_webhook_additional_headers.py new file mode 100644 index 000000000..8b1f04f19 --- /dev/null +++ b/netbox/extras/migrations/0027_webhook_additional_headers.py @@ -0,0 +1,19 @@ +# Generated by Django 2.2 on 2019-10-13 07:06 + +import django.contrib.postgres.fields.jsonb +from django.db import migrations + + +class Migration(migrations.Migration): + + dependencies = [ + ('extras', '0026_webhook_ca_file_path'), + ] + + operations = [ + migrations.AddField( + model_name='webhook', + name='additional_headers', + field=django.contrib.postgres.fields.jsonb.JSONField(blank=True, null=True), + ), + ] diff --git a/netbox/extras/models.py b/netbox/extras/models.py index d4125c327..170035eb7 100644 --- a/netbox/extras/models.py +++ b/netbox/extras/models.py @@ -70,6 +70,12 @@ class Webhook(models.Model): default=WEBHOOK_CT_JSON, verbose_name='HTTP content type' ) + additional_headers = JSONField( + null=True, + blank=True, + help_text="User supplied headers which should be added to the request in addition to the HTTP content type. " + "Headers are supplied as key/value pairs in a JSON object." + ) secret = models.CharField( max_length=255, blank=True, @@ -86,6 +92,14 @@ class Webhook(models.Model): verbose_name='SSL verification', help_text="Enable SSL certificate verification. Disable with caution!" ) + ca_file_path = models.CharField( + max_length=4096, + null=True, + blank=True, + verbose_name='CA File Path', + help_text='The specific CA certificate file to use for SSL verification. ' + 'Leave blank to use the system defaults.' + ) class Meta: unique_together = ('payload_url', 'type_create', 'type_update', 'type_delete',) @@ -102,6 +116,17 @@ class Webhook(models.Model): "You must select at least one type: create, update, and/or delete." ) + if not self.ssl_verification and self.ca_file_path: + raise ValidationError({ + 'ca_file_path': 'Do not specify a CA certificate file if SSL verification is dissabled.' + }) + + # Verify that JSON data is provided as an object + if self.additional_headers and type(self.additional_headers) is not dict: + raise ValidationError({ + 'additional_headers': 'Header JSON data must be in object form. Example: {"X-API-KEY": "abc123"}' + }) + # # Custom fields @@ -113,16 +138,21 @@ class CustomFieldModel(models.Model): class Meta: abstract = True + def cache_custom_fields(self): + """ + Cache all custom field values for this instance + """ + self._cf = { + field.name: value for field, value in self.get_custom_fields().items() + } + @property def cf(self): """ Name-based CustomFieldValue accessor for use in templates """ if self._cf is None: - # Cache all custom field values for this instance - self._cf = { - field.name: value for field, value in self.get_custom_fields().items() - } + self.cache_custom_fields() return self._cf def get_custom_fields(self): diff --git a/netbox/extras/scripts.py b/netbox/extras/scripts.py index f83cffdea..4e0934a6a 100644 --- a/netbox/extras/scripts.py +++ b/netbox/extras/scripts.py @@ -24,6 +24,7 @@ from .signals import purge_changelog __all__ = [ 'BaseScript', 'BooleanVar', + 'ChoiceVar', 'FileVar', 'IntegerVar', 'IPNetworkVar', @@ -133,6 +134,27 @@ class BooleanVar(ScriptVariable): self.field_attrs['required'] = False +class ChoiceVar(ScriptVariable): + """ + Select one of several predefined static choices, passed as a list of two-tuples. Example: + + color = ChoiceVar( + choices=( + ('#ff0000', 'Red'), + ('#00ff00', 'Green'), + ('#0000ff', 'Blue') + ) + ) + """ + form_field = forms.ChoiceField + + def __init__(self, choices, *args, **kwargs): + super().__init__(*args, **kwargs) + + # Set field choices + self.field_attrs['choices'] = choices + + class ObjectVar(ScriptVariable): """ NetBox object representation. The provided QuerySet will determine the choices available. diff --git a/netbox/extras/tests/test_changelog.py b/netbox/extras/tests/test_changelog.py index 0567fa62c..22b4912b9 100644 --- a/netbox/extras/tests/test_changelog.py +++ b/netbox/extras/tests/test_changelog.py @@ -1,33 +1,57 @@ +from django.contrib.contenttypes.models import ContentType from django.urls import reverse from rest_framework import status from dcim.models import Site -from extras.constants import OBJECTCHANGE_ACTION_CREATE, OBJECTCHANGE_ACTION_UPDATE, OBJECTCHANGE_ACTION_DELETE -from extras.models import ObjectChange +from extras.constants import * +from extras.models import CustomField, CustomFieldValue, ObjectChange from utilities.testing import APITestCase class ChangeLogTest(APITestCase): + def setUp(self): + + super().setUp() + + # Create a custom field on the Site model + ct = ContentType.objects.get_for_model(Site) + cf = CustomField( + type=CF_TYPE_TEXT, + name='my_field', + required=False + ) + cf.save() + cf.obj_type.set([ct]) + def test_create_object(self): data = { 'name': 'Test Site 1', 'slug': 'test-site-1', + 'custom_fields': { + 'my_field': 'ABC' + }, + 'tags': [ + 'bar', 'foo' + ], } self.assertEqual(ObjectChange.objects.count(), 0) url = reverse('dcim-api:site-list') response = self.client.post(url, data, format='json', **self.header) - self.assertHttpStatus(response, status.HTTP_201_CREATED) - self.assertEqual(ObjectChange.objects.count(), 1) - oc = ObjectChange.objects.first() site = Site.objects.get(pk=response.data['id']) + oc = ObjectChange.objects.get( + changed_object_type=ContentType.objects.get_for_model(Site), + changed_object_id=site.pk + ) self.assertEqual(oc.changed_object, site) self.assertEqual(oc.action, OBJECTCHANGE_ACTION_CREATE) + self.assertEqual(oc.object_data['custom_fields'], data['custom_fields']) + self.assertListEqual(sorted(oc.object_data['tags']), data['tags']) def test_update_object(self): @@ -37,26 +61,43 @@ class ChangeLogTest(APITestCase): data = { 'name': 'Test Site X', 'slug': 'test-site-x', + 'custom_fields': { + 'my_field': 'DEF' + }, + 'tags': [ + 'abc', 'xyz' + ], } self.assertEqual(ObjectChange.objects.count(), 0) url = reverse('dcim-api:site-detail', kwargs={'pk': site.pk}) response = self.client.put(url, data, format='json', **self.header) - self.assertHttpStatus(response, status.HTTP_200_OK) - self.assertEqual(ObjectChange.objects.count(), 1) - site = Site.objects.get(pk=response.data['id']) - self.assertEqual(site.name, data['name']) - oc = ObjectChange.objects.first() + site = Site.objects.get(pk=response.data['id']) + oc = ObjectChange.objects.get( + changed_object_type=ContentType.objects.get_for_model(Site), + changed_object_id=site.pk + ) self.assertEqual(oc.changed_object, site) self.assertEqual(oc.action, OBJECTCHANGE_ACTION_UPDATE) + self.assertEqual(oc.object_data['custom_fields'], data['custom_fields']) + self.assertListEqual(sorted(oc.object_data['tags']), data['tags']) def test_delete_object(self): - site = Site(name='Test Site 1', slug='test-site-1') + site = Site( + name='Test Site 1', + slug='test-site-1' + ) site.save() + site.tags.add('foo', 'bar') + CustomFieldValue.objects.create( + field=CustomField.objects.get(name='my_field'), + obj=site, + value='ABC' + ) self.assertEqual(ObjectChange.objects.count(), 0) @@ -70,3 +111,5 @@ class ChangeLogTest(APITestCase): self.assertEqual(oc.changed_object, None) self.assertEqual(oc.object_repr, site.name) self.assertEqual(oc.action, OBJECTCHANGE_ACTION_DELETE) + self.assertEqual(oc.object_data['custom_fields'], {'my_field': 'ABC'}) + self.assertListEqual(sorted(oc.object_data['tags']), ['bar', 'foo']) diff --git a/netbox/extras/tests/test_scripts.py b/netbox/extras/tests/test_scripts.py index f9fc98ff2..26e12772f 100644 --- a/netbox/extras/tests/test_scripts.py +++ b/netbox/extras/tests/test_scripts.py @@ -99,6 +99,31 @@ class ScriptVariablesTest(TestCase): self.assertTrue(form.is_valid()) self.assertEqual(form.cleaned_data['var1'], False) + def test_choicevar(self): + + CHOICES = ( + ('ff0000', 'Red'), + ('00ff00', 'Green'), + ('0000ff', 'Blue') + ) + + class TestScript(Script): + + var1 = ChoiceVar( + choices=CHOICES + ) + + # Validate valid choice + data = {'var1': CHOICES[0][0]} + form = TestScript().as_form(data) + self.assertTrue(form.is_valid()) + self.assertEqual(form.cleaned_data['var1'], CHOICES[0][0]) + + # Validate invalid choices + data = {'var1': 'taupe'} + form = TestScript().as_form(data) + self.assertFalse(form.is_valid()) + def test_objectvar(self): class TestScript(Script): diff --git a/netbox/extras/webhooks_worker.py b/netbox/extras/webhooks_worker.py index 8712092d4..9a637e852 100644 --- a/netbox/extras/webhooks_worker.py +++ b/netbox/extras/webhooks_worker.py @@ -25,6 +25,9 @@ def process_webhook(webhook, data, model_name, event, timestamp, username, reque headers = { 'Content-Type': webhook.get_http_content_type_display(), } + if webhook.additional_headers: + headers.update(webhook.additional_headers) + params = { 'method': 'POST', 'url': webhook.payload_url, @@ -49,6 +52,8 @@ def process_webhook(webhook, data, model_name, event, timestamp, username, reque with requests.Session() as session: session.verify = webhook.ssl_verification + if webhook.ca_file_path: + session.verify = webhook.ca_file_path response = session.send(prepared_request) if response.status_code >= 200 and response.status_code <= 299: diff --git a/netbox/netbox/configuration.example.py b/netbox/netbox/configuration.example.py index ebc3d4540..eb092fc29 100644 --- a/netbox/netbox/configuration.example.py +++ b/netbox/netbox/configuration.example.py @@ -17,12 +17,13 @@ DATABASE = { 'PASSWORD': '', # PostgreSQL password 'HOST': 'localhost', # Database server 'PORT': '', # Database port (leave blank for default) + 'CONN_MAX_AGE': 300, # Max database connection age } # This key is used for secure generation of random numbers and strings. It must never be exposed outside of this file. # For optimal security, SECRET_KEY should be at least 50 characters in length and contain a mix of letters, numbers, and # symbols. NetBox will not run without this defined. For more information, see -# https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-SECRET_KEY +# https://docs.djangoproject.com/en/stable/ref/settings/#std:setting-SECRET_KEY SECRET_KEY = '' # Redis database settings. The Redis database is used for caching and background processing such as webhooks @@ -106,7 +107,7 @@ EXEMPT_VIEW_PERMISSIONS = [ ] # Enable custom logging. Please see the Django documentation for detailed guidance on configuring custom logs: -# https://docs.djangoproject.com/en/1.11/topics/logging/ +# https://docs.djangoproject.com/en/stable/topics/logging/ LOGGING = {} # Setting this to True will permit only authenticated users to access any part of NetBox. By default, anonymous users @@ -154,6 +155,10 @@ PREFER_IPV4 = False # this setting is derived from the installed location. # REPORTS_ROOT = '/opt/netbox/netbox/reports' +# The file path where custom scripts will be stored. A trailing slash is not needed. Note that the default value of +# this setting is derived from the installed location. +# SCRIPTS_ROOT = '/opt/netbox/netbox/scripts' + # By default, NetBox will store session data in the database. Alternatively, a file path can be specified here to use # local file storage instead. (This can be useful for enabling authentication on a standby instance with read-only # database access.) Note that the user as which NetBox runs must have read and write permissions to this path. @@ -167,7 +172,7 @@ TIME_ZONE = 'UTC' WEBHOOKS_ENABLED = False # Date/time formatting. See the following link for supported formats: -# https://docs.djangoproject.com/en/dev/ref/templates/builtins/#date +# https://docs.djangoproject.com/en/stable/ref/templates/builtins/#date DATE_FORMAT = 'N j, Y' SHORT_DATE_FORMAT = 'Y-m-d' TIME_FORMAT = 'g:i a' diff --git a/netbox/netbox/settings.py b/netbox/netbox/settings.py index 5a4c7f22e..9a5444b18 100644 --- a/netbox/netbox/settings.py +++ b/netbox/netbox/settings.py @@ -12,7 +12,7 @@ from django.core.exceptions import ImproperlyConfigured # Environment setup # -VERSION = '2.6.6' +VERSION = '2.6.7' # Hostname HOSTNAME = platform.node() @@ -364,6 +364,7 @@ CACHEOPS = { 'auth.user': {'ops': 'get', 'timeout': 60 * 15}, 'auth.*': {'ops': ('fetch', 'get')}, 'auth.permission': {'ops': 'all'}, + 'circuits.*': {'ops': 'all'}, 'dcim.*': {'ops': 'all'}, 'ipam.*': {'ops': 'all'}, 'extras.*': {'ops': 'all'}, diff --git a/netbox/templates/dcim/device_lldp_neighbors.html b/netbox/templates/dcim/device_lldp_neighbors.html index 4764fffa2..221c17a52 100644 --- a/netbox/templates/dcim/device_lldp_neighbors.html +++ b/netbox/templates/dcim/device_lldp_neighbors.html @@ -52,10 +52,10 @@