Merge pull request #5448 from netbox-community/develop

Release v2.9.11

.github/ISSUE_TEMPLATE/bug_report.md

@@ -7,11 +7,9 @@ about: Report a reproducible bug in the current release of NetBox
 <!--
     NOTE: IF YOUR ISSUE DOES NOT FOLLOW THIS TEMPLATE, IT WILL BE CLOSED.
 
-    This form is only for reproducible bugs. If you need assistance with
+    This form is only for reporting reproducible bugs. If you need assistance
-    NetBox installation, or if you have a general question, DO NOT open an
+    with NetBox installation, or if you have a general question, please start a
-    issue. Instead, post to our mailing list:
+    discussion instead: https://github.com/netbox-community/netbox/discussions
-
-        https://groups.google.com/g/netbox-discuss
 
     Please describe the environment in which you are running NetBox. Be sure
     that you are running an unmodified instance of the latest stable release

.github/ISSUE_TEMPLATE/feature_request.md

@@ -8,10 +8,8 @@ about: Propose a new NetBox feature or enhancement
     NOTE: IF YOUR ISSUE DOES NOT FOLLOW THIS TEMPLATE, IT WILL BE CLOSED.
 
     This form is only for proposing specific new features or enhancements.
-    If you have a general idea or question, please post to our mailing list
+    If you have a general idea or question, please start a discussion instead:
-    instead of opening an issue:
+    https://github.com/netbox-community/netbox/discussions
-
-        https://groups.google.com/g/netbox-discuss
 
     NOTE: Due to an excessive backlog of feature requests, we are not currently
     accepting any proposals which significantly extend NetBox's feature scope.

.github/workflows/ci.yml

@@ -0,0 +1,50 @@
+name: CI
+on: push
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: [3.6, 3.7]
+    services:
+      redis:
+        image: redis
+        ports:
+          - 6379:6379
+      postgres:
+        image: postgres
+        env:
+          POSTGRES_USER: netbox
+          POSTGRES_PASSWORD: netbox
+        options: >-
+          --health-cmd pg_isready
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+        ports:
+          - 5432:5432
+
+    steps:
+    - name: Check out repo
+      uses: actions/checkout@v2
+
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v2
+      with:
+        python-version: ${{ matrix.python-version }}
+
+    - name: Install dependencies & set up configuration
+      run: |
+        python -m pip install --upgrade pip
+        pip install -r requirements.txt
+        pip install pycodestyle coverage
+        ln -s configuration.testing.py netbox/netbox/configuration.py
+
+    - name: Check PEP8 compliance
+      run: pycodestyle --ignore=W504,E501 netbox/
+
+    - name: Run tests
+      run: coverage run --source="netbox/" netbox/manage.py test netbox/
+
+    - name: Show coverage report
+      run: coverage report --skip-covered --omit *migrations*

.travis.yml

@@ -1,19 +0,0 @@
-sudo: required
-services:
-  - postgresql
-  - redis-server
-addons:
-  postgresql: "9.6"
-language: python
-python:
-  - "3.6"
-  - "3.7"
-install:
-  - pip install -r requirements.txt
-  - pip install pycodestyle
-  - pip install coverage
-before_script:
-  - psql --version
-  - psql -U postgres -c 'SELECT version();'
-script:
-  - ./scripts/cibuild.sh

CONTRIBUTING.md

@@ -4,15 +4,33 @@ If you encounter any issues installing or using NetBox, try one of the
 following resources to get assistance. Please **do not** open a GitHub issue
 except to report bugs or request features.
 
+### GitHub Discussions
+
+GitHub's discussions are the best place to get help or propose rough ideas for
+new functionality. Their integration with GitHub allows for easily cross-
+referencing and converting posts to issues as needed. There are several
+categories for discussions:
+
+* **General** - General community discussion
+* **Ideas** - Ideas for new functionality that isn't yet ready for a formal
+  feature request
+* **Q&A** - Request help with installing or using NetBox
+* **Show and tell** - Share a plugin, script, or something else you've made
+  using NetBox
+
 ### Mailing List
 
-We have established a Google Groups Mailing List for issues and general
+We also have a Google Groups [mailing list](https://groups.google.com/g/netbox-discuss)
-discussion. This is the best forum for obtaining assistance with NetBox
+for general discussion, however we're encouraging people to use GitHub
-installation. You can find us [here](https://groups.google.com/g/netbox-discuss).
+discussions where possible, as it's much easier for newcomers to review past
+discussions.
 
 ### Slack
 
-For real-time discussion, you can join the #netbox Slack channel on [NetworkToCode](https://slack.networktocode.com/).
+For real-time chat, you can join the **#netbox** Slack channel on [NetworkToCode](https://slack.networktocode.com/).
+Unfortunately, the Slack channel does not provide long-term retention of chat
+history, so try to avoid it for any discussions would benefit from being
+preserved for future reference.
 
 ## Reporting Bugs
 
@@ -171,7 +189,7 @@ overlooked.
 
 * Official channels for communication include:
 
-    * GitHub issues/pull requests
+    * GitHub issues, pull requests, and discussions
     * The [netbox-discuss](https://groups.google.com/g/netbox-discuss) mailing list
     * The **#netbox** channel on [NetworkToCode Slack](https://networktocode.slack.com/)
 

README.md

@@ -12,15 +12,15 @@ complete list of requirements, see `requirements.txt`. The code is available [on
 
 The complete documentation for NetBox can be found at [Read the Docs](http://netbox.readthedocs.io/en/stable/).
 
-Questions? Comments? Please subscribe to [the netbox-discuss mailing list](https://groups.google.com/g/netbox-discuss),
+Questions? Comments? Please start a [discussion on GitHub](https://github.com/netbox-community/netbox/discussions),
-or join us in the #netbox Slack channel on [NetworkToCode](https://networktocode.slack.com/)!
+or join us in the **#netbox** Slack channel on [NetworkToCode](https://networktocode.slack.com/)!
 
 ### Build Status
 
 |             | status |
 |-------------|------------|
-| **master** | [![Build Status](https://travis-ci.org/netbox-community/netbox.svg?branch=master)](https://travis-ci.com/netbox-community/netbox/) |
+| **master** | ![Build status](https://github.com/netbox-community/netbox/workflows/CI/badge.svg?branch=master) |
-| **develop** | [![Build Status](https://travis-ci.org/netbox-community/netbox.svg?branch=develop)](https://travis-ci.com/netbox-community/netbox/) |
+| **develop** | ![Build status](https://github.com/netbox-community/netbox/workflows/CI/badge.svg?branch=develop) |
 
 ### Screenshots
 
@@ -37,18 +37,21 @@ or join us in the #netbox Slack channel on [NetworkToCode](https://networktocode
 ## Installation
 
 Please see [the documentation](http://netbox.readthedocs.io/en/stable/) for
-instructions on installing NetBox. To upgrade NetBox, please download the [latest release](https://github.com/netbox-community/netbox/releases)
+instructions on installing NetBox. To upgrade NetBox, please download the
-and run `upgrade.sh`.
+[latest release](https://github.com/netbox-community/netbox/releases) and
+run `upgrade.sh`.
 
 ## Providing Feedback
 
-Feature requests and bug reports must be submitted as GiHub issues. (Please be
+The best platform for general feedback, assistance, and other discussion is our
-sure to use the [appropriate template](https://github.com/netbox-community/netbox/issues/new/choose).)
+[GitHub discussions](https://github.com/netbox-community/netbox/discussions).
-For general discussion, please consider joining our [mailing list](https://groups.google.com/g/netbox-discuss).
+To report a bug or request a specific feature, please open a GitHub issue using
+the [appropriate template](https://github.com/netbox-community/netbox/issues/new/choose).
 
 If you are interested in contributing to the development of NetBox, please read
 our [contributing guide](CONTRIBUTING.md) prior to beginning any work.
 
 ## Related projects
 
-Please see [our wiki](https://github.com/netbox-community/netbox/wiki/Community-Contributions) for a list of relevant community projects.
+Please see [our wiki](https://github.com/netbox-community/netbox/wiki/Community-Contributions)
+for a list of relevant community projects.

docs/development/index.md

@@ -7,7 +7,8 @@ NetBox is maintained as a [GitHub project](https://github.com/netbox-community/n
 Communication among developers should always occur via public channels:
 
 * [GitHub issues](https://github.com/netbox-community/netbox/issues) - All feature requests, bug reports, and other substantial changes to the code base **must** be documented in an issue.
-* [The mailing list](https://groups.google.com/g/netbox-discuss) - The preferred forum for general discussion and support issues. Ideal for shaping a feature request prior to submitting an issue.
+* [GitHub discussions](https://github.com/netbox-community/netbox/discussions) - The preferred forum for general discussion and support issues. Ideal for shaping a feature request prior to submitting an issue.
+* [The mailing list](https://groups.google.com/g/netbox-discuss) - An alternative forum for general discussion (GitHub is preferred).
 * [#netbox on NetworkToCode](http://slack.networktocode.com/) - Good for quick chats. Avoid any discussion that might need to be referenced later on, as the chat history is not retained long.
 
 ## Governance

docs/installation/5-http-server.md

@@ -3,7 +3,7 @@
 This documentation provides example configurations for both [nginx](https://www.nginx.com/resources/wiki/) and [Apache](http://httpd.apache.org/docs/2.4), though any HTTP server which supports WSGI should be compatible.
 
 !!! info
-    For the sake of brevity, only Ubuntu 18.04 instructions are provided here, these tasks are not unique to NetBox and should carry over to other distributions with mininal changes. Please consult your distribution's documentation for assistance if needed.
+    For the sake of brevity, only Ubuntu 18.04 instructions are provided here, these tasks are not unique to NetBox and should carry over to other distributions with minimal changes. Please consult your distribution's documentation for assistance if needed.
 
 ## Obtain an SSL Certificate
 

docs/release-notes/version-2.9.md

@@ -1,5 +1,26 @@
 # NetBox v2.9
 
+## v2.9.11 (2020-12-11)
+
+### Enhancements
+
+* [#5424](https://github.com/netbox-community/netbox/issues/5424) - Allow passing Python code to `nbshell` using `--command`
+* [#5439](https://github.com/netbox-community/netbox/issues/5439) - Add CS and SN fiber port types
+
+### Bug Fixes
+
+* [#5383](https://github.com/netbox-community/netbox/issues/5383) - Fix setting user password via REST API
+* [#5396](https://github.com/netbox-community/netbox/issues/5396) - Fix uniqueness constraint for virtual machine names
+* [#5387](https://github.com/netbox-community/netbox/issues/5387) - Fix error when rendering config contexts when objects have multiple tags assigned
+* [#5407](https://github.com/netbox-community/netbox/issues/5407) - Add direct link to secret on secrets list
+* [#5408](https://github.com/netbox-community/netbox/issues/5408) - Fix updating secrets without setting new plaintext
+* [#5410](https://github.com/netbox-community/netbox/issues/5410) - Restore tags field on cable connection forms
+* [#5433](https://github.com/netbox-community/netbox/issues/5433) - Exclude SVG files from front/rear image upload for device types (currently unsupported)
+* [#5436](https://github.com/netbox-community/netbox/issues/5436) - Show assigned IP addresses in interfaces list
+* [#5446](https://github.com/netbox-community/netbox/issues/5446) - Fix validation for plugin version and required settings
+
+---
+
 ## v2.9.10 (2020-11-24)
 
 ### Enhancements

netbox/circuits/api/views.py

@@ -1,4 +1,4 @@
-from django.db.models import Count, Prefetch
+from django.db.models import Prefetch
 from django.shortcuts import get_object_or_404
 from rest_framework.decorators import action
 from rest_framework.response import Response
@@ -10,6 +10,7 @@ from extras.api.serializers import RenderedGraphSerializer
 from extras.api.views import CustomFieldModelViewSet
 from extras.models import Graph
 from utilities.api import ModelViewSet
+from utilities.utils import get_subquery
 from . import serializers
 
 
@@ -27,8 +28,8 @@ class CircuitsRootView(APIRootView):
 
 class ProviderViewSet(CustomFieldModelViewSet):
     queryset = Provider.objects.prefetch_related('tags').annotate(
-        circuit_count=Count('circuits')
+        circuit_count=get_subquery(Circuit, 'provider')
-    ).order_by(*Provider._meta.ordering)
+    )
     serializer_class = serializers.ProviderSerializer
     filterset_class = filters.ProviderFilterSet
 
@@ -49,8 +50,8 @@ class ProviderViewSet(CustomFieldModelViewSet):
 
 class CircuitTypeViewSet(ModelViewSet):
     queryset = CircuitType.objects.annotate(
-        circuit_count=Count('circuits')
+        circuit_count=get_subquery(Circuit, 'type')
-    ).order_by(*CircuitType._meta.ordering)
+    )
     serializer_class = serializers.CircuitTypeSerializer
     filterset_class = filters.CircuitTypeFilterSet
 

netbox/circuits/views.py

@@ -1,12 +1,12 @@
 from django.contrib import messages
 from django.db import transaction
-from django.db.models import Count
 from django.shortcuts import get_object_or_404, redirect, render
 from django_tables2 import RequestConfig
 
 from extras.models import Graph
 from utilities.forms import ConfirmationForm
 from utilities.paginator import EnhancedPaginator, get_paginate_count
+from utilities.utils import get_subquery
 from utilities.views import (
     BulkDeleteView, BulkEditView, BulkImportView, ObjectView, ObjectDeleteView, ObjectEditView, ObjectListView,
 )
@@ -20,7 +20,9 @@ from .models import Circuit, CircuitTermination, CircuitType, Provider
 #
 
 class ProviderListView(ObjectListView):
-    queryset = Provider.objects.annotate(count_circuits=Count('circuits')).order_by(*Provider._meta.ordering)
+    queryset = Provider.objects.annotate(
+        count_circuits=get_subquery(Circuit, 'provider')
+    )
     filterset = filters.ProviderFilterSet
     filterset_form = forms.ProviderFilterForm
     table = tables.ProviderTable
@@ -72,14 +74,18 @@ class ProviderBulkImportView(BulkImportView):
 
 
 class ProviderBulkEditView(BulkEditView):
-    queryset = Provider.objects.annotate(count_circuits=Count('circuits')).order_by(*Provider._meta.ordering)
+    queryset = Provider.objects.annotate(
+        count_circuits=get_subquery(Circuit, 'provider')
+    )
     filterset = filters.ProviderFilterSet
     table = tables.ProviderTable
     form = forms.ProviderBulkEditForm
 
 
 class ProviderBulkDeleteView(BulkDeleteView):
-    queryset = Provider.objects.annotate(count_circuits=Count('circuits')).order_by(*Provider._meta.ordering)
+    queryset = Provider.objects.annotate(
+        count_circuits=get_subquery(Circuit, 'provider')
+    )
     filterset = filters.ProviderFilterSet
     table = tables.ProviderTable
 
@@ -89,7 +95,9 @@ class ProviderBulkDeleteView(BulkDeleteView):
 #
 
 class CircuitTypeListView(ObjectListView):
-    queryset = CircuitType.objects.annotate(circuit_count=Count('circuits')).order_by(*CircuitType._meta.ordering)
+    queryset = CircuitType.objects.annotate(
+        circuit_count=get_subquery(Circuit, 'type')
+    )
     table = tables.CircuitTypeTable
 
 
@@ -109,7 +117,9 @@ class CircuitTypeBulkImportView(BulkImportView):
 
 
 class CircuitTypeBulkDeleteView(BulkDeleteView):
-    queryset = CircuitType.objects.annotate(circuit_count=Count('circuits')).order_by(*CircuitType._meta.ordering)
+    queryset = CircuitType.objects.annotate(
+        circuit_count=get_subquery(Circuit, 'type')
+    )
     table = tables.CircuitTypeTable
 
 

netbox/dcim/api/views.py

@@ -2,7 +2,7 @@ import socket
 from collections import OrderedDict
 
 from django.conf import settings
-from django.db.models import Count, F
+from django.db.models import F
 from django.http import HttpResponseForbidden, HttpResponse
 from django.shortcuts import get_object_or_404
 from drf_yasg import openapi
@@ -109,7 +109,7 @@ class SiteViewSet(CustomFieldModelViewSet):
         vlan_count=get_subquery(VLAN, 'site'),
         circuit_count=get_subquery(Circuit, 'terminations__site'),
         virtualmachine_count=get_subquery(VirtualMachine, 'cluster__site'),
-    ).order_by(*Site._meta.ordering)
+    )
     serializer_class = serializers.SiteSerializer
     filterset_class = filters.SiteFilterSet
 
@@ -146,8 +146,8 @@ class RackGroupViewSet(ModelViewSet):
 
 class RackRoleViewSet(ModelViewSet):
     queryset = RackRole.objects.annotate(
-        rack_count=Count('racks')
+        rack_count=get_subquery(Rack, 'role')
-    ).order_by(*RackRole._meta.ordering)
+    )
     serializer_class = serializers.RackRoleSerializer
     filterset_class = filters.RackRoleFilterSet
 
@@ -162,7 +162,7 @@ class RackViewSet(CustomFieldModelViewSet):
     ).annotate(
         device_count=get_subquery(Device, 'rack'),
         powerfeed_count=get_subquery(PowerFeed, 'rack')
-    ).order_by(*Rack._meta.ordering)
+    )
     serializer_class = serializers.RackSerializer
     filterset_class = filters.RackFilterSet
 
@@ -237,7 +237,7 @@ class ManufacturerViewSet(ModelViewSet):
         devicetype_count=get_subquery(DeviceType, 'manufacturer'),
         inventoryitem_count=get_subquery(InventoryItem, 'manufacturer'),
         platform_count=get_subquery(Platform, 'manufacturer')
-    ).order_by(*Manufacturer._meta.ordering)
+    )
     serializer_class = serializers.ManufacturerSerializer
     filterset_class = filters.ManufacturerFilterSet
 
@@ -248,8 +248,8 @@ class ManufacturerViewSet(ModelViewSet):
 
 class DeviceTypeViewSet(CustomFieldModelViewSet):
     queryset = DeviceType.objects.prefetch_related('manufacturer', 'tags').annotate(
-        device_count=Count('instances')
+        device_count=get_subquery(Device, 'device_type')
-    ).order_by(*DeviceType._meta.ordering)
+    )
     serializer_class = serializers.DeviceTypeSerializer
     filterset_class = filters.DeviceTypeFilterSet
 
@@ -314,7 +314,7 @@ class DeviceRoleViewSet(ModelViewSet):
     queryset = DeviceRole.objects.annotate(
         device_count=get_subquery(Device, 'device_role'),
         virtualmachine_count=get_subquery(VirtualMachine, 'role')
-    ).order_by(*DeviceRole._meta.ordering)
+    )
     serializer_class = serializers.DeviceRoleSerializer
     filterset_class = filters.DeviceRoleFilterSet
 
@@ -327,7 +327,7 @@ class PlatformViewSet(ModelViewSet):
     queryset = Platform.objects.annotate(
         device_count=get_subquery(Device, 'platform'),
         virtualmachine_count=get_subquery(VirtualMachine, 'platform')
-    ).order_by(*Platform._meta.ordering)
+    )
     serializer_class = serializers.PlatformSerializer
     filterset_class = filters.PlatformFilterSet
 
@@ -619,8 +619,8 @@ class CableViewSet(ModelViewSet):
 
 class VirtualChassisViewSet(ModelViewSet):
     queryset = VirtualChassis.objects.prefetch_related('tags').annotate(
-        member_count=Count('members', distinct=True)
+        member_count=get_subquery(Device, 'virtual_chassis')
-    ).order_by(*VirtualChassis._meta.ordering)
+    )
     serializer_class = serializers.VirtualChassisSerializer
     filterset_class = filters.VirtualChassisFilterSet
 
@@ -633,8 +633,8 @@ class PowerPanelViewSet(ModelViewSet):
     queryset = PowerPanel.objects.prefetch_related(
         'site', 'rack_group'
     ).annotate(
-        powerfeed_count=Count('powerfeeds')
+        powerfeed_count=get_subquery(PowerFeed, 'power_panel')
-    ).order_by(*PowerPanel._meta.ordering)
+    )
     serializer_class = serializers.PowerPanelSerializer
     filterset_class = filters.PowerPanelFilterSet
 

netbox/dcim/choices.py

@@ -861,6 +861,8 @@ class PortTypeChoices(ChoiceSet):
     TYPE_LSH = 'lsh'
     TYPE_LSH_APC = 'lsh-apc'
     TYPE_SPLICE = 'splice'
+    TYPE_CS = 'cs'
+    TYPE_SN = 'sn'
 
     CHOICES = (
         (
@@ -888,6 +890,8 @@ class PortTypeChoices(ChoiceSet):
                 (TYPE_SC, 'SC'),
                 (TYPE_SC_APC, 'SC/APC'),
                 (TYPE_ST, 'ST'),
+                (TYPE_CS, 'CS'),
+                (TYPE_SN, 'SN'),
                 (TYPE_SPLICE, 'Splice'),
             )
         )

netbox/dcim/forms.py

@@ -922,7 +922,14 @@ class DeviceTypeForm(BootstrapMixin, CustomFieldModelForm):
             'front_image', 'rear_image', 'comments', 'tags',
         ]
         widgets = {
-            'subdevice_role': StaticSelect2()
+            'subdevice_role': StaticSelect2(),
+            # Exclude SVG images (unsupported by PIL)
+            'front_image': forms.FileInput(attrs={
+                'accept': 'image/bmp,image/gif,image/jpeg,image/png,image/tiff'
+            }),
+            'rear_image': forms.FileInput(attrs={
+                'accept': 'image/bmp,image/gif,image/jpeg,image/png,image/tiff'
+            })
         }
 
 
@@ -3502,12 +3509,16 @@ class ConnectCableToDeviceForm(BootstrapMixin, forms.ModelForm):
             'rack_id': '$termination_b_rack',
         }
     )
+    tags = DynamicModelMultipleChoiceField(
+        queryset=Tag.objects.all(),
+        required=False
+    )
 
     class Meta:
         model = Cable
         fields = [
             'termination_b_region', 'termination_b_site', 'termination_b_rack', 'termination_b_device',
-            'termination_b_id', 'type', 'status', 'label', 'color', 'length', 'length_unit',
+            'termination_b_id', 'type', 'status', 'label', 'color', 'length', 'length_unit', 'tags',
         ]
         widgets = {
             'status': StaticSelect2,
@@ -3635,12 +3646,16 @@ class ConnectCableToCircuitTerminationForm(BootstrapMixin, forms.ModelForm):
             'circuit_id': '$termination_b_circuit'
         }
     )
+    tags = DynamicModelMultipleChoiceField(
+        queryset=Tag.objects.all(),
+        required=False
+    )
 
     class Meta:
         model = Cable
         fields = [
             'termination_b_provider', 'termination_b_region', 'termination_b_site', 'termination_b_circuit',
-            'termination_b_id', 'type', 'status', 'label', 'color', 'length', 'length_unit',
+            'termination_b_id', 'type', 'status', 'label', 'color', 'length', 'length_unit', 'tags',
         ]
 
     def clean_termination_b_id(self):
@@ -3688,12 +3703,16 @@ class ConnectCableToPowerFeedForm(BootstrapMixin, forms.ModelForm):
             'power_panel_id': '$termination_b_powerpanel'
         }
     )
+    tags = DynamicModelMultipleChoiceField(
+        queryset=Tag.objects.all(),
+        required=False
+    )
 
     class Meta:
         model = Cable
         fields = [
             'termination_b_rackgroup', 'termination_b_powerpanel', 'termination_b_id', 'type', 'status', 'label',
-            'color', 'length', 'length_unit',
+            'color', 'length', 'length_unit', 'tags',
         ]
 
     def clean_termination_b_id(self):

netbox/dcim/tables.py

@@ -141,7 +141,7 @@ POWERPANEL_POWERFEED_COUNT = """
 """
 
 INTERFACE_IPADDRESSES = """
-{% for ip in record.ip_addresses.unrestricted %}
+{% for ip in record.ip_addresses.all %}
     <a href="{{ ip.get_absolute_url }}">{{ ip }}</a><br />
 {% endfor %}
 """

netbox/dcim/views.py

@@ -4,7 +4,7 @@ from django.contrib import messages
 from django.contrib.contenttypes.models import ContentType
 from django.core.paginator import EmptyPage, PageNotAnInteger
 from django.db import transaction
-from django.db.models import Count, F, Prefetch
+from django.db.models import F, Prefetch
 from django.forms import ModelMultipleChoiceField, MultipleHiddenInput, modelformset_factory
 from django.shortcuts import get_object_or_404, redirect, render
 from django.urls import reverse
@@ -263,7 +263,9 @@ class RackGroupBulkDeleteView(BulkDeleteView):
 #
 
 class RackRoleListView(ObjectListView):
-    queryset = RackRole.objects.annotate(rack_count=Count('racks')).order_by(*RackRole._meta.ordering)
+    queryset = RackRole.objects.annotate(
+        rack_count=get_subquery(Rack, 'role')
+    )
     table = tables.RackRoleTable
 
 
@@ -283,7 +285,9 @@ class RackRoleBulkImportView(BulkImportView):
 
 
 class RackRoleBulkDeleteView(BulkDeleteView):
-    queryset = RackRole.objects.annotate(rack_count=Count('racks')).order_by(*RackRole._meta.ordering)
+    queryset = RackRole.objects.annotate(
+        rack_count=get_subquery(Rack, 'role')
+    )
     table = tables.RackRoleTable
 
 
@@ -295,8 +299,8 @@ class RackListView(ObjectListView):
     queryset = Rack.objects.prefetch_related(
         'site', 'group', 'tenant', 'role', 'devices__device_type'
     ).annotate(
-        device_count=Count('devices')
+        device_count=get_subquery(Device, 'rack')
-    ).order_by(*Rack._meta.ordering)
+    )
     filterset = filters.RackFilterSet
     filterset_form = forms.RackFilterForm
     table = tables.RackDetailTable
@@ -507,8 +511,8 @@ class ManufacturerBulkImportView(BulkImportView):
 
 class ManufacturerBulkDeleteView(BulkDeleteView):
     queryset = Manufacturer.objects.annotate(
-        devicetype_count=Count('device_types')
+        devicetype_count=get_subquery(DeviceType, 'manufacturer')
-    ).order_by(*Manufacturer._meta.ordering)
+    )
     table = tables.ManufacturerTable
 
 
@@ -518,8 +522,8 @@ class ManufacturerBulkDeleteView(BulkDeleteView):
 
 class DeviceTypeListView(ObjectListView):
     queryset = DeviceType.objects.prefetch_related('manufacturer').annotate(
-        instance_count=Count('instances')
+        instance_count=get_subquery(Device, 'device_type')
-    ).order_by(*DeviceType._meta.ordering)
+    )
     filterset = filters.DeviceTypeFilterSet
     filterset_form = forms.DeviceTypeFilterForm
     table = tables.DeviceTypeTable
@@ -628,8 +632,8 @@ class DeviceTypeImportView(ObjectImportView):
 
 class DeviceTypeBulkEditView(BulkEditView):
     queryset = DeviceType.objects.prefetch_related('manufacturer').annotate(
-        instance_count=Count('instances')
+        instance_count=get_subquery(Device, 'device_type')
-    ).order_by(*DeviceType._meta.ordering)
+    )
     filterset = filters.DeviceTypeFilterSet
     table = tables.DeviceTypeTable
     form = forms.DeviceTypeBulkEditForm
@@ -637,8 +641,8 @@ class DeviceTypeBulkEditView(BulkEditView):
 
 class DeviceTypeBulkDeleteView(BulkDeleteView):
     queryset = DeviceType.objects.prefetch_related('manufacturer').annotate(
-        instance_count=Count('instances')
+        instance_count=get_subquery(Device, 'device_type')
-    ).order_by(*DeviceType._meta.ordering)
+    )
     filterset = filters.DeviceTypeFilterSet
     table = tables.DeviceTypeTable
 
@@ -2198,8 +2202,8 @@ class InterfaceConnectionsListView(ObjectListView):
 
 class VirtualChassisListView(ObjectListView):
     queryset = VirtualChassis.objects.prefetch_related('master').annotate(
-        member_count=Count('members', distinct=True)
+        member_count=get_subquery(Device, 'virtual_chassis')
-    ).order_by(*VirtualChassis._meta.ordering)
+    )
     table = tables.VirtualChassisTable
     filterset = filters.VirtualChassisFilterSet
     filterset_form = forms.VirtualChassisFilterForm
@@ -2430,8 +2434,8 @@ class PowerPanelListView(ObjectListView):
     queryset = PowerPanel.objects.prefetch_related(
         'site', 'rack_group'
     ).annotate(
-        powerfeed_count=Count('powerfeeds')
+        powerfeed_count=get_subquery(PowerFeed, 'power_panel')
-    ).order_by(*PowerPanel._meta.ordering)
+    )
     filterset = filters.PowerPanelFilterSet
     filterset_form = forms.PowerPanelFilterForm
     table = tables.PowerPanelTable
@@ -2482,8 +2486,8 @@ class PowerPanelBulkDeleteView(BulkDeleteView):
     queryset = PowerPanel.objects.prefetch_related(
         'site', 'rack_group'
     ).annotate(
-        rack_count=Count('powerfeeds')
+        powerfeed_count=get_subquery(PowerFeed, 'power_panel')
-    ).order_by(*PowerPanel._meta.ordering)
+    )
     filterset = filters.PowerPanelFilterSet
     table = tables.PowerPanelTable
 

netbox/extras/api/views.py

@@ -1,7 +1,6 @@
 from collections import OrderedDict
 
 from django.contrib.contenttypes.models import ContentType
-from django.db.models import Count
 from django.http import Http404
 from django_rq.queues import get_connection
 from rest_framework import status
@@ -15,14 +14,14 @@ from rq import Worker
 from extras import filters
 from extras.choices import JobResultStatusChoices
 from extras.models import (
-    ConfigContext, CustomFieldChoice, ExportTemplate, Graph, ImageAttachment, ObjectChange, JobResult, Tag,
+    ConfigContext, CustomFieldChoice, ExportTemplate, Graph, ImageAttachment, ObjectChange, JobResult, Tag, TaggedItem,
 )
 from extras.reports import get_report, get_reports, run_report
 from extras.scripts import get_script, get_scripts, run_script
 from utilities.api import IsAuthenticatedOrLoginNotRequired, ModelViewSet
 from utilities.exceptions import RQWorkerNotRunningException
 from utilities.metadata import ContentTypeMetadata
-from utilities.utils import copy_safe_request
+from utilities.utils import copy_safe_request, get_subquery
 from . import serializers
 
 
@@ -149,8 +148,8 @@ class ExportTemplateViewSet(ModelViewSet):
 
 class TagViewSet(ModelViewSet):
     queryset = Tag.objects.annotate(
-        tagged_items=Count('extras_taggeditem_items')
+        tagged_items=get_subquery(TaggedItem, 'tag')
-    ).order_by(*Tag._meta.ordering)
+    )
     serializer_class = serializers.TagSerializer
     filterset_class = filters.TagFilterSet
 

netbox/extras/management/commands/nbshell.py

@@ -25,12 +25,18 @@ class Command(BaseCommand):
     help = "Start the Django shell with all NetBox models already imported"
     django_models = {}
 
+    def add_arguments(self, parser):
+        parser.add_argument(
+            '-c', '--command',
+            help='Python code to execute (instead of starting an interactive shell)',
+        )
+
     def _lsmodels(self):
         for app, models in self.django_models.items():
             app_name = apps.get_app_config(app).verbose_name
-            print('{}:'.format(app_name))
+            print(f'{app_name}:')
             for m in models:
-                print('  {}'.format(m))
+                print(f'  {m}')
 
     def get_namespace(self):
         namespace = {}
@@ -46,7 +52,7 @@ class Command(BaseCommand):
 
             # Constants
             try:
-                app_constants = sys.modules['{}.constants'.format(app)]
+                app_constants = sys.modules[f'{app}.constants']
                 for name in dir(app_constants):
                     namespace[name] = getattr(app_constants, name)
             except KeyError:
@@ -64,5 +70,10 @@ class Command(BaseCommand):
         return namespace
 
     def handle(self, **options):
+        # If Python code has been passed, execute it and exit.
+        if options['command']:
+            exec(options['command'], self.get_namespace())
+            return
+
         shell = code.interact(banner=BANNER_TEXT, local=self.get_namespace())
         return shell

netbox/extras/plugins/__init__.py

@@ -3,7 +3,6 @@ import inspect
 from packaging import version
 
 from django.apps import AppConfig
-from django.conf import settings
 from django.core.exceptions import ImproperlyConfigured
 from django.template.loader import get_template
 
@@ -71,10 +70,10 @@ class PluginConfig(AppConfig):
             register_menu_items(self.verbose_name, menu_items)
 
     @classmethod
-    def validate(cls, user_config):
+    def validate(cls, user_config, netbox_version):
 
         # Enforce version constraints
-        current_version = version.parse(settings.VERSION)
+        current_version = version.parse(netbox_version)
         if cls.min_version is not None:
             min_version = version.parse(cls.min_version)
             if current_version < min_version:

netbox/extras/querysets.py

@@ -2,6 +2,7 @@ from collections import OrderedDict
 
 from django.db.models import OuterRef, Subquery, Q
 
+from extras.models.tags import TaggedItem
 from utilities.query_functions import EmptyGroupByJSONBAgg, OrderableJSONBAgg
 from utilities.querysets import RestrictedQuerySet
 
@@ -99,11 +100,25 @@ class ConfigContextModelQuerySet(RestrictedQuerySet):
 
     def _get_config_context_filters(self):
         # Construct the set of Q objects for the specific object types
+        tag_query_filters = {
+            "object_id": OuterRef(OuterRef('pk')),
+            "content_type__app_label": self.model._meta.app_label,
+            "content_type__model": self.model._meta.model_name
+        }
         base_query = Q(
             Q(platforms=OuterRef('platform')) | Q(platforms=None),
             Q(tenant_groups=OuterRef('tenant__group')) | Q(tenant_groups=None),
             Q(tenants=OuterRef('tenant')) | Q(tenants=None),
-            Q(tags=OuterRef('tags')) | Q(tags=None),
+            Q(
+                tags__pk__in=Subquery(
+                    TaggedItem.objects.filter(
+                        **tag_query_filters
+                    ).values_list(
+                        'tag_id',
+                        flat=True
+                    )
+                )
+            ) | Q(tags=None),
             is_active=True,
         )
 

netbox/extras/tests/test_models.py

@@ -363,3 +363,46 @@ class ConfigContextTest(TestCase):
         annotated_queryset = Device.objects.filter(name=device.name).annotate_config_context_data()
         self.assertEqual(ConfigContext.objects.get_for_object(device).count(), 1)
         self.assertEqual(device.get_config_context(), annotated_queryset[0].get_config_context())
+
+    def test_multiple_tags_return_distinct_objects_with_seperate_config_contexts(self):
+        """
+        Tagged items use a generic relationship, which results in duplicate rows being returned when queried.
+        This is combatted by by appending distinct() to the config context querysets. This test creates a config
+        context assigned to two tags and ensures objects related by those same two tags result in only a single
+        config context record being returned.
+
+        This test case is seperate from the above in that it deals with multiple config context objects in play.
+
+        See https://github.com/netbox-community/netbox/issues/5387
+        """
+        tag_context_1 = ConfigContext.objects.create(
+            name="tag-1",
+            weight=100,
+            data={
+                "tag": 1
+            }
+        )
+        tag_context_1.tags.add(self.tag)
+        tag_context_2 = ConfigContext.objects.create(
+            name="tag-2",
+            weight=100,
+            data={
+                "tag": 1
+            }
+        )
+        tag_context_2.tags.add(self.tag2)
+
+        device = Device.objects.create(
+            name="Device 3",
+            site=self.site,
+            tenant=self.tenant,
+            platform=self.platform,
+            device_role=self.devicerole,
+            device_type=self.devicetype
+        )
+        device.tags.add(self.tag)
+        device.tags.add(self.tag2)
+
+        annotated_queryset = Device.objects.filter(name=device.name).annotate_config_context_data()
+        self.assertEqual(ConfigContext.objects.get_for_object(device).count(), 2)
+        self.assertEqual(device.get_config_context(), annotated_queryset[0].get_config_context())

netbox/extras/tests/test_plugins.py

@@ -86,21 +86,19 @@ class PluginTest(TestCase):
         """
         self.assertIn('extras.tests.dummy_plugin.*', settings.CACHEOPS)
 
-    @override_settings(VERSION='0.9')
     def test_min_version(self):
         """
         Check enforcement of minimum NetBox version.
         """
         with self.assertRaises(ImproperlyConfigured):
-            dummy_config.validate({})
+            dummy_config.validate({}, '0.9')
 
-    @override_settings(VERSION='10.0')
     def test_max_version(self):
         """
         Check enforcement of maximum NetBox version.
         """
         with self.assertRaises(ImproperlyConfigured):
-            dummy_config.validate({})
+            dummy_config.validate({}, '10.0')
 
     def test_required_settings(self):
         """
@@ -110,11 +108,11 @@ class PluginTest(TestCase):
             required_settings = ['foo']
 
         # Validation should pass when all required settings are present
-        DummyConfigWithRequiredSettings.validate({'foo': True})
+        DummyConfigWithRequiredSettings.validate({'foo': True}, settings.VERSION)
 
         # Validation should fail when a required setting is missing
         with self.assertRaises(ImproperlyConfigured):
-            DummyConfigWithRequiredSettings.validate({})
+            DummyConfigWithRequiredSettings.validate({}, settings.VERSION)
 
     def test_default_settings(self):
         """
@@ -127,10 +125,10 @@ class PluginTest(TestCase):
 
         # Populate the default value if setting has not been specified
         user_config = {}
-        DummyConfigWithDefaultSettings.validate(user_config)
+        DummyConfigWithDefaultSettings.validate(user_config, settings.VERSION)
         self.assertEqual(user_config['bar'], 123)
 
         # Don't overwrite specified values
         user_config = {'bar': 456}
-        DummyConfigWithDefaultSettings.validate(user_config)
+        DummyConfigWithDefaultSettings.validate(user_config, settings.VERSION)
         self.assertEqual(user_config['bar'], 456)

netbox/extras/views.py

@@ -1,7 +1,7 @@
 from django import template
 from django.contrib import messages
 from django.contrib.contenttypes.models import ContentType
-from django.db.models import Count, Prefetch, Q
+from django.db.models import Prefetch, Q
 from django.http import Http404, HttpResponseForbidden
 from django.shortcuts import get_object_or_404, redirect, render
 from django.views.generic import View
@@ -13,7 +13,7 @@ from dcim.models import DeviceRole, Platform, Region, Site
 from tenancy.models import Tenant, TenantGroup
 from utilities.forms import ConfirmationForm
 from utilities.paginator import EnhancedPaginator, get_paginate_count
-from utilities.utils import copy_safe_request, shallow_compare_dict
+from utilities.utils import copy_safe_request, get_subquery, shallow_compare_dict
 from utilities.views import (
     BulkDeleteView, BulkEditView, BulkImportView, ObjectView, ObjectDeleteView, ObjectEditView, ObjectListView,
     ContentTypePermissionRequiredMixin,
@@ -21,7 +21,7 @@ from utilities.views import (
 from virtualization.models import Cluster, ClusterGroup
 from . import filters, forms, tables
 from .choices import JobResultStatusChoices
-from .models import ConfigContext, ImageAttachment, ObjectChange, JobResult, Tag
+from .models import ConfigContext, ImageAttachment, ObjectChange, JobResult, Tag, TaggedItem
 from .reports import get_report, get_reports, run_report
 from .scripts import get_scripts, run_script
 
@@ -32,8 +32,8 @@ from .scripts import get_scripts, run_script
 
 class TagListView(ObjectListView):
     queryset = Tag.objects.annotate(
-        items=Count('extras_taggeditem_items')
+        items=get_subquery(TaggedItem, 'tag')
-    ).order_by(*Tag._meta.ordering)
+    )
     filterset = filters.TagFilterSet
     filterset_form = forms.TagFilterForm
     table = tables.TagTable
@@ -57,16 +57,16 @@ class TagBulkImportView(BulkImportView):
 
 class TagBulkEditView(BulkEditView):
     queryset = Tag.objects.annotate(
-        items=Count('extras_taggeditem_items')
+        items=get_subquery(TaggedItem, 'tag')
-    ).order_by(*Tag._meta.ordering)
+    )
     table = tables.TagTable
     form = forms.TagBulkEditForm
 
 
 class TagBulkDeleteView(BulkDeleteView):
     queryset = Tag.objects.annotate(
-        items=Count('extras_taggeditem_items')
+        items=get_subquery(TaggedItem, 'tag')
-    ).order_by(*Tag._meta.ordering)
+    )
     table = tables.TagTable
 
 

netbox/ipam/api/views.py

@@ -1,5 +1,4 @@
 from django.conf import settings
-from django.db.models import Count
 from django.shortcuts import get_object_or_404
 from django_pglocks import advisory_lock
 from drf_yasg.utils import swagger_auto_schema
@@ -33,7 +32,7 @@ class VRFViewSet(CustomFieldModelViewSet):
     queryset = VRF.objects.prefetch_related('tenant').prefetch_related('tags').annotate(
         ipaddress_count=get_subquery(IPAddress, 'vrf'),
         prefix_count=get_subquery(Prefix, 'vrf')
-    ).order_by(*VRF._meta.ordering)
+    )
     serializer_class = serializers.VRFSerializer
     filterset_class = filters.VRFFilterSet
 
@@ -44,8 +43,8 @@ class VRFViewSet(CustomFieldModelViewSet):
 
 class RIRViewSet(ModelViewSet):
     queryset = RIR.objects.annotate(
-        aggregate_count=Count('aggregates')
+        aggregate_count=get_subquery(Aggregate, 'rir')
-    ).order_by(*RIR._meta.ordering)
+    )
     serializer_class = serializers.RIRSerializer
     filterset_class = filters.RIRFilterSet
 
@@ -68,7 +67,7 @@ class RoleViewSet(ModelViewSet):
     queryset = Role.objects.annotate(
         prefix_count=get_subquery(Prefix, 'role'),
         vlan_count=get_subquery(VLAN, 'role')
-    ).order_by(*Role._meta.ordering)
+    )
     serializer_class = serializers.RoleSerializer
     filterset_class = filters.RoleFilterSet
 
@@ -80,7 +79,7 @@ class RoleViewSet(ModelViewSet):
 class PrefixViewSet(CustomFieldModelViewSet):
     queryset = Prefix.objects.prefetch_related(
         'site', 'vrf__tenant', 'tenant', 'vlan', 'role', 'tags'
-    ).order_by(*Prefix._meta.ordering)
+    )
     serializer_class = serializers.PrefixSerializer
     filterset_class = filters.PrefixFilterSet
 
@@ -250,7 +249,7 @@ class PrefixViewSet(CustomFieldModelViewSet):
 class IPAddressViewSet(CustomFieldModelViewSet):
     queryset = IPAddress.objects.prefetch_related(
         'vrf__tenant', 'tenant', 'nat_inside', 'nat_outside', 'tags', 'assigned_object'
-    ).order_by(*IPAddress._meta.ordering)
+    )
     serializer_class = serializers.IPAddressSerializer
     filterset_class = filters.IPAddressFilterSet
 
@@ -261,8 +260,8 @@ class IPAddressViewSet(CustomFieldModelViewSet):
 
 class VLANGroupViewSet(ModelViewSet):
     queryset = VLANGroup.objects.prefetch_related('site').annotate(
-        vlan_count=Count('vlans')
+        vlan_count=get_subquery(VLAN, 'group')
-    ).order_by(*VLANGroup._meta.ordering)
+    )
     serializer_class = serializers.VLANGroupSerializer
     filterset_class = filters.VLANGroupFilterSet
 
@@ -276,7 +275,7 @@ class VLANViewSet(CustomFieldModelViewSet):
         'site', 'group', 'tenant', 'role', 'tags'
     ).annotate(
         prefix_count=get_subquery(Prefix, 'vlan')
-    ).order_by(*VLAN._meta.ordering)
+    )
     serializer_class = serializers.VLANSerializer
     filterset_class = filters.VLANFilterSet
 

netbox/ipam/views.py

@@ -1,6 +1,5 @@
 import netaddr
-from django.conf import settings
+from django.db.models import Prefetch
-from django.db.models import Count, Prefetch
 from django.db.models.expressions import RawSQL
 from django.shortcuts import get_object_or_404, redirect, render
 from django_tables2 import RequestConfig
@@ -79,7 +78,9 @@ class VRFBulkDeleteView(BulkDeleteView):
 #
 
 class RIRListView(ObjectListView):
-    queryset = RIR.objects.annotate(aggregate_count=Count('aggregates')).order_by(*RIR._meta.ordering)
+    queryset = RIR.objects.annotate(
+        aggregate_count=get_subquery(Aggregate, 'rir')
+    )
     filterset = filters.RIRFilterSet
     filterset_form = forms.RIRFilterForm
     table = tables.RIRDetailTable
@@ -172,7 +173,9 @@ class RIRBulkImportView(BulkImportView):
 
 
 class RIRBulkDeleteView(BulkDeleteView):
-    queryset = RIR.objects.annotate(aggregate_count=Count('aggregates')).order_by(*RIR._meta.ordering)
+    queryset = RIR.objects.annotate(
+        aggregate_count=get_subquery(Aggregate, 'rir')
+    )
     filterset = filters.RIRFilterSet
     table = tables.RIRTable
 
@@ -184,7 +187,7 @@ class RIRBulkDeleteView(BulkDeleteView):
 class AggregateListView(ObjectListView):
     queryset = Aggregate.objects.prefetch_related('rir').annotate(
         child_count=RawSQL('SELECT COUNT(*) FROM ipam_prefix WHERE ipam_prefix.prefix <<= ipam_aggregate.prefix', ())
-    ).order_by(*Aggregate._meta.ordering)
+    )
     filterset = filters.AggregateFilterSet
     filterset_form = forms.AggregateFilterForm
     table = tables.AggregateDetailTable
@@ -652,8 +655,8 @@ class IPAddressBulkDeleteView(BulkDeleteView):
 
 class VLANGroupListView(ObjectListView):
     queryset = VLANGroup.objects.prefetch_related('site').annotate(
-        vlan_count=Count('vlans')
+        vlan_count=get_subquery(VLAN, 'group')
-    ).order_by(*VLANGroup._meta.ordering)
+    )
     filterset = filters.VLANGroupFilterSet
     filterset_form = forms.VLANGroupFilterForm
     table = tables.VLANGroupTable
@@ -676,8 +679,8 @@ class VLANGroupBulkImportView(BulkImportView):
 
 class VLANGroupBulkDeleteView(BulkDeleteView):
     queryset = VLANGroup.objects.prefetch_related('site').annotate(
-        vlan_count=Count('vlans')
+        vlan_count=get_subquery(VLAN, 'group')
-    ).order_by(*VLANGroup._meta.ordering)
+    )
     filterset = filters.VLANGroupFilterSet
     table = tables.VLANGroupTable
 

netbox/netbox/configuration.testing.py

@@ -7,8 +7,8 @@ ALLOWED_HOSTS = ['*']
 
 DATABASE = {
     'NAME': 'netbox',
-    'USER': '',
+    'USER': 'netbox',
-    'PASSWORD': '',
+    'PASSWORD': 'netbox',
     'HOST': 'localhost',
     'PORT': '',
     'CONN_MAX_AGE': 300,

netbox/netbox/settings.py

@@ -16,7 +16,7 @@ from django.core.validators import URLValidator
 # Environment setup
 #
 
-VERSION = '2.9.10'
+VERSION = '2.9.11'
 
 # Hostname
 HOSTNAME = platform.node()
@@ -621,7 +621,7 @@ for plugin_name in PLUGINS:
     # Validate user-provided configuration settings and assign defaults
     if plugin_name not in PLUGINS_CONFIG:
         PLUGINS_CONFIG[plugin_name] = {}
-    plugin_config.validate(PLUGINS_CONFIG[plugin_name])
+    plugin_config.validate(PLUGINS_CONFIG[plugin_name], VERSION)
 
     # Add middleware
     plugin_middleware = plugin_config.middleware

netbox/netbox/views.py

@@ -48,8 +48,8 @@ SEARCH_TYPES = OrderedDict((
     # Circuits
     ('provider', {
         'queryset': Provider.objects.annotate(
-            count_circuits=Count('circuits')
+            count_circuits=get_subquery(Circuit, 'provider')
-        ).order_by(*Provider._meta.ordering),
+        ),
         'filterset': ProviderFilterSet,
         'table': ProviderTable,
         'url': 'circuits:provider_list',
@@ -76,17 +76,21 @@ SEARCH_TYPES = OrderedDict((
         'url': 'dcim:rack_list',
     }),
     ('rackgroup', {
-        'queryset': RackGroup.objects.prefetch_related('site').annotate(
+        'queryset': RackGroup.objects.add_related_count(
-            rack_count=Count('racks')
+            RackGroup.objects.all(),
-        ).order_by(*RackGroup._meta.ordering),
+            Rack,
+            'group',
+            'rack_count',
+            cumulative=True
+        ).prefetch_related('site'),
         'filterset': RackGroupFilterSet,
         'table': RackGroupTable,
         'url': 'dcim:rackgroup_list',
     }),
     ('devicetype', {
         'queryset': DeviceType.objects.prefetch_related('manufacturer').annotate(
-            instance_count=Count('instances')
+            instance_count=get_subquery(Device, 'device_type')
-        ).order_by(*DeviceType._meta.ordering),
+        ),
         'filterset': DeviceTypeFilterSet,
         'table': DeviceTypeTable,
         'url': 'dcim:devicetype_list',
@@ -101,8 +105,8 @@ SEARCH_TYPES = OrderedDict((
     }),
     ('virtualchassis', {
         'queryset': VirtualChassis.objects.prefetch_related('master').annotate(
-            member_count=Count('members', distinct=True)
+            member_count=get_subquery(Device, 'virtual_chassis')
-        ).order_by(*VirtualChassis._meta.ordering),
+        ),
         'filterset': VirtualChassisFilterSet,
         'table': VirtualChassisTable,
         'url': 'dcim:virtualchassis_list',

netbox/secrets/api/views.py

@@ -1,7 +1,6 @@
 import base64
 
 from Crypto.PublicKey import RSA
-from django.db.models import Count
 from django.http import HttpResponseBadRequest
 from rest_framework.exceptions import ValidationError
 from rest_framework.permissions import IsAuthenticated
@@ -13,6 +12,7 @@ from secrets import filters
 from secrets.exceptions import InvalidKey
 from secrets.models import Secret, SecretRole, SessionKey, UserKey
 from utilities.api import ModelViewSet
+from utilities.utils import get_subquery
 from . import serializers
 
 ERR_USERKEY_MISSING = "No UserKey found for the current user."
@@ -35,8 +35,8 @@ class SecretsRootView(APIRootView):
 
 class SecretRoleViewSet(ModelViewSet):
     queryset = SecretRole.objects.annotate(
-        secret_count=Count('secrets')
+        secret_count=get_subquery(Secret, 'role')
-    ).order_by(*SecretRole._meta.ordering)
+    )
     serializer_class = serializers.SecretRoleSerializer
     filterset_class = filters.SecretRoleFilterSet
 

netbox/secrets/tables.py

@@ -28,12 +28,20 @@ class SecretRoleTable(BaseTable):
 
 class SecretTable(BaseTable):
     pk = ToggleColumn()
-    device = tables.LinkColumn()
+    id = tables.Column(
+        linkify=True
+    )
+    device = tables.Column(
+        linkify=True
+    )
+    role = tables.Column(
+        linkify=True
+    )
     tags = TagColumn(
         url_name='secrets:secret_list'
     )
 
     class Meta(BaseTable.Meta):
         model = Secret
-        fields = ('pk', 'device', 'role', 'name', 'last_updated', 'hash', 'tags')
+        fields = ('pk', 'id', 'device', 'role', 'name', 'last_updated', 'hash', 'tags')
-        default_columns = ('pk', 'device', 'role', 'name', 'last_updated')
+        default_columns = ('pk', 'id', 'device', 'role', 'name', 'last_updated')

netbox/secrets/views.py

@@ -2,11 +2,11 @@ import base64
 import logging
 
 from django.contrib import messages
-from django.db.models import Count
 from django.shortcuts import get_object_or_404, redirect, render
 from django.utils.html import escape
 from django.utils.safestring import mark_safe
 
+from utilities.utils import get_subquery
 from utilities.views import (
     BulkDeleteView, BulkEditView, BulkImportView, ObjectView, ObjectDeleteView, ObjectEditView, ObjectListView,
 )
@@ -29,7 +29,9 @@ def get_session_key(request):
 #
 
 class SecretRoleListView(ObjectListView):
-    queryset = SecretRole.objects.annotate(secret_count=Count('secrets')).order_by(*SecretRole._meta.ordering)
+    queryset = SecretRole.objects.annotate(
+        secret_count=get_subquery(Secret, 'role')
+    )
     table = tables.SecretRoleTable
 
 
@@ -49,7 +51,9 @@ class SecretRoleBulkImportView(BulkImportView):
 
 
 class SecretRoleBulkDeleteView(BulkDeleteView):
-    queryset = SecretRole.objects.annotate(secret_count=Count('secrets')).order_by(*SecretRole._meta.ordering)
+    queryset = SecretRole.objects.annotate(
+        secret_count=get_subquery(Secret, 'role')
+    )
     table = tables.SecretRoleTable
 
 
@@ -104,13 +108,14 @@ class SecretEditView(ObjectEditView):
 
         if form.is_valid():
             logger.debug("Form validation was successful")
+            secret = form.save(commit=False)
 
-            # We must have a session key in order to create a secret or update the plaintext of an existing secret
+            # We must have a session key in order to set the plaintext of a Secret
-            if (form.cleaned_data['plaintext'] or secret.pk is None) and session_key is None:
+            if form.cleaned_data['plaintext'] and session_key is None:
                 logger.debug("Unable to proceed: No session key was provided with the request")
                 form.add_error(None, "No session key was provided with the request. Unable to encrypt secret data.")
 
-            else:
+            elif form.cleaned_data['plaintext']:
                 master_key = None
                 try:
                     sk = SessionKey.objects.get(userkey__user=request.user)
@@ -121,19 +126,18 @@ class SecretEditView(ObjectEditView):
 
                 if master_key is not None:
                     logger.debug("Successfully resolved master key for encryption")
-                    secret = form.save(commit=False)
+                    secret.plaintext = str(form.cleaned_data['plaintext'])
-                    if form.cleaned_data['plaintext']:
-                        secret.plaintext = str(form.cleaned_data['plaintext'])
                     secret.encrypt(master_key)
-                    secret.save()
-                    form.save_m2m()
 
-                    msg = '{} secret'.format('Created' if not form.instance.pk else 'Modified')
+            secret.save()
-                    logger.info(f"{msg} {secret} (PK: {secret.pk})")
+            form.save_m2m()
-                    msg = '{} <a href="{}">{}</a>'.format(msg, secret.get_absolute_url(), escape(secret))
-                    messages.success(request, mark_safe(msg))
 
-                    return redirect(self.get_return_url(request, secret))
+            msg = '{} secret'.format('Created' if not form.instance.pk else 'Modified')
+            logger.info(f"{msg} {secret} (PK: {secret.pk})")
+            msg = f'{msg} <a href="{secret.get_absolute_url()}">{escape(secret)}</a>'
+            messages.success(request, mark_safe(msg))
+
+            return redirect(self.get_return_url(request, secret))
 
         else:
             logger.debug("Form validation failed")

netbox/users/api/serializers.py

@@ -19,9 +19,23 @@ class UserSerializer(ValidatedModelSerializer):
     class Meta:
         model = User
         fields = (
-            'id', 'url', 'username', 'first_name', 'last_name', 'email', 'is_staff', 'is_active', 'date_joined',
+            'id', 'url', 'username', 'password', 'first_name', 'last_name', 'email', 'is_staff', 'is_active',
-            'groups',
+            'date_joined', 'groups',
         )
+        extra_kwargs = {
+            'password': {'write_only': True}
+        }
+
+    def create(self, validated_data):
+        """
+        Extract the password from validated data and set it separately to ensure proper hash generation.
+        """
+        password = validated_data.pop('password')
+        user = super().create(validated_data)
+        user.set_password(password)
+        user.save()
+
+        return user
 
 
 class GroupSerializer(ValidatedModelSerializer):

netbox/users/tests/test_api.py

@@ -22,15 +22,19 @@ class UserTest(APIViewTestCases.APIViewTestCase):
     model = User
     view_namespace = 'users'
     brief_fields = ['id', 'url', 'username']
+    validation_excluded_fields = ['password']
     create_data = [
         {
             'username': 'User_4',
+            'password': 'password4',
         },
         {
             'username': 'User_5',
+            'password': 'password5',
         },
         {
             'username': 'User_6',
+            'password': 'password6',
         },
     ]
 

netbox/utilities/testing/api.py

@@ -174,6 +174,7 @@ class APIViewTestCases:
 
     class CreateObjectViewTestCase(APITestCase):
         create_data = []
+        validation_excluded_fields = []
 
         def test_create_object_without_permission(self):
             """
@@ -205,6 +206,7 @@ class APIViewTestCases:
             self.assertInstanceEqual(
                 self._get_queryset().get(pk=response.data['id']),
                 self.create_data[0],
+                exclude=self.validation_excluded_fields,
                 api=True
             )
 
@@ -229,11 +231,13 @@ class APIViewTestCases:
                 self.assertInstanceEqual(
                     self._get_queryset().get(pk=obj['id']),
                     self.create_data[i],
+                    exclude=self.validation_excluded_fields,
                     api=True
                 )
 
     class UpdateObjectViewTestCase(APITestCase):
         update_data = {}
+        validation_excluded_fields = []
 
         def test_update_object_without_permission(self):
             """
@@ -266,7 +270,12 @@ class APIViewTestCases:
             response = self.client.patch(url, update_data, format='json', **self.header)
             self.assertHttpStatus(response, status.HTTP_200_OK)
             instance.refresh_from_db()
-            self.assertInstanceEqual(instance, update_data, api=True)
+            self.assertInstanceEqual(
+                instance,
+                update_data,
+                exclude=self.validation_excluded_fields,
+                api=True
+            )
 
     class DeleteObjectViewTestCase(APITestCase):
 

netbox/utilities/testing/views.py

@@ -126,20 +126,25 @@ class TestCase(_TestCase):
             err_message = f"Expected HTTP status {expected_status}; received {response.status_code}: {err}"
         self.assertEqual(response.status_code, expected_status, err_message)
 
-    def assertInstanceEqual(self, instance, data, api=False):
+    def assertInstanceEqual(self, instance, data, exclude=None, api=False):
         """
         Compare a model instance to a dictionary, checking that its attribute values match those specified
         in the dictionary.
 
-        :instance: Python object instance
+        :param instance: Python object instance
-        :data: Dictionary of test data used to define the instance
+        :param data: Dictionary of test data used to define the instance
-        :api: Set to True is the data is a JSON representation of the instance
+        :param exclude: List of fields to exclude from comparison (e.g. passwords, which get hashed)
+        :param api: Set to True is the data is a JSON representation of the instance
         """
-        model_dict = self.model_to_dict(instance, fields=data.keys(), api=api)
+        if exclude is None:
+            exclude = []
 
-        # Omit any dictionary keys which are not instance attributes
+        fields = [k for k in data.keys() if k not in exclude]
+        model_dict = self.model_to_dict(instance, fields=fields, api=api)
+
+        # Omit any dictionary keys which are not instance attributes or have been excluded
         relevant_data = {
-            k: v for k, v in data.items() if hasattr(instance, k)
+            k: v for k, v in data.items() if hasattr(instance, k) and k not in exclude
         }
 
         self.assertDictEqual(model_dict, relevant_data)

netbox/virtualization/api/views.py

@@ -1,4 +1,3 @@
-from django.db.models import Count
 from django.shortcuts import get_object_or_404
 from rest_framework.decorators import action
 from rest_framework.response import Response
@@ -29,16 +28,16 @@ class VirtualizationRootView(APIRootView):
 
 class ClusterTypeViewSet(ModelViewSet):
     queryset = ClusterType.objects.annotate(
-        cluster_count=Count('clusters')
+        cluster_count=get_subquery(Cluster, 'type')
-    ).order_by(*ClusterType._meta.ordering)
+    )
     serializer_class = serializers.ClusterTypeSerializer
     filterset_class = filters.ClusterTypeFilterSet
 
 
 class ClusterGroupViewSet(ModelViewSet):
     queryset = ClusterGroup.objects.annotate(
-        cluster_count=Count('clusters')
+        cluster_count=get_subquery(Cluster, 'group')
-    ).order_by(*ClusterGroup._meta.ordering)
+    )
     serializer_class = serializers.ClusterGroupSerializer
     filterset_class = filters.ClusterGroupFilterSet
 
@@ -49,7 +48,7 @@ class ClusterViewSet(CustomFieldModelViewSet):
     ).annotate(
         device_count=get_subquery(Device, 'cluster'),
         virtualmachine_count=get_subquery(VirtualMachine, 'cluster')
-    ).order_by(*Cluster._meta.ordering)
+    )
     serializer_class = serializers.ClusterSerializer
     filterset_class = filters.ClusterFilterSet
 

netbox/virtualization/models.py

@@ -319,10 +319,10 @@ class VirtualMachine(ChangeLoggedModel, ConfigContextModel, CustomFieldModel):
         # because Django does not consider two NULL fields to be equal, and thus will not trigger a violation
         # of the uniqueness constraint without manual intervention.
         if self.tenant is None and VirtualMachine.objects.exclude(pk=self.pk).filter(
-                name=self.name, tenant__isnull=True
+                name=self.name, cluster=self.cluster, tenant__isnull=True
         ):
             raise ValidationError({
-                'name': 'A virtual machine with this name already exists.'
+                'name': 'A virtual machine with this name already exists in the assigned cluster.'
             })
 
         super().validate_unique(exclude)

netbox/virtualization/views.py

@@ -1,6 +1,6 @@
 from django.contrib import messages
 from django.db import transaction
-from django.db.models import Count, Prefetch
+from django.db.models import Prefetch
 from django.shortcuts import get_object_or_404, redirect, render
 from django.urls import reverse
 
@@ -23,7 +23,9 @@ from .models import Cluster, ClusterGroup, ClusterType, VirtualMachine, VMInterf
 #
 
 class ClusterTypeListView(ObjectListView):
-    queryset = ClusterType.objects.annotate(cluster_count=Count('clusters')).order_by(*ClusterType._meta.ordering)
+    queryset = ClusterType.objects.annotate(
+        cluster_count=get_subquery(Cluster, 'type')
+    )
     table = tables.ClusterTypeTable
 
 
@@ -43,7 +45,9 @@ class ClusterTypeBulkImportView(BulkImportView):
 
 
 class ClusterTypeBulkDeleteView(BulkDeleteView):
-    queryset = ClusterType.objects.annotate(cluster_count=Count('clusters')).order_by(*ClusterType._meta.ordering)
+    queryset = ClusterType.objects.annotate(
+        cluster_count=get_subquery(Cluster, 'type')
+    )
     table = tables.ClusterTypeTable
 
 
@@ -52,7 +56,9 @@ class ClusterTypeBulkDeleteView(BulkDeleteView):
 #
 
 class ClusterGroupListView(ObjectListView):
-    queryset = ClusterGroup.objects.annotate(cluster_count=Count('clusters')).order_by(*ClusterGroup._meta.ordering)
+    queryset = ClusterGroup.objects.annotate(
+        cluster_count=get_subquery(Cluster, 'group')
+    )
     table = tables.ClusterGroupTable
 
 
@@ -72,7 +78,9 @@ class ClusterGroupBulkImportView(BulkImportView):
 
 
 class ClusterGroupBulkDeleteView(BulkDeleteView):
-    queryset = ClusterGroup.objects.annotate(cluster_count=Count('clusters')).order_by(*ClusterGroup._meta.ordering)
+    queryset = ClusterGroup.objects.annotate(
+        cluster_count=get_subquery(Cluster, 'group')
+    )
     table = tables.ClusterGroupTable
 
 

scripts/cibuild.sh

@@ -1,60 +0,0 @@
-#!/bin/bash
-
-# Exit code starts at 0 but is modified if any checks fail
-EXIT=0
-
-# Output a line prefixed with a timestamp
-info()
-{
-	echo "$(date +'%F %T') |"
-}
-
-# Track number of seconds required to run script
-START=$(date +%s)
-echo "$(info) starting build checks."
-
-# Syntax check all python source files
-SYNTAX=$(find . -name "*.py" -type f -exec python -m py_compile {} \; 2>&1)
-if [[ ! -z $SYNTAX ]]; then
-	echo -e "$SYNTAX"
-	echo -e "\n$(info) detected one or more syntax errors, failing build."
-	EXIT=1
-fi
-
-# Check all python source files for PEP 8 compliance, but explicitly
-# ignore:
-#  - W504: line break after binary operator
-#  - E501: line greater than 80 characters in length
-pycodestyle \
-    --ignore=W504,E501 \
-    netbox/
-RC=$?
-if [[ $RC != 0 ]]; then
-	echo -e "\n$(info) one or more PEP 8 errors detected, failing build."
-	EXIT=$RC
-fi
-
-# Point to the testing configuration file for use in CI
-ln -s configuration.testing.py netbox/netbox/configuration.py
-
-# Run NetBox tests
-coverage run --source="netbox/" netbox/manage.py test netbox/
-RC=$?
-if [[ $RC != 0 ]]; then
-	echo -e "\n$(info) one or more tests failed, failing build."
-	EXIT=$RC
-fi
-
-# Show code coverage report
-coverage report --skip-covered --omit *migrations*
-RC=$?
-if [[ $RC != 0 ]]; then
-	echo -e "\n$(info) failed to generate code coverage report."
-	EXIT=$RC
-fi
-
-# Show build duration
-END=$(date +%s)
-echo "$(info) exiting with code $EXIT after $(($END - $START)) seconds."
-
-exit $EXIT