NeoAnd/netbox
1
0
Fork 0
mirror of https://github.com/netbox-community/netbox.git synced 2025-07-14 09:51:22 -06:00
Code Issues Actions 8 Packages Projects Releases Wiki Activity

Closes #10560: New global search (#10676)

Browse Source 
* Initial work on new search backend

* Clean up search backends

* Return only the most relevant result per object

* Clear any pre-existing cached entries on cache()

* #6003: Implement global search functionality for custom field values

* Tweak field weights & document guidance

* Extend search() to accept a lookup type

* Move get_registry() out of SearchBackend

* Enforce object permissions when returning search results

* Add indexers for remaining models

* Avoid calling remove() on non-cacheable objects

* Use new search backend by default

* Extend search backend to filter by object type

* Clean up search view form

* Enable specifying lookup logic

* Add indexes for value field

* Remove object type selector from search bar

* Introduce SearchTable and enable HTMX for results

* Enable pagination

* Remove legacy search backend

* Cleanup

* Use a UUID for CachedValue primary key

* Refactoring search methods

* Define max search results limit

* Extend reindex command to support specifying particular models

* Add clear() and size to SearchBackend

* Optimize bulk caching performance

* Highlight matched portion of field value

* Performance improvements for reindexing

* Started on search tests

* Cleanup & docs

* Documentation updates

* Clean up SearchIndex

* Flatten search registry to register by app_label.model_name

* Clean up search backend classes

* Clean up RestrictedGenericForeignKey and RestrictedPrefetch

* Resolve migrations conflict
This commit is contained in:
Jeremy Stretch 2022-10-21 13:16:16 -04:00 committed by GitHub
parent 5d56d95fda
commit 9628dead07
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
50 changed files with 1571 additions and 667 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
docs/configuration/system.md
View File

@ -157,6 +157,14 @@ The file path to the location where [custom scripts](../customization/custom-scr
--- ---
## SEARCH_BACKEND
Default: `'netbox.search.backends.CachedValueSearchBackend'`
The dotted path to the desired search backend class. `CachedValueSearchBackend` is currently the only search backend provided in NetBox, however this setting can be used to enable a custom backend.
---
## STORAGE_BACKEND ## STORAGE_BACKEND
Default: None (local storage) Default: None (local storage)

37
docs/development/search.md Normal file
View File

@ -0,0 +1,37 @@
# Search
NetBox v3.4 introduced a new global search mechanism, which employs the `extras.CachedValue` model to store discrete field values from many models in a single table.
## SearchIndex
To enable search support for a model, declare and register a subclass of `netbox.search.SearchIndex` for it. Typically, this will be done within an app's `search.py` module.
```python
from netbox.search import SearchIndex, register_search
@register_search
class MyModelIndex(SearchIndex):
model = MyModel
fields = (
('name', 100),
('description', 500),
('comments', 5000),
)
```
A SearchIndex subclass defines both its model and a list of two-tuples specifying which model fields to be indexed and the weight (precedence) associated with each. Guidance on weight assignment for fields is provided below.
### Field Weight Guidance
| Weight | Field Role | Examples |
|--------|--------------------------------------------------|----------------------------------------------------|
| 50 | Unique serialized attribute | Device.asset_tag |
| 60 | Unique serialized attribute (per related object) | Device.serial |
| 100 | Primary human identifier | Device.name, Circuit.cid, Cable.label |
| 110 | Slug | Site.slug |
| 200 | Secondary identifier | Provider.account, DeviceType.part_number |
| 300 | Highly unique descriptive attribute | CircuitTermination.xconnect_id, IPAddress.dns_name |
| 500 | Description | Site.description |
| 1000 | Custom field default | - |
| 2000 | Other discrete attribute | CircuitTermination.port_speed |
| 5000 | Comment field | Site.comments |

15
docs/plugins/development/search.md
View File

@ -4,17 +4,16 @@ Plugins can define and register their own models to extend NetBox's core search
```python ```python
# search.py # search.py
from netbox.search import SearchMixin from netbox.search import SearchIndex
from .filters import MyModelFilterSet
from .tables import MyModelTable
from .models import MyModel from .models import MyModel
class MyModelIndex(SearchMixin): class MyModelIndex(SearchIndex):
model = MyModel model = MyModel
queryset = MyModel.objects.all() fields = (
filterset = MyModelFilterSet ('name', 100),
table = MyModelTable ('description', 500),
url = 'plugins:myplugin:mymodel_list' ('comments', 5000),
)
``` ```
To register one or more indexes with NetBox, define a list named `indexes` at the end of this file: To register one or more indexes with NetBox, define a list named `indexes` at the end of this file:

4
docs/release-notes/version-3.4.md
View File

@ -11,6 +11,10 @@
### New Features ### New Features
#### New Global Search ([#10560](https://github.com/netbox-community/netbox/issues/10560))
NetBox's global search functionality has been completely overhauled and replaced by a new cache-based lookup.
#### Top-Level Plugin Navigation Menus ([#9071](https://github.com/netbox-community/netbox/issues/9071)) #### Top-Level Plugin Navigation Menus ([#9071](https://github.com/netbox-community/netbox/issues/9071))
A new `PluginMenu` class has been introduced, which enables a plugin to inject a top-level menu in NetBox's navigation menu. This menu can have one or more groups of menu items, just like core items. Backward compatibility with the existing `menu_items` has been maintained. A new `PluginMenu` class has been introduced, which enables a plugin to inject a top-level menu in NetBox's navigation menu. This menu can have one or more groups of menu items, just like core items. Backward compatibility with the existing `menu_items` has been maintained.

1
mkdocs.yml
View File

@ -245,6 +245,7 @@ nav:
- Adding Models: 'development/adding-models.md' - Adding Models: 'development/adding-models.md'
- Extending Models: 'development/extending-models.md' - Extending Models: 'development/extending-models.md'
- Signals: 'development/signals.md' - Signals: 'development/signals.md'
- Search: 'development/search.md'
- Application Registry: 'development/application-registry.md' - Application Registry: 'development/application-registry.md'
- User Preferences: 'development/user-preferences.md' - User Preferences: 'development/user-preferences.md'
- Web UI: 'development/web-ui.md' - Web UI: 'development/web-ui.md'

73
netbox/circuits/search.py
View File

@ -1,34 +1,55 @@
import circuits.filtersets
import circuits.tables
from circuits.models import Circuit, Provider, ProviderNetwork
from netbox.search import SearchIndex, register_search from netbox.search import SearchIndex, register_search
from utilities.utils import count_related from . import models
@register_search() @register_search
class ProviderIndex(SearchIndex):
model = Provider
queryset = Provider.objects.annotate(count_circuits=count_related(Circuit, 'provider'))
filterset = circuits.filtersets.ProviderFilterSet
table = circuits.tables.ProviderTable
url = 'circuits:provider_list'
@register_search()
class CircuitIndex(SearchIndex): class CircuitIndex(SearchIndex):
model = Circuit model = models.Circuit
queryset = Circuit.objects.prefetch_related( fields = (
'type', 'provider', 'tenant', 'tenant__group', 'terminations__site' ('cid', 100),
('description', 500),
('comments', 5000),
) )
filterset = circuits.filtersets.CircuitFilterSet
table = circuits.tables.CircuitTable
url = 'circuits:circuit_list'
@register_search() @register_search
class CircuitTerminationIndex(SearchIndex):
model = models.CircuitTermination
fields = (
('xconnect_id', 300),
('pp_info', 300),
('description', 500),
('port_speed', 2000),
('upstream_speed', 2000),
)
@register_search
class CircuitTypeIndex(SearchIndex):
model = models.CircuitType
fields = (
('name', 100),
('slug', 110),
('description', 500),
)
@register_search
class ProviderIndex(SearchIndex):
model = models.Provider
fields = (
('name', 100),
('account', 200),
('comments', 5000),
)
@register_search
class ProviderNetworkIndex(SearchIndex): class ProviderNetworkIndex(SearchIndex):
model = ProviderNetwork model = models.ProviderNetwork
queryset = ProviderNetwork.objects.prefetch_related('provider') fields = (
filterset = circuits.filtersets.ProviderNetworkFilterSet ('name', 100),
table = circuits.tables.ProviderNetworkTable ('service_id', 200),
url = 'circuits:providernetwork_list' ('description', 500),
('comments', 5000),
)

422
netbox/dcim/search.py
View File

@ -1,143 +1,293 @@
import dcim.filtersets
import dcim.tables
from dcim.models import (
Cable,
Device,
DeviceType,
Location,
Module,
ModuleType,
PowerFeed,
Rack,
RackReservation,
Site,
VirtualChassis,
)
from netbox.search import SearchIndex, register_search from netbox.search import SearchIndex, register_search
from utilities.utils import count_related from . import models
@register_search() @register_search
class SiteIndex(SearchIndex):
model = Site
queryset = Site.objects.prefetch_related('region', 'tenant', 'tenant__group')
filterset = dcim.filtersets.SiteFilterSet
table = dcim.tables.SiteTable
url = 'dcim:site_list'
@register_search()
class RackIndex(SearchIndex):
model = Rack
queryset = Rack.objects.prefetch_related('site', 'location', 'tenant', 'tenant__group', 'role').annotate(
device_count=count_related(Device, 'rack')
)
filterset = dcim.filtersets.RackFilterSet
table = dcim.tables.RackTable
url = 'dcim:rack_list'
@register_search()
class RackReservationIndex(SearchIndex):
model = RackReservation
queryset = RackReservation.objects.prefetch_related('rack', 'user')
filterset = dcim.filtersets.RackReservationFilterSet
table = dcim.tables.RackReservationTable
url = 'dcim:rackreservation_list'
@register_search()
class LocationIndex(SearchIndex):
model = Location
queryset = Location.objects.add_related_count(
Location.objects.add_related_count(Location.objects.all(), Device, 'location', 'device_count', cumulative=True),
Rack,
'location',
'rack_count',
cumulative=True,
).prefetch_related('site')
filterset = dcim.filtersets.LocationFilterSet
table = dcim.tables.LocationTable
url = 'dcim:location_list'
@register_search()
class DeviceTypeIndex(SearchIndex):
model = DeviceType
queryset = DeviceType.objects.prefetch_related('manufacturer').annotate(
instance_count=count_related(Device, 'device_type')
)
filterset = dcim.filtersets.DeviceTypeFilterSet
table = dcim.tables.DeviceTypeTable
url = 'dcim:devicetype_list'
@register_search()
class DeviceIndex(SearchIndex):
model = Device
queryset = Device.objects.prefetch_related(
'device_type__manufacturer',
'device_role',
'tenant',
'tenant__group',
'site',
'rack',
'primary_ip4',
'primary_ip6',
)
filterset = dcim.filtersets.DeviceFilterSet
table = dcim.tables.DeviceTable
url = 'dcim:device_list'
@register_search()
class ModuleTypeIndex(SearchIndex):
model = ModuleType
queryset = ModuleType.objects.prefetch_related('manufacturer').annotate(
instance_count=count_related(Module, 'module_type')
)
filterset = dcim.filtersets.ModuleTypeFilterSet
table = dcim.tables.ModuleTypeTable
url = 'dcim:moduletype_list'
@register_search()
class ModuleIndex(SearchIndex):
model = Module
queryset = Module.objects.prefetch_related(
'module_type__manufacturer',
'device',
'module_bay',
)
filterset = dcim.filtersets.ModuleFilterSet
table = dcim.tables.ModuleTable
url = 'dcim:module_list'
@register_search()
class VirtualChassisIndex(SearchIndex):
model = VirtualChassis
queryset = VirtualChassis.objects.prefetch_related('master').annotate(
member_count=count_related(Device, 'virtual_chassis')
)
filterset = dcim.filtersets.VirtualChassisFilterSet
table = dcim.tables.VirtualChassisTable
url = 'dcim:virtualchassis_list'
@register_search()
class CableIndex(SearchIndex): class CableIndex(SearchIndex):
model = Cable model = models.Cable
queryset = Cable.objects.all() fields = (
filterset = dcim.filtersets.CableFilterSet ('label', 100),
table = dcim.tables.CableTable )
url = 'dcim:cable_list'
@register_search() @register_search
class ConsolePortIndex(SearchIndex):
model = models.ConsolePort
fields = (
('name', 100),
('label', 200),
('description', 500),
('speed', 2000),
)
@register_search
class ConsoleServerPortIndex(SearchIndex):
model = models.ConsoleServerPort
fields = (
('name', 100),
('label', 200),
('description', 500),
('speed', 2000),
)
@register_search
class DeviceIndex(SearchIndex):
model = models.Device
fields = (
('asset_tag', 50),
('serial', 60),
('name', 100),
('comments', 5000),
)
@register_search
class DeviceBayIndex(SearchIndex):
model = models.DeviceBay
fields = (
('name', 100),
('label', 200),
('description', 500),
)
@register_search
class DeviceRoleIndex(SearchIndex):
model = models.DeviceRole
fields = (
('name', 100),
('slug', 110),
('description', 500),
)
@register_search
class DeviceTypeIndex(SearchIndex):
model = models.DeviceType
fields = (
('model', 100),
('part_number', 200),
('comments', 5000),
)
@register_search
class FrontPortIndex(SearchIndex):
model = models.FrontPort
fields = (
('name', 100),
('label', 200),
('description', 500),
)
@register_search
class InterfaceIndex(SearchIndex):
model = models.Interface
fields = (
('name', 100),
('label', 200),
('mac_address', 300),
('wwn', 300),
('description', 500),
('mtu', 2000),
('speed', 2000),
)
@register_search
class InventoryItemIndex(SearchIndex):
model = models.InventoryItem
fields = (
('asset_tag', 50),
('serial', 60),
('name', 100),
('label', 200),
('description', 500),
('part_id', 2000),
)
@register_search
class LocationIndex(SearchIndex):
model = models.Location
fields = (
('name', 100),
('slug', 110),
('description', 500),
)
@register_search
class ManufacturerIndex(SearchIndex):
model = models.Manufacturer
fields = (
('name', 100),
('slug', 110),
('description', 500),
)
@register_search
class ModuleIndex(SearchIndex):
model = models.Module
fields = (
('asset_tag', 50),
('serial', 60),
('comments', 5000),
)
@register_search
class ModuleBayIndex(SearchIndex):
model = models.ModuleBay
fields = (
('name', 100),
('label', 200),
('description', 500),
)
@register_search
class ModuleTypeIndex(SearchIndex):
model = models.ModuleType
fields = (
('model', 100),
('part_number', 200),
('comments', 5000),
)
@register_search
class PlatformIndex(SearchIndex):
model = models.Platform
fields = (
('name', 100),
('slug', 110),
('napalm_driver', 300),
('description', 500),
)
@register_search
class PowerFeedIndex(SearchIndex): class PowerFeedIndex(SearchIndex):
model = PowerFeed model = models.PowerFeed
queryset = PowerFeed.objects.all() fields = (
filterset = dcim.filtersets.PowerFeedFilterSet ('name', 100),
table = dcim.tables.PowerFeedTable ('comments', 5000),
url = 'dcim:powerfeed_list' )
@register_search
class PowerOutletIndex(SearchIndex):
model = models.PowerOutlet
fields = (
('name', 100),
('label', 200),
('description', 500),
)
@register_search
class PowerPanelIndex(SearchIndex):
model = models.PowerPanel
fields = (
('name', 100),
)
@register_search
class PowerPortIndex(SearchIndex):
model = models.PowerPort
fields = (
('name', 100),
('label', 200),
('description', 500),
('maximum_draw', 2000),
('allocated_draw', 2000),
)
@register_search
class RackIndex(SearchIndex):
model = models.Rack
fields = (
('asset_tag', 50),
('serial', 60),
('name', 100),
('facility_id', 200),
('comments', 5000),
)
@register_search
class RackReservationIndex(SearchIndex):
model = models.RackReservation
fields = (
('description', 500),
)
@register_search
class RackRoleIndex(SearchIndex):
model = models.RackRole
fields = (
('name', 100),
('slug', 110),
('description', 500),
)
@register_search
class RearPortIndex(SearchIndex):
model = models.RearPort
fields = (
('name', 100),
('label', 200),
('description', 500),
)
@register_search
class RegionIndex(SearchIndex):
model = models.Region
fields = (
('name', 100),
('slug', 110),
('description', 500)
)
@register_search
class SiteIndex(SearchIndex):
model = models.Site
fields = (
('name', 100),
('facility', 100),
('slug', 110),
('description', 500),
('physical_address', 2000),
('shipping_address', 2000),
('comments', 5000),
)
@register_search
class SiteGroupIndex(SearchIndex):
model = models.SiteGroup
fields = (
('name', 100),
('slug', 110),
('description', 500)
)
@register_search
class VirtualChassisIndex(SearchIndex):
model = models.VirtualChassis
fields = (
('name', 100),
('domain', 300)
)

4
netbox/extras/api/serializers.py
View File

@ -92,8 +92,8 @@ class CustomFieldSerializer(ValidatedModelSerializer):
model = CustomField model = CustomField
fields = [ fields = [
'id', 'url', 'display', 'content_types', 'type', 'object_type', 'data_type', 'name', 'label', 'group_name', 'id', 'url', 'display', 'content_types', 'type', 'object_type', 'data_type', 'name', 'label', 'group_name',
'description', 'required', 'filter_logic', 'ui_visibility', 'default', 'weight', 'validation_minimum', 'description', 'required', 'search_weight', 'filter_logic', 'ui_visibility', 'default', 'weight',
'validation_maximum', 'validation_regex', 'choices', 'created', 'last_updated', 'validation_minimum', 'validation_maximum', 'validation_regex', 'choices', 'created', 'last_updated',
] ]
def get_data_type(self, obj): def get_data_type(self, obj):

4
netbox/extras/filtersets.py
View File

@ -73,8 +73,8 @@ class CustomFieldFilterSet(BaseFilterSet):
class Meta: class Meta:
model = CustomField model = CustomField
fields = [ fields = [
'id', 'content_types', 'name', 'group_name', 'required', 'filter_logic', 'ui_visibility', 'weight', 'id', 'content_types', 'name', 'group_name', 'required', 'search_weight', 'filter_logic', 'ui_visibility',
'description', 'weight', 'description',
] ]
def search(self, queryset, name, value): def search(self, queryset, name, value):

4
netbox/extras/forms/bulk_import.py
View File

@ -46,8 +46,8 @@ class CustomFieldCSVForm(CSVModelForm):
class Meta: class Meta:
model = CustomField model = CustomField
fields = ( fields = (
'name', 'label', 'group_name', 'type', 'content_types', 'object_type', 'required', 'description', 'weight', 'name', 'label', 'group_name', 'type', 'content_types', 'object_type', 'required', 'description',
'filter_logic', 'default', 'choices', 'weight', 'validation_minimum', 'validation_maximum', 'search_weight', 'filter_logic', 'default', 'choices', 'weight', 'validation_minimum', 'validation_maximum',
'validation_regex', 'ui_visibility', 'validation_regex', 'ui_visibility',
) )

4
netbox/extras/forms/models.py
View File

@ -41,9 +41,9 @@ class CustomFieldForm(BootstrapMixin, forms.ModelForm):
fieldsets = ( fieldsets = (
('Custom Field', ( ('Custom Field', (
'content_types', 'name', 'label', 'group_name', 'type', 'object_type', 'weight', 'required', 'description', 'content_types', 'name', 'label', 'group_name', 'type', 'object_type', 'required', 'description',
)), )),
('Behavior', ('filter_logic', 'ui_visibility')), ('Behavior', ('search_weight', 'filter_logic', 'ui_visibility', 'weight')),
('Values', ('default', 'choices')), ('Values', ('default', 'choices')),
('Validation', ('validation_minimum', 'validation_maximum', 'validation_regex')), ('Validation', ('validation_minimum', 'validation_maximum', 'validation_regex')),
) )

77
netbox/extras/management/commands/reindex.py Normal file
View File

@ -0,0 +1,77 @@
from django.contrib.contenttypes.models import ContentType
from django.core.management.base import BaseCommand, CommandError
from extras.registry import registry
from netbox.search.backends import search_backend
class Command(BaseCommand):
help = 'Reindex objects for search'
def add_arguments(self, parser):
parser.add_argument(
'args',
metavar='app_label[.ModelName]',
nargs='*',
help='One or more apps or models to reindex',
)
def _get_indexers(self, *model_names):
indexers = {}
# No models specified; pull in all registered indexers
if not model_names:
for idx in registry['search'].values():
indexers[idx.model] = idx
# Return only indexers for the specified models
else:
for label in model_names:
try:
app_label, model_name = label.lower().split('.')
except ValueError:
raise CommandError(
f"Invalid model: {label}. Model names must be in the format <app_label>.<model_name>."
)
try:
idx = registry['search'][f'{app_label}.{model_name}']
indexers[idx.model] = idx
except KeyError:
raise CommandError(f"No indexer registered for {label}")
return indexers
def handle(self, *model_labels, **kwargs):
# Determine which models to reindex
indexers = self._get_indexers(*model_labels)
if not indexers:
raise CommandError("No indexers found!")
self.stdout.write(f'Reindexing {len(indexers)} models.')
# Clear all cached values for the specified models
self.stdout.write('Clearing cached values... ', ending='')
self.stdout.flush()
content_types = [
ContentType.objects.get_for_model(model) for model in indexers.keys()
]
deleted_count = search_backend.clear(content_types)
self.stdout.write(f'{deleted_count} entries deleted.')
# Index models
self.stdout.write('Indexing models')
for model, idx in indexers.items():
app_label = model._meta.app_label
model_name = model._meta.model_name
self.stdout.write(f' {app_label}.{model_name}... ', ending='')
self.stdout.flush()
i = search_backend.cache(model.objects.iterator(), remove_existing=False)
if i:
self.stdout.write(f'{i} entries cached.')
else:
self.stdout.write(f'None found.')
msg = f'Completed.'
if total_count := search_backend.size:
msg += f' Total entries: {total_count}'
self.stdout.write(msg, self.style.SUCCESS)

17
netbox/extras/migrations/0079_change_jobresult_order.py
View File

@ -1,17 +0,0 @@
# Generated by Django 4.1.1 on 2022-10-09 18:37
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
('extras', '0078_unique_constraints'),
]
operations = [
migrations.AlterModelOptions(
name='jobresult',
options={'ordering': ['-created']},
),
]

8
netbox/extras/migrations/0080_add_jobresult_scheduled_time.py → netbox/extras/migrations/0079_jobresult_scheduled_time.py
View File

@ -1,12 +1,10 @@
# Generated by Django 4.1.1 on 2022-10-16 09:52
from django.db import migrations, models from django.db import migrations, models
class Migration(migrations.Migration): class Migration(migrations.Migration):
dependencies = [ dependencies = [
('extras', '0079_change_jobresult_order'), ('extras', '0078_unique_constraints'),
] ]
operations = [ operations = [
@ -15,4 +13,8 @@ class Migration(migrations.Migration):
name='scheduled_time', name='scheduled_time',
field=models.DateTimeField(blank=True, null=True), field=models.DateTimeField(blank=True, null=True),
), ),
migrations.AlterModelOptions(
name='jobresult',
options={'ordering': ['-created']},
),
] ]

35
netbox/extras/migrations/0080_search.py Normal file
View File

@ -0,0 +1,35 @@
from django.db import migrations, models
import django.db.models.deletion
import uuid
class Migration(migrations.Migration):
dependencies = [
('contenttypes', '0002_remove_content_type_name'),
('extras', '0079_jobresult_scheduled_time'),
]
operations = [
migrations.AddField(
model_name='customfield',
name='search_weight',
field=models.PositiveSmallIntegerField(default=1000),
),
migrations.CreateModel(
name='CachedValue',
fields=[
('id', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False)),
('timestamp', models.DateTimeField(auto_now_add=True)),
('object_id', models.PositiveBigIntegerField()),
('field', models.CharField(max_length=200)),
('type', models.CharField(max_length=30)),
('value', models.TextField(db_index=True)),
('weight', models.PositiveSmallIntegerField(default=1000)),
('object_type', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='+', to='contenttypes.contenttype')),
],
options={
'ordering': ('weight', 'object_type', 'object_id'),
},
),
]

2
netbox/extras/models/__init__.py
View File

@ -2,9 +2,11 @@ from .change_logging import ObjectChange
from .configcontexts import ConfigContext, ConfigContextModel from .configcontexts import ConfigContext, ConfigContextModel
from .customfields import CustomField from .customfields import CustomField
from .models import * from .models import *
from .search import *
from .tags import Tag, TaggedItem from .tags import Tag, TaggedItem
__all__ = ( __all__ = (
'CachedValue',
'ConfigContext', 'ConfigContext',
'ConfigContextModel', 'ConfigContextModel',
'ConfigRevision', 'ConfigRevision',

25
netbox/extras/models/customfields.py
View File

@ -16,6 +16,7 @@ from extras.choices import *
from extras.utils import FeatureQuery from extras.utils import FeatureQuery
from netbox.models import ChangeLoggedModel from netbox.models import ChangeLoggedModel
from netbox.models.features import CloningMixin, ExportTemplatesMixin, WebhooksMixin from netbox.models.features import CloningMixin, ExportTemplatesMixin, WebhooksMixin
from netbox.search import FieldTypes
from utilities import filters from utilities import filters
from utilities.forms import ( from utilities.forms import (
CSVChoiceField, CSVMultipleChoiceField, DatePicker, DynamicModelChoiceField, DynamicModelMultipleChoiceField, CSVChoiceField, CSVMultipleChoiceField, DatePicker, DynamicModelChoiceField, DynamicModelMultipleChoiceField,
@ -30,6 +31,15 @@ __all__ = (
'CustomFieldManager', 'CustomFieldManager',
) )
SEARCH_TYPES = {
CustomFieldTypeChoices.TYPE_TEXT: FieldTypes.STRING,
CustomFieldTypeChoices.TYPE_LONGTEXT: FieldTypes.STRING,
CustomFieldTypeChoices.TYPE_INTEGER: FieldTypes.INTEGER,
CustomFieldTypeChoices.TYPE_DECIMAL: FieldTypes.FLOAT,
CustomFieldTypeChoices.TYPE_DATE: FieldTypes.STRING,
CustomFieldTypeChoices.TYPE_URL: FieldTypes.STRING,
}
class CustomFieldManager(models.Manager.from_queryset(RestrictedQuerySet)): class CustomFieldManager(models.Manager.from_queryset(RestrictedQuerySet)):
use_in_migrations = True use_in_migrations = True
@ -94,6 +104,11 @@ class CustomField(CloningMixin, ExportTemplatesMixin, WebhooksMixin, ChangeLogge
help_text='If true, this field is required when creating new objects ' help_text='If true, this field is required when creating new objects '
'or editing an existing object.' 'or editing an existing object.'
) )
search_weight = models.PositiveSmallIntegerField(
default=1000,
help_text='Weighting for search. Lower values are considered more important. '
'Fields with a search weight of zero will be ignored.'
)
filter_logic = models.CharField( filter_logic = models.CharField(
max_length=50, max_length=50,
choices=CustomFieldFilterLogicChoices, choices=CustomFieldFilterLogicChoices,
@ -109,6 +124,7 @@ class CustomField(CloningMixin, ExportTemplatesMixin, WebhooksMixin, ChangeLogge
) )
weight = models.PositiveSmallIntegerField( weight = models.PositiveSmallIntegerField(
default=100, default=100,
verbose_name='Display weight',
help_text='Fields with higher weights appear lower in a form.' help_text='Fields with higher weights appear lower in a form.'
) )
validation_minimum = models.IntegerField( validation_minimum = models.IntegerField(
@ -148,8 +164,9 @@ class CustomField(CloningMixin, ExportTemplatesMixin, WebhooksMixin, ChangeLogge
objects = CustomFieldManager() objects = CustomFieldManager()
clone_fields = ( clone_fields = (
'content_types', 'type', 'object_type', 'group_name', 'description', 'required', 'filter_logic', 'default', 'content_types', 'type', 'object_type', 'group_name', 'description', 'required', 'search_weight',
'weight', 'validation_minimum', 'validation_maximum', 'validation_regex', 'choices', 'ui_visibility', 'filter_logic', 'default', 'weight', 'validation_minimum', 'validation_maximum', 'validation_regex', 'choices',
'ui_visibility',
) )
class Meta: class Meta:
@ -167,6 +184,10 @@ class CustomField(CloningMixin, ExportTemplatesMixin, WebhooksMixin, ChangeLogge
# Cache instance's original name so we can check later whether it has changed # Cache instance's original name so we can check later whether it has changed
self._name = self.name self._name = self.name
@property
def search_type(self):
return SEARCH_TYPES.get(self.type)
def populate_initial_data(self, content_types): def populate_initial_data(self, content_types):
""" """
Populate initial custom field data upon either a) the creation of a new CustomField, or Populate initial custom field data upon either a) the creation of a new CustomField, or

50
netbox/extras/models/search.py Normal file
View File

@ -0,0 +1,50 @@
import uuid
from django.contrib.contenttypes.models import ContentType
from django.db import models
from utilities.fields import RestrictedGenericForeignKey
__all__ = (
'CachedValue',
)
class CachedValue(models.Model):
id = models.UUIDField(
primary_key=True,
default=uuid.uuid4,
editable=False
)
timestamp = models.DateTimeField(
auto_now_add=True,
editable=False
)
object_type = models.ForeignKey(
to=ContentType,
on_delete=models.CASCADE,
related_name='+'
)
object_id = models.PositiveBigIntegerField()
object = RestrictedGenericForeignKey(
ct_field='object_type',
fk_field='object_id'
)
field = models.CharField(
max_length=200
)
type = models.CharField(
max_length=30
)
value = models.TextField(
db_index=True
)
weight = models.PositiveSmallIntegerField(
default=1000
)
class Meta:
ordering = ('weight', 'object_type', 'object_id')
def __str__(self):
return f'{self.object_type} {self.object_id}: {self.field}={self.value}'

2
netbox/extras/plugins/__init__.py
View File

@ -75,7 +75,7 @@ class PluginConfig(AppConfig):
try: try:
search_indexes = import_string(f"{self.__module__}.{self.search_indexes}") search_indexes = import_string(f"{self.__module__}.{self.search_indexes}")
for idx in search_indexes: for idx in search_indexes:
register_search()(idx) register_search(idx)
except ImportError: except ImportError:
pass pass

2
netbox/extras/registry.py
View File

@ -29,5 +29,5 @@ registry['model_features'] = {
feature: collections.defaultdict(set) for feature in EXTRAS_FEATURES feature: collections.defaultdict(set) for feature in EXTRAS_FEATURES
} }
registry['denormalized_fields'] = collections.defaultdict(list) registry['denormalized_fields'] = collections.defaultdict(list)
registry['search'] = collections.defaultdict(dict) registry['search'] = dict()
registry['views'] = collections.defaultdict(dict) registry['views'] = collections.defaultdict(dict)

15
netbox/extras/search.py
View File

@ -1,14 +1,11 @@
import extras.filtersets
import extras.tables
from extras.models import JournalEntry
from netbox.search import SearchIndex, register_search from netbox.search import SearchIndex, register_search
from . import models
@register_search() @register_search
class JournalEntryIndex(SearchIndex): class JournalEntryIndex(SearchIndex):
model = JournalEntry model = models.JournalEntry
queryset = JournalEntry.objects.prefetch_related('assigned_object', 'created_by') fields = (
filterset = extras.filtersets.JournalEntryFilterSet ('comments', 5000),
table = extras.tables.JournalEntryTable )
url = 'extras:journalentry_list'
category = 'Journal' category = 'Journal'

4
netbox/extras/tables/tables.py
View File

@ -34,8 +34,8 @@ class CustomFieldTable(NetBoxTable):
class Meta(NetBoxTable.Meta): class Meta(NetBoxTable.Meta):
model = CustomField model = CustomField
fields = ( fields = (
'pk', 'id', 'name', 'content_types', 'label', 'type', 'group_name', 'required', 'weight', 'default', 'pk', 'id', 'name', 'content_types', 'label', 'type', 'group_name', 'required', 'default', 'description',
'description', 'filter_logic', 'ui_visibility', 'choices', 'created', 'last_updated', 'search_weight', 'filter_logic', 'ui_visibility', 'weight', 'choices', 'created', 'last_updated',
) )
default_columns = ('pk', 'name', 'content_types', 'label', 'group_name', 'type', 'required', 'description') default_columns = ('pk', 'name', 'content_types', 'label', 'group_name', 'type', 'required', 'description')

5
netbox/extras/tests/dummy_plugin/search.py
View File

@ -4,8 +4,9 @@ from .models import DummyModel
class DummyModelIndex(SearchIndex): class DummyModelIndex(SearchIndex):
model = DummyModel model = DummyModel
queryset = DummyModel.objects.all() fields = (
url = 'plugins:dummy_plugin:dummy_models' ('name', 100),
)
indexes = ( indexes = (

2
netbox/extras/tests/test_customfields.py
View File

@ -292,6 +292,7 @@ class CustomFieldTest(TestCase):
cf = CustomField.objects.create( cf = CustomField.objects.create(
name='object_field', name='object_field',
type=CustomFieldTypeChoices.TYPE_OBJECT, type=CustomFieldTypeChoices.TYPE_OBJECT,
object_type=ContentType.objects.get_for_model(VLAN),
required=False required=False
) )
cf.content_types.set([self.object_type]) cf.content_types.set([self.object_type])
@ -323,6 +324,7 @@ class CustomFieldTest(TestCase):
cf = CustomField.objects.create( cf = CustomField.objects.create(
name='object_field', name='object_field',
type=CustomFieldTypeChoices.TYPE_MULTIOBJECT, type=CustomFieldTypeChoices.TYPE_MULTIOBJECT,
object_type=ContentType.objects.get_for_model(VLAN),
required=False required=False
) )
cf.content_types.set([self.object_type]) cf.content_types.set([self.object_type])

11
netbox/extras/tests/test_views.py
View File

@ -32,6 +32,7 @@ class CustomFieldTestCase(ViewTestCases.PrimaryObjectViewTestCase):
'label': 'Field X', 'label': 'Field X',
'type': 'text', 'type': 'text',
'content_types': [site_ct.pk], 'content_types': [site_ct.pk],
'search_weight': 2000,
'filter_logic': CustomFieldFilterLogicChoices.FILTER_EXACT, 'filter_logic': CustomFieldFilterLogicChoices.FILTER_EXACT,
'default': None, 'default': None,
'weight': 200, 'weight': 200,
@ -40,11 +41,11 @@ class CustomFieldTestCase(ViewTestCases.PrimaryObjectViewTestCase):
} }
cls.csv_data = ( cls.csv_data = (
'name,label,type,content_types,object_type,weight,filter_logic,choices,validation_minimum,validation_maximum,validation_regex,ui_visibility', 'name,label,type,content_types,object_type,weight,search_weight,filter_logic,choices,validation_minimum,validation_maximum,validation_regex,ui_visibility',
'field4,Field 4,text,dcim.site,,100,exact,,,,[a-z]{3},read-write', 'field4,Field 4,text,dcim.site,,100,1000,exact,,,,[a-z]{3},read-write',
'field5,Field 5,integer,dcim.site,,100,exact,,1,100,,read-write', 'field5,Field 5,integer,dcim.site,,100,2000,exact,,1,100,,read-write',
'field6,Field 6,select,dcim.site,,100,exact,"A,B,C",,,,read-write', 'field6,Field 6,select,dcim.site,,100,3000,exact,"A,B,C",,,,read-write',
'field7,Field 7,object,dcim.site,dcim.region,100,exact,,,,,read-write', 'field7,Field 7,object,dcim.site,dcim.region,100,4000,exact,,,,,read-write',
) )
cls.bulk_edit_data = { cls.bulk_edit_data = {

186
netbox/ipam/search.py
View File

@ -1,69 +1,139 @@
import ipam.filtersets from . import models
import ipam.tables
from ipam.models import ASN, VLAN, VRF, Aggregate, IPAddress, Prefix, Service
from netbox.search import SearchIndex, register_search from netbox.search import SearchIndex, register_search
@register_search() @register_search
class VRFIndex(SearchIndex):
model = VRF
queryset = VRF.objects.prefetch_related('tenant', 'tenant__group')
filterset = ipam.filtersets.VRFFilterSet
table = ipam.tables.VRFTable
url = 'ipam:vrf_list'
@register_search()
class AggregateIndex(SearchIndex): class AggregateIndex(SearchIndex):
model = Aggregate model = models.Aggregate
queryset = Aggregate.objects.prefetch_related('rir') fields = (
filterset = ipam.filtersets.AggregateFilterSet ('prefix', 100),
table = ipam.tables.AggregateTable ('description', 500),
url = 'ipam:aggregate_list' ('date_added', 2000),
@register_search()
class PrefixIndex(SearchIndex):
model = Prefix
queryset = Prefix.objects.prefetch_related(
'site', 'vrf__tenant', 'tenant', 'tenant__group', 'vlan', 'role'
) )
filterset = ipam.filtersets.PrefixFilterSet
table = ipam.tables.PrefixTable
url = 'ipam:prefix_list'
@register_search() @register_search
class IPAddressIndex(SearchIndex):
model = IPAddress
queryset = IPAddress.objects.prefetch_related('vrf__tenant', 'tenant', 'tenant__group')
filterset = ipam.filtersets.IPAddressFilterSet
table = ipam.tables.IPAddressTable
url = 'ipam:ipaddress_list'
@register_search()
class VLANIndex(SearchIndex):
model = VLAN
queryset = VLAN.objects.prefetch_related('site', 'group', 'tenant', 'tenant__group', 'role')
filterset = ipam.filtersets.VLANFilterSet
table = ipam.tables.VLANTable
url = 'ipam:vlan_list'
@register_search()
class ASNIndex(SearchIndex): class ASNIndex(SearchIndex):
model = ASN model = models.ASN
queryset = ASN.objects.prefetch_related('rir', 'tenant', 'tenant__group') fields = (
filterset = ipam.filtersets.ASNFilterSet ('asn', 100),
table = ipam.tables.ASNTable ('description', 500),
url = 'ipam:asn_list' )
@register_search() @register_search
class FHRPGroupIndex(SearchIndex):
model = models.FHRPGroup
fields = (
('name', 100),
('group_id', 2000),
('description', 500),
)
@register_search
class IPAddressIndex(SearchIndex):
model = models.IPAddress
fields = (
('address', 100),
('dns_name', 300),
('description', 500),
)
@register_search
class IPRangeIndex(SearchIndex):
model = models.IPRange
fields = (
('start_address', 100),
('end_address', 300),
('description', 500),
)
@register_search
class L2VPNIndex(SearchIndex):
model = models.L2VPN
fields = (
('name', 100),
('slug', 110),
('description', 500),
)
@register_search
class PrefixIndex(SearchIndex):
model = models.Prefix
fields = (
('prefix', 100),
('description', 500),
)
@register_search
class RIRIndex(SearchIndex):
model = models.RIR
fields = (
('name', 100),
('slug', 110),
('description', 500),
)
@register_search
class RoleIndex(SearchIndex):
model = models.Role
fields = (
('name', 100),
('slug', 110),
('description', 500),
)
@register_search
class RouteTargetIndex(SearchIndex):
model = models.RouteTarget
fields = (
('name', 100),
('description', 500),
)
@register_search
class ServiceIndex(SearchIndex): class ServiceIndex(SearchIndex):
model = Service model = models.Service
queryset = Service.objects.prefetch_related('device', 'virtual_machine') fields = (
filterset = ipam.filtersets.ServiceFilterSet ('name', 100),
table = ipam.tables.ServiceTable ('description', 500),
url = 'ipam:service_list' )
@register_search
class VLANIndex(SearchIndex):
model = models.VLAN
fields = (
('name', 100),
('vid', 100),
('description', 500),
)
@register_search
class VLANGroupIndex(SearchIndex):
model = models.VLANGroup
fields = (
('name', 100),
('slug', 110),
('description', 500),
('max_vid', 2000),
)
@register_search
class VRFIndex(SearchIndex):
model = models.VRF
fields = (
('name', 100),
('rd', 200),
('description', 500),
)

3
netbox/netbox/constants.py
View File

@ -1,5 +1,2 @@
# Prefix for nested serializers # Prefix for nested serializers
NESTED_SERIALIZER_PREFIX = 'Nested' NESTED_SERIALIZER_PREFIX = 'Nested'
# Max results per object type
SEARCH_MAX_RESULTS = 15

59
netbox/netbox/forms/__init__.py
View File

@ -1,38 +1,45 @@
from django import forms from django import forms
from django.utils.translation import gettext as _
from netbox.search.backends import default_search_engine from netbox.search import LookupTypes
from utilities.forms import BootstrapMixin from netbox.search.backends import search_backend
from utilities.forms import BootstrapMixin, StaticSelect, StaticSelectMultiple
from .base import * from .base import *
LOOKUP_CHOICES = (
def build_options(choices): ('', _('Partial match')),
options = [{"label": choices[0][1], "items": []}] (LookupTypes.EXACT, _('Exact match')),
(LookupTypes.STARTSWITH, _('Starts with')),
for label, choices in choices[1:]: (LookupTypes.ENDSWITH, _('Ends with')),
items = [] )
for value, choice_label in choices:
items.append({"label": choice_label, "value": value})
options.append({"label": label, "items": items})
return options
class SearchForm(BootstrapMixin, forms.Form): class SearchForm(BootstrapMixin, forms.Form):
q = forms.CharField(label='Search') q = forms.CharField(
options = None label='Search',
widget=forms.TextInput(
attrs={
'hx-get': '',
'hx-target': '#object_list',
'hx-trigger': 'keyup[target.value.length >= 3] changed delay:500ms',
}
)
)
obj_types = forms.MultipleChoiceField(
choices=[],
required=False,
label='Object type(s)',
widget=StaticSelectMultiple()
)
lookup = forms.ChoiceField(
choices=LOOKUP_CHOICES,
initial=LookupTypes.PARTIAL,
required=False,
widget=StaticSelect()
)
def __init__(self, *args, **kwargs): def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs) super().__init__(*args, **kwargs)
self.fields["obj_type"] = forms.ChoiceField(
choices=default_search_engine.get_search_choices(),
required=False,
label='Type'
)
def get_options(self): self.fields['obj_types'].choices = search_backend.get_object_types()
if not self.options:
self.options = build_options(default_search_engine.get_search_choices())
return self.options

106
netbox/netbox/search/__init__.py
View File

@ -1,5 +1,24 @@
from collections import namedtuple
from django.db import models
from extras.registry import registry from extras.registry import registry
ObjectFieldValue = namedtuple('ObjectFieldValue', ('name', 'type', 'weight', 'value'))
class FieldTypes:
FLOAT = 'float'
INTEGER = 'int'
STRING = 'str'
class LookupTypes:
PARTIAL = 'icontains'
EXACT = 'iexact'
STARTSWITH = 'istartswith'
ENDSWITH = 'iendswith'
class SearchIndex: class SearchIndex:
""" """
@ -7,27 +26,90 @@ class SearchIndex:
Attrs: Attrs:
model: The model class for which this index is used. model: The model class for which this index is used.
category: The label of the group under which this indexer is categorized (for form field display). If none,
the name of the model's app will be used.
fields: An iterable of two-tuples defining the model fields to be indexed and the weight associated with each.
""" """
model = None model = None
category = None
fields = ()
@staticmethod
def get_field_type(instance, field_name):
"""
Return the data type of the specified model field.
"""
field_cls = instance._meta.get_field(field_name).__class__
if issubclass(field_cls, (models.FloatField, models.DecimalField)):
return FieldTypes.FLOAT
if issubclass(field_cls, models.IntegerField):
return FieldTypes.INTEGER
return FieldTypes.STRING
@staticmethod
def get_field_value(instance, field_name):
"""
Return the value of the specified model field as a string.
"""
return str(getattr(instance, field_name))
@classmethod @classmethod
def get_category(cls): def get_category(cls):
return cls.category or cls.model._meta.app_config.verbose_name
@classmethod
def to_cache(cls, instance, custom_fields=None):
""" """
Return the title of the search category under which this model is registered. Return a list of ObjectFieldValue representing the instance fields to be cached.
Args:
instance: The instance being cached.
custom_fields: An iterable of CustomFields to include when caching the instance. If None, all custom fields
defined for the model will be included. (This can also be provided during bulk caching to avoid looking
up the available custom fields for each instance.)
""" """
if hasattr(cls, 'category'): values = []
return cls.category
return cls.model._meta.app_config.verbose_name # Capture built-in fields
for name, weight in cls.fields:
type_ = cls.get_field_type(instance, name)
value = cls.get_field_value(instance, name)
if type_ and value:
values.append(
ObjectFieldValue(name, type_, weight, value)
)
# Capture custom fields
if getattr(instance, 'custom_field_data', None):
if custom_fields is None:
custom_fields = instance.get_custom_fields().keys()
for cf in custom_fields:
type_ = cf.search_type
value = instance.custom_field_data.get(cf.name)
weight = cf.search_weight
if type_ and value and weight:
values.append(
ObjectFieldValue(f'cf_{cf.name}', type_, weight, value)
)
return values
def register_search(): def get_indexer(model):
def _wrapper(cls): """
Get the SearchIndex class for the given model.
"""
label = f'{model._meta.app_label}.{model._meta.model_name}'
return registry['search'][label]
def register_search(cls):
"""
Decorator for registering a SearchIndex class.
"""
model = cls.model model = cls.model
app_label = model._meta.app_label label = f'{model._meta.app_label}.{model._meta.model_name}'
model_name = model._meta.model_name registry['search'][label] = cls
registry['search'][app_label][model_name] = cls
return cls return cls
return _wrapper

267
netbox/netbox/search/backends.py
View File

@ -1,125 +1,236 @@
from collections import defaultdict from collections import defaultdict
from importlib import import_module
from django.conf import settings from django.conf import settings
from django.contrib.contenttypes.models import ContentType
from django.core.exceptions import ImproperlyConfigured from django.core.exceptions import ImproperlyConfigured
from django.urls import reverse from django.db.models import F, Window
from django.db.models.functions import window
from django.db.models.signals import post_delete, post_save
from django.utils.module_loading import import_string
from extras.models import CachedValue, CustomField
from extras.registry import registry from extras.registry import registry
from netbox.constants import SEARCH_MAX_RESULTS from utilities.querysets import RestrictedPrefetch
from utilities.templatetags.builtins.filters import bettertitle
from . import FieldTypes, LookupTypes, get_indexer
# The cache for the initialized backend. DEFAULT_LOOKUP_TYPE = LookupTypes.PARTIAL
_backends_cache = {} MAX_RESULTS = 1000
class SearchEngineError(Exception):
"""Something went wrong with a search engine."""
pass
class SearchBackend: class SearchBackend:
"""A search engine capable of performing multi-table searches.""" """
_search_choice_options = tuple() Base class for search backends. Subclasses must extend the `cache()`, `remove()`, and `clear()` methods below.
"""
_object_types = None
def get_registry(self): def get_object_types(self):
r = {} """
for app_label, models in registry['search'].items(): Return a list of all registered object types, organized by category, suitable for populating a form's
r.update(**models) ChoiceField.
"""
return r if not self._object_types:
def get_search_choices(self):
"""Return the set of choices for individual object types, organized by category."""
if not self._search_choice_options:
# Organize choices by category # Organize choices by category
categories = defaultdict(dict) categories = defaultdict(dict)
for app_label, models in registry['search'].items(): for label, idx in registry['search'].items():
for name, cls in models.items(): title = bettertitle(idx.model._meta.verbose_name)
title = cls.model._meta.verbose_name.title() categories[idx.get_category()][label] = title
categories[cls.get_category()][name] = title
# Compile a nested tuple of choices for form rendering # Compile a nested tuple of choices for form rendering
results = ( results = (
('', 'All Objects'), ('', 'All Objects'),
*[(category, choices.items()) for category, choices in categories.items()] *[(category, list(choices.items())) for category, choices in categories.items()]
) )
self._search_choice_options = results self._object_types = results
return self._search_choice_options return self._object_types
def search(self, request, value, **kwargs): def search(self, value, user=None, object_types=None, lookup=DEFAULT_LOOKUP_TYPE):
"""Execute a search query for the given value.""" """
Search cached object representations for the given value.
"""
raise NotImplementedError raise NotImplementedError
def cache(self, instance): def caching_handler(self, sender, instance, **kwargs):
"""Create or update the cached copy of an instance.""" """
Receiver for the post_save signal, responsible for caching object creation/changes.
"""
self.cache(instance)
def removal_handler(self, sender, instance, **kwargs):
"""
Receiver for the post_delete signal, responsible for caching object deletion.
"""
self.remove(instance)
def cache(self, instances, indexer=None, remove_existing=True):
"""
Create or update the cached representation of an instance.
"""
raise NotImplementedError raise NotImplementedError
def remove(self, instance):
class FilterSetSearchBackend(SearchBackend):
""" """
Legacy search backend. Performs a discrete database query for each registered object type, using the FilterSet Delete any cached representation of an instance.
class specified by the index for each.
""" """
def search(self, request, value, **kwargs): raise NotImplementedError
results = []
search_registry = self.get_registry() def clear(self, object_types=None):
for obj_type in search_registry.keys(): """
Delete *all* cached data.
"""
raise NotImplementedError
queryset = search_registry[obj_type].queryset @property
url = search_registry[obj_type].url def size(self):
"""
Return a total number of cached entries. The meaning of this value will be
backend-dependent.
"""
return None
# Restrict the queryset for the current user
if hasattr(queryset, 'restrict'):
queryset = queryset.restrict(request.user, 'view')
filterset = getattr(search_registry[obj_type], 'filterset', None) class CachedValueSearchBackend(SearchBackend):
if not filterset:
# This backend requires a FilterSet class for the model
continue
table = getattr(search_registry[obj_type], 'table', None) def search(self, value, user=None, object_types=None, lookup=DEFAULT_LOOKUP_TYPE):
if not table:
# This backend requires a Table class for the model
continue
# Construct the results table for this object type # Define the search parameters
filtered_queryset = filterset({'q': value}, queryset=queryset).qs params = {
table = table(filtered_queryset, orderable=False) f'value__{lookup}': value
table.paginate(per_page=SEARCH_MAX_RESULTS) }
if lookup != LookupTypes.EXACT:
# Partial matches are valid only on string values
params['type'] = FieldTypes.STRING
if object_types:
params['object_type__in'] = object_types
if table.page: # Construct the base queryset to retrieve matching results
results.append({ queryset = CachedValue.objects.filter(**params).annotate(
'name': queryset.model._meta.verbose_name_plural, # Annotate the rank of each result for its object according to its weight
'table': table, row_number=Window(
'url': f"{reverse(url)}?q={value}" expression=window.RowNumber(),
}) partition_by=[F('object_type'), F('object_id')],
order_by=[F('weight').asc()],
)
)[:MAX_RESULTS]
return results # Construct a Prefetch to pre-fetch only those related objects for which the
# user has permission to view.
if user:
prefetch = (RestrictedPrefetch('object', user, 'view'), 'object_type')
else:
prefetch = ('object', 'object_type')
def cache(self, instance): # Wrap the base query to return only the lowest-weight result for each object
# This backend does not utilize a cache # Hat-tip to https://blog.oyam.dev/django-filter-by-window-function/ for the solution
pass sql, params = queryset.query.sql_with_params()
results = CachedValue.objects.prefetch_related(*prefetch).raw(
f"SELECT * FROM ({sql}) t WHERE row_number = 1",
params
)
# Omit any results pertaining to an object the user does not have permission to view
return [
r for r in results if r.object is not None
]
def cache(self, instances, indexer=None, remove_existing=True):
content_type = None
custom_fields = None
# Convert a single instance to an iterable
if not hasattr(instances, '__iter__'):
instances = [instances]
buffer = []
counter = 0
for instance in instances:
# First item
if not counter:
# Determine the indexer
if indexer is None:
try:
indexer = get_indexer(instance)
except KeyError:
break
# Prefetch any associated custom fields
content_type = ContentType.objects.get_for_model(indexer.model)
custom_fields = CustomField.objects.filter(content_types=content_type).exclude(search_weight=0)
# Wipe out any previously cached values for the object
if remove_existing:
self.remove(instance)
# Generate cache data
for field in indexer.to_cache(instance, custom_fields=custom_fields):
buffer.append(
CachedValue(
object_type=content_type,
object_id=instance.pk,
field=field.name,
type=field.type,
weight=field.weight,
value=field.value
)
)
# Check whether the buffer needs to be flushed
if len(buffer) >= 2000:
counter += len(CachedValue.objects.bulk_create(buffer))
buffer = []
# Final buffer flush
if buffer:
counter += len(CachedValue.objects.bulk_create(buffer))
return counter
def remove(self, instance):
# Avoid attempting to query for non-cacheable objects
try:
get_indexer(instance)
except KeyError:
return
ct = ContentType.objects.get_for_model(instance)
qs = CachedValue.objects.filter(object_type=ct, object_id=instance.pk)
# Call _raw_delete() on the queryset to avoid first loading instances into memory
return qs._raw_delete(using=qs.db)
def clear(self, object_types=None):
qs = CachedValue.objects.all()
if object_types:
qs = qs.filter(object_type__in=object_types)
# Call _raw_delete() on the queryset to avoid first loading instances into memory
return qs._raw_delete(using=qs.db)
@property
def size(self):
return CachedValue.objects.count()
def get_backend(): def get_backend():
"""Initializes and returns the configured search backend.""" """
backend_name = settings.SEARCH_BACKEND Initializes and returns the configured search backend.
"""
# Load the backend class
backend_module_name, backend_cls_name = backend_name.rsplit('.', 1)
backend_module = import_module(backend_module_name)
try: try:
backend_cls = getattr(backend_module, backend_cls_name) backend_cls = import_string(settings.SEARCH_BACKEND)
except AttributeError: except AttributeError:
raise ImproperlyConfigured(f"Could not find a class named {backend_module_name} in {backend_cls_name}") raise ImproperlyConfigured(f"Failed to import configured SEARCH_BACKEND: {settings.SEARCH_BACKEND}")
# Initialize and return the backend instance # Initialize and return the backend instance
return backend_cls() return backend_cls()
default_search_engine = get_backend() search_backend = get_backend()
search = default_search_engine.search
# Connect handlers to the appropriate model signals
post_save.connect(search_backend.caching_handler)
post_delete.connect(search_backend.removal_handler)

2
netbox/netbox/settings.py
View File

@ -116,7 +116,7 @@ REMOTE_AUTH_GROUP_SEPARATOR = getattr(configuration, 'REMOTE_AUTH_GROUP_SEPARATO
REPORTS_ROOT = getattr(configuration, 'REPORTS_ROOT', os.path.join(BASE_DIR, 'reports')).rstrip('/') REPORTS_ROOT = getattr(configuration, 'REPORTS_ROOT', os.path.join(BASE_DIR, 'reports')).rstrip('/')
RQ_DEFAULT_TIMEOUT = getattr(configuration, 'RQ_DEFAULT_TIMEOUT', 300) RQ_DEFAULT_TIMEOUT = getattr(configuration, 'RQ_DEFAULT_TIMEOUT', 300)
SCRIPTS_ROOT = getattr(configuration, 'SCRIPTS_ROOT', os.path.join(BASE_DIR, 'scripts')).rstrip('/') SCRIPTS_ROOT = getattr(configuration, 'SCRIPTS_ROOT', os.path.join(BASE_DIR, 'scripts')).rstrip('/')
SEARCH_BACKEND = getattr(configuration, 'SEARCH_BACKEND', 'netbox.search.backends.FilterSetSearchBackend') SEARCH_BACKEND = getattr(configuration, 'SEARCH_BACKEND', 'netbox.search.backends.CachedValueSearchBackend')
SENTRY_DSN = getattr(configuration, 'SENTRY_DSN', DEFAULT_SENTRY_DSN) SENTRY_DSN = getattr(configuration, 'SENTRY_DSN', DEFAULT_SENTRY_DSN)
SENTRY_ENABLED = getattr(configuration, 'SENTRY_ENABLED', False) SENTRY_ENABLED = getattr(configuration, 'SENTRY_ENABLED', False)
SENTRY_SAMPLE_RATE = getattr(configuration, 'SENTRY_SAMPLE_RATE', 1.0) SENTRY_SAMPLE_RATE = getattr(configuration, 'SENTRY_SAMPLE_RATE', 1.0)

41
netbox/netbox/tables/tables.py
View File

@ -4,16 +4,21 @@ from django.contrib.contenttypes.fields import GenericForeignKey
from django.contrib.contenttypes.models import ContentType from django.contrib.contenttypes.models import ContentType
from django.core.exceptions import FieldDoesNotExist from django.core.exceptions import FieldDoesNotExist
from django.db.models.fields.related import RelatedField from django.db.models.fields.related import RelatedField
from django.utils.safestring import mark_safe
from django.utils.translation import gettext as _
from django_tables2.data import TableQuerysetData from django_tables2.data import TableQuerysetData
from extras.models import CustomField, CustomLink from extras.models import CustomField, CustomLink
from extras.choices import CustomFieldVisibilityChoices from extras.choices import CustomFieldVisibilityChoices
from netbox.tables import columns from netbox.tables import columns
from utilities.paginator import EnhancedPaginator, get_paginate_count from utilities.paginator import EnhancedPaginator, get_paginate_count
from utilities.templatetags.builtins.filters import bettertitle
from utilities.utils import highlight_string
__all__ = ( __all__ = (
'BaseTable', 'BaseTable',
'NetBoxTable', 'NetBoxTable',
'SearchTable',
) )
@ -192,3 +197,39 @@ class NetBoxTable(BaseTable):
]) ])
super().__init__(*args, extra_columns=extra_columns, **kwargs) super().__init__(*args, extra_columns=extra_columns, **kwargs)
class SearchTable(tables.Table):
object_type = columns.ContentTypeColumn(
verbose_name=_('Type')
)
object = tables.Column(
linkify=True
)
field = tables.Column()
value = tables.Column()
trim_length = 30
class Meta:
attrs = {
'class': 'table table-hover object-list',
}
empty_text = _('No results found')
def __init__(self, data, highlight=None, **kwargs):
self.highlight = highlight
super().__init__(data, **kwargs)
def render_field(self, value, record):
if hasattr(record.object, value):
return bettertitle(record.object._meta.get_field(value).verbose_name)
return value
def render_value(self, value):
if not self.highlight:
return value
value = highlight_string(value, self.highlight, trim_pre=self.trim_length, trim_post=self.trim_length)
return mark_safe(value)

153
netbox/netbox/tests/test_search.py Normal file
View File

@ -0,0 +1,153 @@
from django.contrib.contenttypes.models import ContentType
from django.test import TestCase
from dcim.models import Site
from dcim.search import SiteIndex
from extras.models import CachedValue
from netbox.search.backends import search_backend
class SearchBackendTestCase(TestCase):
@classmethod
def setUpTestData(cls):
# Create sites with a value for each cacheable field defined on SiteIndex
sites = (
Site(
name='Site 1',
slug='site-1',
facility='Alpha',
description='First test site',
physical_address='123 Fake St Lincoln NE 68588',
shipping_address='123 Fake St Lincoln NE 68588',
comments='Lorem ipsum etcetera'
),
Site(
name='Site 2',
slug='site-2',
facility='Bravo',
description='Second test site',
physical_address='725 Cyrus Valleys Suite 761 Douglasfort NE 57761',
shipping_address='725 Cyrus Valleys Suite 761 Douglasfort NE 57761',
comments='Lorem ipsum etcetera'
),
Site(
name='Site 3',
slug='site-3',
facility='Charlie',
description='Third test site',
physical_address='2321 Dovie Dale East Cristobal AK 71959',
shipping_address='2321 Dovie Dale East Cristobal AK 71959',
comments='Lorem ipsum etcetera'
),
)
Site.objects.bulk_create(sites)
def test_cache_single_object(self):
"""
Test that a single object is cached appropriately
"""
site = Site.objects.first()
search_backend.cache(site)
content_type = ContentType.objects.get_for_model(Site)
self.assertEqual(
CachedValue.objects.filter(object_type=content_type, object_id=site.pk).count(),
len(SiteIndex.fields)
)
for field_name, weight in SiteIndex.fields:
self.assertTrue(
CachedValue.objects.filter(
object_type=content_type,
object_id=site.pk,
field=field_name,
value=getattr(site, field_name),
weight=weight
),
)
def test_cache_multiple_objects(self):
"""
Test that multiples objects are cached appropriately
"""
sites = Site.objects.all()
search_backend.cache(sites)
content_type = ContentType.objects.get_for_model(Site)
self.assertEqual(
CachedValue.objects.filter(object_type=content_type).count(),
len(SiteIndex.fields) * sites.count()
)
for site in sites:
for field_name, weight in SiteIndex.fields:
self.assertTrue(
CachedValue.objects.filter(
object_type=content_type,
object_id=site.pk,
field=field_name,
value=getattr(site, field_name),
weight=weight
),
)
def test_cache_on_save(self):
"""
Test that an object is automatically cached on calling save().
"""
site = Site(
name='Site 4',
slug='site-4',
facility='Delta',
description='Fourth test site',
physical_address='7915 Lilla Plains West Ladariusport TX 19429',
shipping_address='7915 Lilla Plains West Ladariusport TX 19429',
comments='Lorem ipsum etcetera'
)
site.save()
content_type = ContentType.objects.get_for_model(Site)
self.assertEqual(
CachedValue.objects.filter(object_type=content_type, object_id=site.pk).count(),
len(SiteIndex.fields)
)
def test_remove_on_delete(self):
"""
Test that any cached value for an object are automatically removed on delete().
"""
site = Site.objects.first()
site.delete()
content_type = ContentType.objects.get_for_model(Site)
self.assertFalse(
CachedValue.objects.filter(object_type=content_type, object_id=site.pk).exists()
)
def test_clear_all(self):
"""
Test that calling clear() on the backend removes all cached entries.
"""
sites = Site.objects.all()
search_backend.cache(sites)
self.assertTrue(
CachedValue.objects.exists()
)
search_backend.clear()
self.assertFalse(
CachedValue.objects.exists()
)
def test_search(self):
"""
Test various searches.
"""
sites = Site.objects.all()
search_backend.cache(sites)
results = search_backend.search('site')
self.assertEqual(len(results), 3)
results = search_backend.search('first')
self.assertEqual(len(results), 1)
results = search_backend.search('xxxxx')
self.assertEqual(len(results), 0)

54
netbox/netbox/views/__init__.py
View File

@ -2,15 +2,16 @@ import platform
import sys import sys
from django.conf import settings from django.conf import settings
from django.contrib.contenttypes.models import ContentType
from django.core.cache import cache from django.core.cache import cache
from django.http import HttpResponseServerError from django.http import HttpResponseServerError
from django.shortcuts import redirect, render from django.shortcuts import redirect, render
from django.template import loader from django.template import loader
from django.template.exceptions import TemplateDoesNotExist from django.template.exceptions import TemplateDoesNotExist
from django.urls import reverse
from django.views.decorators.csrf import requires_csrf_token from django.views.decorators.csrf import requires_csrf_token
from django.views.defaults import ERROR_500_TEMPLATE_NAME, page_not_found from django.views.defaults import ERROR_500_TEMPLATE_NAME, page_not_found
from django.views.generic import View from django.views.generic import View
from django_tables2 import RequestConfig
from packaging import version from packaging import version
from sentry_sdk import capture_message from sentry_sdk import capture_message
@ -21,10 +22,13 @@ from dcim.models import (
from extras.models import ObjectChange from extras.models import ObjectChange
from extras.tables import ObjectChangeTable from extras.tables import ObjectChangeTable
from ipam.models import Aggregate, IPAddress, IPRange, Prefix, VLAN, VRF from ipam.models import Aggregate, IPAddress, IPRange, Prefix, VLAN, VRF
from netbox.constants import SEARCH_MAX_RESULTS
from netbox.forms import SearchForm from netbox.forms import SearchForm
from netbox.search.backends import default_search_engine from netbox.search import LookupTypes
from netbox.search.backends import search_backend
from netbox.tables import SearchTable
from tenancy.models import Tenant from tenancy.models import Tenant
from utilities.htmx import is_htmx
from utilities.paginator import EnhancedPaginator, get_paginate_count
from virtualization.models import Cluster, VirtualMachine from virtualization.models import Cluster, VirtualMachine
from wireless.models import WirelessLAN, WirelessLink from wireless.models import WirelessLAN, WirelessLink
@ -149,22 +153,48 @@ class HomeView(View):
class SearchView(View): class SearchView(View):
def get(self, request): def get(self, request):
form = SearchForm(request.GET)
results = [] results = []
highlight = None
# Initialize search form
form = SearchForm(request.GET) if 'q' in request.GET else SearchForm()
if form.is_valid(): if form.is_valid():
search_registry = default_search_engine.get_registry()
# If an object type has been specified, redirect to the dedicated view for it
if form.cleaned_data['obj_type']:
object_type = form.cleaned_data['obj_type']
url = reverse(search_registry[object_type].url)
return redirect(f"{url}?q={form.cleaned_data['q']}")
results = default_search_engine.search(request, form.cleaned_data['q']) # Restrict results by object type
object_types = []
for obj_type in form.cleaned_data['obj_types']:
app_label, model_name = obj_type.split('.')
object_types.append(ContentType.objects.get_by_natural_key(app_label, model_name))
lookup = form.cleaned_data['lookup'] or LookupTypes.PARTIAL
results = search_backend.search(
form.cleaned_data['q'],
user=request.user,
object_types=object_types,
lookup=lookup
)
if form.cleaned_data['lookup'] != LookupTypes.EXACT:
highlight = form.cleaned_data['q']
table = SearchTable(results, highlight=highlight)
# Paginate the table results
RequestConfig(request, {
'paginator_class': EnhancedPaginator,
'per_page': get_paginate_count(request)
}).configure(table)
# If this is an HTMX request, return only the rendered table HTML
if is_htmx(request):
return render(request, 'htmx/table.html', {
'table': table,
})
return render(request, 'search.html', { return render(request, 'search.html', {
'form': form, 'form': form,
'results': results, 'table': table,
}) })

BIN
netbox/project-static/dist/netbox.js vendored
View File

Binary file not shown.

BIN
netbox/project-static/dist/netbox.js.map vendored
View File

Binary file not shown.

4
netbox/project-static/src/netbox.ts
View File

@ -1,6 +1,6 @@
import { initForms } from './forms'; import { initForms } from './forms';
import { initBootstrap } from './bs'; import { initBootstrap } from './bs';
import { initSearch } from './search'; import { initQuickSearch } from './search';
import { initSelect } from './select'; import { initSelect } from './select';
import { initButtons } from './buttons'; import { initButtons } from './buttons';
import { initColorMode } from './colorMode'; import { initColorMode } from './colorMode';
@ -20,7 +20,7 @@ function initDocument(): void {
initColorMode, initColorMode,
initMessages, initMessages,
initForms, initForms,
initSearch, initQuickSearch,
initSelect, initSelect,
initDateSelector, initDateSelector,
initButtons, initButtons,

51
netbox/project-static/src/search.ts
View File

@ -1,31 +1,4 @@
import { getElements, findFirstAdjacent, isTruthy } from './util'; import { isTruthy } from './util';
/**
* Change the display value and hidden input values of the search filter based on dropdown
* selection.
*
* @param event "click" event for each dropdown item.
* @param button Each dropdown item element.
*/
function handleSearchDropdownClick(event: Event, button: HTMLButtonElement): void {
const dropdown = event.currentTarget as HTMLButtonElement;
const selectedValue = findFirstAdjacent<HTMLSpanElement>(dropdown, 'span.search-obj-selected');
const selectedType = findFirstAdjacent<HTMLInputElement>(dropdown, 'input.search-obj-type');
const searchValue = dropdown.getAttribute('data-search-value');
let selected = '' as string;
if (selectedValue !== null && selectedType !== null) {
if (isTruthy(searchValue) && selected !== searchValue) {
selected = searchValue;
selectedValue.innerHTML = button.textContent ?? 'Error';
selectedType.value = searchValue;
} else {
selected = '';
selectedValue.innerHTML = 'All Objects';
selectedType.value = '';
}
}
}
/** /**
* Show/hide quicksearch clear button. * Show/hide quicksearch clear button.
@ -44,23 +17,10 @@ function quickSearchEventHandler(event: Event): void {
} }
} }
/**
* Initialize Search Bar Elements.
*/
function initSearchBar(): void {
for (const dropdown of getElements<HTMLUListElement>('.search-obj-selector')) {
for (const button of dropdown.querySelectorAll<HTMLButtonElement>(
'li > button.dropdown-item',
)) {
button.addEventListener('click', event => handleSearchDropdownClick(event, button));
}
}
}
/** /**
* Initialize Quicksearch Event listener/handlers. * Initialize Quicksearch Event listener/handlers.
*/ */
function initQuickSearch(): void { export function initQuickSearch(): void {
const quicksearch = document.getElementById("quicksearch") as HTMLInputElement; const quicksearch = document.getElementById("quicksearch") as HTMLInputElement;
const clearbtn = document.getElementById("quicksearch_clear") as HTMLButtonElement; const clearbtn = document.getElementById("quicksearch_clear") as HTMLButtonElement;
if (isTruthy(quicksearch)) { if (isTruthy(quicksearch)) {
@ -82,10 +42,3 @@ function initQuickSearch(): void {
} }
} }
} }
export function initSearch(): void {
for (const func of [initSearchBar]) {
func();
}
initQuickSearch();
}

5
netbox/templates/base/layout.html
View File

@ -1,7 +1,6 @@
{# Base layout for the core NetBox UI w/navbar and page content #} {# Base layout for the core NetBox UI w/navbar and page content #}
{% extends 'base/base.html' %} {% extends 'base/base.html' %}
{% load helpers %} {% load helpers %}
{% load search %}
{% load static %} {% load static %}
{% comment %} {% comment %}
@ -41,7 +40,7 @@ Blocks:
</button> </button>
</div> </div>
<div class="d-flex my-1 flex-grow-1 justify-content-center w-100"> <div class="d-flex my-1 flex-grow-1 justify-content-center w-100">
{% search_options request %} {% include 'inc/searchbar.html' %}
</div> </div>
</div> </div>
@ -53,7 +52,7 @@ Blocks:
{# Search bar #} {# Search bar #}
<div class="col-6 d-flex flex-grow-1 justify-content-center"> <div class="col-6 d-flex flex-grow-1 justify-content-center">
{% search_options request %} {% include 'inc/searchbar.html' %}
</div> </div>
{# Proflie/login button #} {# Proflie/login button #}

14
netbox/templates/extras/customfield.html
View File

@ -39,13 +39,23 @@
<td>{% checkmark object.required %}</td> <td>{% checkmark object.required %}</td>
</tr> </tr>
<tr> <tr>
<th scope="row">Weight</th> <th scope="row">Search Weight</th>
<td>{{ object.weight }}</td> <td>
{% if object.search_weight %}
{{ object.search_weight }}
{% else %}
<span class="text-muted">Disabled</span>
{% endif %}
</td>
</tr> </tr>
<tr> <tr>
<th scope="row">Filter Logic</th> <th scope="row">Filter Logic</th>
<td>{{ object.get_filter_logic_display }}</td> <td>{{ object.get_filter_logic_display }}</td>
</tr> </tr>
<tr>
<th scope="row">Display Weight</th>
<td>{{ object.weight }}</td>
</tr>
<tr> <tr>
<th scope="row">UI Visibility</th> <th scope="row">UI Visibility</th>
<td>{{ object.get_ui_visibility_display }}</td> <td>{{ object.get_ui_visibility_display }}</td>

6
netbox/templates/inc/searchbar.html Normal file
View File

@ -0,0 +1,6 @@
<form class="input-group" action="{% url 'search' %}" method="get">
<input name="q" type="text" aria-label="Search" placeholder="Search" class="form-control" />
<button class="btn btn-primary" type="submit">
<i class="mdi mdi-magnify"></i>
</button>
</form>

72
netbox/templates/search.html
View File

@ -15,74 +15,24 @@
</ul> </ul>
{% endblock tabs %} {% endblock tabs %}
{% block content-wrapper %} {% block content %}
<div class="tab-content"> <div class="row px-3">
{% if request.GET.q %} <div class="col col-6 offset-3 py-3">
{% if results %}
<div class="row">
<div class="col col-md-9">
{% for obj_type in results %}
<div class="card">
<h5 class="card-header" id="{{ obj_type.name|lower }}">{{ obj_type.name|bettertitle }}</h5>
<div class="card-body table-responsive">
{% render_table obj_type.table 'inc/table.html' %}
</div>
<div class="card-footer text-end">
<a href="{{ obj_type.url }}" class="btn btn-sm btn-primary my-1">
<i class="mdi mdi-arrow-right-bold" aria-hidden="true"></i>
{% if obj_type.table.page.has_next %}
See All {{ obj_type.table.page.paginator.count }} Results
{% else %}
Refine Search
{% endif %}
</a>
</div>
</div>
{% endfor %}
</div>
<div class="col col-md-3">
<div class="card">
<h5 class="card-header">
Search Results
</h5>
<div class="card-body">
<div class="list-group list-group-flush">
{% for obj_type in results %}
<a href="#{{ obj_type.name|lower }}" class="list-group-item">
<div class="float-end">
{% badge obj_type.table.page.paginator.count %}
</div>
{{ obj_type.name|bettertitle }}
</a>
{% endfor %}
</div>
</div>
</div>
</div>
</div>
{% else %}
<h3 class="text-muted text-center">No results found</h3>
{% endif %}
{% else %}
<div class="row">
<div class="col col-12 col-lg-6 offset-lg-3">
<form action="{% url 'search' %}" method="get" class="form form-horizontal"> <form action="{% url 'search' %}" method="get" class="form form-horizontal">
<div class="card">
<h5 class="card-header">
Search
</h5>
<div class="card-body">
{% render_form form %} {% render_form form %}
</div> <div class="text-end">
<div class="card-footer text-end">
<button type="submit" class="btn btn-primary"> <button type="submit" class="btn btn-primary">
<span class="mdi mdi-magnify" aria-hidden="true"></span> Search <span class="mdi mdi-magnify" aria-hidden="true"></span> Search
</button> </button>
</div> </div>
</div>
</form> </form>
</div> </div>
</div> </div>
{% endif %} <div class="row px-3">
<div class="card">
<div class="card-body" id="object_list">
{% include 'htmx/table.html' %}
</div> </div>
{% endblock content-wrapper %} </div>
</div>
{% endblock content %}

72
netbox/tenancy/search.py
View File

@ -1,25 +1,57 @@
import tenancy.filtersets
import tenancy.tables
from netbox.search import SearchIndex, register_search from netbox.search import SearchIndex, register_search
from tenancy.models import Contact, ContactAssignment, Tenant from . import models
from utilities.utils import count_related
@register_search() @register_search
class TenantIndex(SearchIndex):
model = Tenant
queryset = Tenant.objects.prefetch_related('group')
filterset = tenancy.filtersets.TenantFilterSet
table = tenancy.tables.TenantTable
url = 'tenancy:tenant_list'
@register_search()
class ContactIndex(SearchIndex): class ContactIndex(SearchIndex):
model = Contact model = models.Contact
queryset = Contact.objects.prefetch_related('group', 'assignments').annotate( fields = (
assignment_count=count_related(ContactAssignment, 'contact') ('name', 100),
('title', 300),
('phone', 300),
('email', 300),
('address', 300),
('link', 300),
('comments', 5000),
)
@register_search
class ContactGroupIndex(SearchIndex):
model = models.ContactGroup
fields = (
('name', 100),
('slug', 110),
('description', 500),
)
@register_search
class ContactRoleIndex(SearchIndex):
model = models.ContactRole
fields = (
('name', 100),
('slug', 110),
('description', 500),
)
@register_search
class TenantIndex(SearchIndex):
model = models.Tenant
fields = (
('name', 100),
('slug', 110),
('description', 500),
('comments', 5000),
)
@register_search
class TenantGroupIndex(SearchIndex):
model = models.TenantGroup
fields = (
('name', 100),
('slug', 110),
('description', 500),
) )
filterset = tenancy.filtersets.ContactFilterSet
table = tenancy.tables.ContactTable
url = 'tenancy:contact_list'

70
netbox/utilities/fields.py
View File

@ -1,3 +1,6 @@
from collections import defaultdict
from django.contrib.contenttypes.fields import GenericForeignKey
from django.core.validators import RegexValidator from django.core.validators import RegexValidator
from django.db import models from django.db import models
@ -71,3 +74,70 @@ class NaturalOrderingField(models.CharField):
[self.target_field], [self.target_field],
kwargs, kwargs,
) )
class RestrictedGenericForeignKey(GenericForeignKey):
# Replicated largely from GenericForeignKey. Changes include:
# 1. Capture restrict_params from RestrictedPrefetch (hack)
# 2. If restrict_params is set, call restrict() on the queryset for
# the related model
def get_prefetch_queryset(self, instances, queryset=None):
restrict_params = {}
# Compensate for the hack in RestrictedPrefetch
if type(queryset) is dict:
restrict_params = queryset
elif queryset is not None:
raise ValueError("Custom queryset can't be used for this lookup.")
# For efficiency, group the instances by content type and then do one
# query per model
fk_dict = defaultdict(set)
# We need one instance for each group in order to get the right db:
instance_dict = {}
ct_attname = self.model._meta.get_field(self.ct_field).get_attname()
for instance in instances:
# We avoid looking for values if either ct_id or fkey value is None
ct_id = getattr(instance, ct_attname)
if ct_id is not None:
fk_val = getattr(instance, self.fk_field)
if fk_val is not None:
fk_dict[ct_id].add(fk_val)
instance_dict[ct_id] = instance
ret_val = []
for ct_id, fkeys in fk_dict.items():
instance = instance_dict[ct_id]
ct = self.get_content_type(id=ct_id, using=instance._state.db)
if restrict_params:
# Override the default behavior to call restrict() on each model's queryset
qs = ct.model_class().objects.filter(pk__in=fkeys).restrict(**restrict_params)
ret_val.extend(qs)
else:
# Default behavior
ret_val.extend(ct.get_all_objects_for_this_type(pk__in=fkeys))
# For doing the join in Python, we have to match both the FK val and the
# content type, so we use a callable that returns a (fk, class) pair.
def gfk_key(obj):
ct_id = getattr(obj, ct_attname)
if ct_id is None:
return None
else:
model = self.get_content_type(
id=ct_id, using=obj._state.db
).model_class()
return (
model._meta.pk.get_prep_value(getattr(obj, self.fk_field)),
model,
)
return (
ret_val,
lambda obj: (obj.pk, obj.__class__),
gfk_key,
True,
self.name,
False,
)

28
netbox/utilities/querysets.py
View File

@ -1,9 +1,35 @@
from django.db.models import QuerySet from django.db.models import Prefetch, QuerySet
from users.constants import CONSTRAINT_TOKEN_USER from users.constants import CONSTRAINT_TOKEN_USER
from utilities.permissions import permission_is_exempt, qs_filter_from_constraints from utilities.permissions import permission_is_exempt, qs_filter_from_constraints
class RestrictedPrefetch(Prefetch):
"""
Extend Django's Prefetch to accept a user and action to be passed to the
`restrict()` method of the related object's queryset.
"""
def __init__(self, lookup, user, action='view', queryset=None, to_attr=None):
self.restrict_user = user
self.restrict_action = action
super().__init__(lookup, queryset=queryset, to_attr=to_attr)
def get_current_queryset(self, level):
params = {
'user': self.restrict_user,
'action': self.restrict_action,
}
if qs := super().get_current_queryset(level):
return qs.restrict(**params)
# Bit of a hack. If no queryset is defined, pass through the dict of restrict()
# kwargs to be handled by the field. This is necessary e.g. for GenericForeignKey
# fields, which do not permit setting a queryset on a Prefetch object.
return params
class RestrictedQuerySet(QuerySet): class RestrictedQuerySet(QuerySet):
def restrict(self, user, action='view'): def restrict(self, user, action='view'):

50
netbox/utilities/templates/search/searchbar.html
View File

@ -1,50 +0,0 @@
<form class="input-group" action="{% url 'search' %}" method="get">
<input
name="q"
type="text"
aria-label="Search"
placeholder="Search"
class="form-control"
value="{{ request.GET.q|escape }}"
/>
<input name="obj_type" hidden type="text" class="search-obj-type" />
<span class="input-group-text search-obj-selected">All Objects</span>
<button type="button" aria-expanded="false" data-bs-toggle="dropdown" class="btn dropdown-toggle">
<i class="mdi mdi-filter-variant"></i>
</button>
<ul class="dropdown-menu dropdown-menu-end search-obj-selector">
{% for option in options %}
{% if option.items|length == 0 %}
<li>
<button class="dropdown-item" type="button" data-search-value="{{ option.value }}">
{{ option.label }}
</button>
</li>
{% else %}
<li><h6 class="dropdown-header">{{ option.label }}</h6></li>
{% endif %}
{% for item in option.items %}
<li>
<button class="dropdown-item" type="button" data-search-value="{{ item.value }}">
{{ item.label }}
</button>
</li>
{% endfor %}
{% if forloop.counter != options|length %}
<li><hr class="dropdown-divider" /></li>
{% endif %}
{% endfor %}
</ul>
<button class="btn btn-primary" type="submit">
<i class="mdi mdi-magnify"></i>
</button>
</form>

18
netbox/utilities/templatetags/search.py
View File

@ -1,18 +0,0 @@
from typing import Dict
from django import template
from netbox.forms import SearchForm
register = template.Library()
search_form = SearchForm()
@register.inclusion_tag("search/searchbar.html")
def search_options(request) -> Dict:
# Provide search options to template.
return {
'options': search_form.get_options(),
'request': request,
}

22
netbox/utilities/utils.py
View File

@ -1,6 +1,7 @@
import datetime import datetime
import decimal import decimal
import json import json
import re
from decimal import Decimal from decimal import Decimal
from itertools import count, groupby from itertools import count, groupby
@ -9,6 +10,7 @@ from django.core.serializers import serialize
from django.db.models import Count, OuterRef, Subquery from django.db.models import Count, OuterRef, Subquery
from django.db.models.functions import Coalesce from django.db.models.functions import Coalesce
from django.http import QueryDict from django.http import QueryDict
from django.utils.html import escape
from jinja2.sandbox import SandboxedEnvironment from jinja2.sandbox import SandboxedEnvironment
from mptt.models import MPTTModel from mptt.models import MPTTModel
@ -472,3 +474,23 @@ def clean_html(html, schemes):
attributes=ALLOWED_ATTRIBUTES, attributes=ALLOWED_ATTRIBUTES,
protocols=schemes protocols=schemes
) )
def highlight_string(value, highlight, trim_pre=None, trim_post=None, trim_placeholder='...'):
"""
Highlight a string within a string and optionally trim the pre/post portions of the original string.
"""
# Split value on highlight string
try:
pre, match, post = re.split(fr'({highlight})', value, maxsplit=1, flags=re.IGNORECASE)
except ValueError:
# Match not found
return escape(value)
# Trim pre/post sections to length
if trim_pre and len(pre) > trim_pre:
pre = trim_placeholder + pre[-trim_pre:]
if trim_post and len(post) > trim_post:
post = post[:trim_post] + trim_placeholder
return f'{escape(pre)}<mark>{escape(match)}</mark>{escape(post)}'

64
netbox/virtualization/search.py
View File

@ -1,33 +1,49 @@
import virtualization.filtersets
import virtualization.tables
from dcim.models import Device
from netbox.search import SearchIndex, register_search from netbox.search import SearchIndex, register_search
from utilities.utils import count_related from . import models
from virtualization.models import Cluster, VirtualMachine
@register_search() @register_search
class ClusterIndex(SearchIndex): class ClusterIndex(SearchIndex):
model = Cluster model = models.Cluster
queryset = Cluster.objects.prefetch_related('type', 'group').annotate( fields = (
device_count=count_related(Device, 'cluster'), vm_count=count_related(VirtualMachine, 'cluster') ('name', 100),
('comments', 5000),
) )
filterset = virtualization.filtersets.ClusterFilterSet
table = virtualization.tables.ClusterTable
url = 'virtualization:cluster_list'
@register_search() @register_search
class ClusterGroupIndex(SearchIndex):
model = models.ClusterGroup
fields = (
('name', 100),
('slug', 110),
('description', 500),
)
@register_search
class ClusterTypeIndex(SearchIndex):
model = models.ClusterType
fields = (
('name', 100),
('slug', 110),
('description', 500),
)
@register_search
class VirtualMachineIndex(SearchIndex): class VirtualMachineIndex(SearchIndex):
model = VirtualMachine model = models.VirtualMachine
queryset = VirtualMachine.objects.prefetch_related( fields = (
'cluster', ('name', 100),
'tenant', ('comments', 5000),
'tenant__group', )
'platform',
'primary_ip4',
'primary_ip6', @register_search
class VMInterfaceIndex(SearchIndex):
model = models.VMInterface
fields = (
('name', 100),
('description', 500),
) )
filterset = virtualization.filtersets.VirtualMachineFilterSet
table = virtualization.tables.VirtualMachineTable
url = 'virtualization:virtualmachine_list'

42
netbox/wireless/search.py
View File

@ -1,26 +1,32 @@
import wireless.filtersets
import wireless.tables
from dcim.models import Interface
from netbox.search import SearchIndex, register_search from netbox.search import SearchIndex, register_search
from utilities.utils import count_related from . import models
from wireless.models import WirelessLAN, WirelessLink
@register_search() @register_search
class WirelessLANIndex(SearchIndex): class WirelessLANIndex(SearchIndex):
model = WirelessLAN model = models.WirelessLAN
queryset = WirelessLAN.objects.prefetch_related('group', 'vlan').annotate( fields = (
interface_count=count_related(Interface, 'wireless_lans') ('ssid', 100),
('description', 500),
('auth_psk', 2000),
) )
filterset = wireless.filtersets.WirelessLANFilterSet
table = wireless.tables.WirelessLANTable
url = 'wireless:wirelesslan_list'
@register_search() @register_search
class WirelessLANGroupIndex(SearchIndex):
model = models.WirelessLANGroup
fields = (
('name', 100),
('slug', 110),
('description', 500),
)
@register_search
class WirelessLinkIndex(SearchIndex): class WirelessLinkIndex(SearchIndex):
model = WirelessLink model = models.WirelessLink
queryset = WirelessLink.objects.prefetch_related('interface_a__device', 'interface_b__device') fields = (
filterset = wireless.filtersets.WirelessLinkFilterSet ('ssid', 100),
table = wireless.tables.WirelessLinkTable ('description', 500),
url = 'wireless:wirelesslink_list' ('auth_psk', 2000),
)