* Introduce reusable BackgroundJob framework
A new abstract class can be used to implement job function classes. It
handles the necessary logic for starting and stopping jobs, including
exception handling and rescheduling of recurring jobs.
This commit also includes the migration of data source jobs to the new
framework.
* Restore using import_string for jobs
Using the 'import_string()' utility from Django allows the job script
class to be simplified, as module imports no longer need to avoid loops.
This should make it easier to queue and maintain jobs.
* Use SyncDataSourceJob for management command
Instead of maintaining two separate job execution logics, the same job
is now used for both background and interactive execution.
* Implement BackgroundJob for running scripts
The independent implementations of interactive and background script
execution have been merged into a single BackgroundJob implementation.
* Fix documentation of model features
* Ensure consitent code style
* Introduce reusable ScheduledJob
A new abstract class can be used to implement job function classes that
specialize in scheduling. These use the same logic as regular
BackgroundJobs, but ensure that they are only scheduled once at any given
time.
* Introduce reusable SystemJob
A new abstract class can be used to implement job function classes that
specialize in system background tasks (e.g. synchronization or
housekeeping). In addition to the features of the BackgroundJob and
ScheduledJob classes, these implement additional logic to not need to be
bound to an existing NetBox object and to setup job schedules on plugin
load instead of an interactive request.
* Add documentation for jobs framework
* Revert "Use SyncDataSourceJob for management"
This partially reverts commit db591d4. The 'run_now' parameter of
'enqueue()' remains, as its being used by following commits.
* Merge enqueued status into JobStatusChoices
* Fix logger for ScriptJob
* Remove job name for scripts
Because scripts are already linked through the Job Instance field, the
name is displayed twice. Removing this reduces redundancy and opens up
the possibility of simplifying the BackgroundJob framework in future
commits.
* Merge ScheduledJob into BackgroundJob
Instead of using separate classes, the logic of ScheduledJob is now
merged into the generic BackgroundJob class. This allows reusing the
same logic, but dynamically deciding whether to enqueue the same job
once or multiple times.
* Add name attribute for BackgroundJob
Instead of defining individual names on enqueue, BackgroundJob classes
can now set a job name in their meta class. This is equivalent to other
Django classes and NetBox scripts.
* Drop enqueue_sync_job() method from DataSource
* Import ScriptJob directly
* Relax requirement for Jobs to reference a specific object
* Rename 'run_now' arg on Job.enqueue() to 'immediate'
* Fix queue lookup in Job enqueue
* Collapse SystemJob into BackgroundJob
* Remove legacy JobResultStatusChoices
ChoiceSet was moved to core in 40572b5.
* Use queue 'low' for system jobs by default
System jobs usually perform low-priority background tasks and therefore
can use a different queue than 'default', which is used for regular jobs
related to specific objects.
* Add test cases for BackgroundJob handling
* Fix enqueue interval jobs
As the job's name is set by enqueue(), it must not be passed in handle()
to avoid duplicate kwargs with the same name.
* Honor schedule_at for job's enqueue_once
Not only can a job's interval change, but so can the time at which it is
scheduled to run. If a specific scheduled time is set, it will also be
checked against the current job schedule. If there are any changes, the
job is rescheduled with the new time.
* Switch BackgroundJob to regular methods
Instead of using a class method for run(), a regular method is used for
this purpose. This gives the possibility to add more convenience methods
in the future, e.g. for interacting with the job object or for logging,
as implemented for scripts.
* Fix background tasks documentation
* Test enqueue in combination with enqueue_once
* Rename background jobs to tasks (to differentiate from RQ)
* Touch up docs
* Revert "Use queue 'low' for system jobs by default"
This reverts commit b17b2050df.
* Remove system background job
This commit reverts commits 4880d81 and 0b15ecf. Using the database
'connection_created' signal for job registration feels a little wrong at
this point, as it would trigger registration very often. However, the
background job framework is prepared for this use case and can be used
by plugins once the auto-registration of jobs is solved.
* Fix runscript management command
Defining names for background jobs was disabled with fb75389. The
preceeding changes in 257976d did forget the management command.
* Use regular imports for ScriptJob
* Rename BackgroundJob to JobRunner
---------
Co-authored-by: Jeremy Stretch <jstretch@netboxlabs.com>
* Do not delete all search indexes when reindexing specific models
* Clear all indexes only if neither --lazy nor a list of models are
specified for "manage.py reindex"
* Otherwise, clear the index for a model immediately before rebuilding
it
* Separated clearing from re-indexing the search cache
* 12794 change User ref to get_user_model
* 12794 call get_user_model once in tests
* 12794 call get_user_model once in tests
* 12794 use settings.AUTH_USER_MODEL for FK reference
* Reference database object by GFK when running scripts & reports via UI
* Reference database object by GFK when running scripts & reports via API
* Remove old enqueue_job() method
* Enable filtering jobs by object
* Introduce ObjectJobsView
* Add tabbed views for report & script jobs
* Add object_id to JobSerializer
* Move generic relation to JobsMixin
* Clean up old naming
* WIP
* WIP
* Add git sync
* Fix file hashing
* Add last_synced to DataSource
* Build out UI & API resources
* Add status field to DataSource
* Add UI control to sync data source
* Add API endpoint to sync data sources
* Fix display of DataSource job results
* DataSource password should be write-only
* General cleanup
* Add data file UI view
* Punt on HTTP, FTP support for now
* Add DataSource URL validation
* Add HTTP proxy support to git fetcher
* Add management command to sync data sources
* DataFile REST API endpoints should be read-only
* Refactor fetch methods into backend classes
* Replace auth & git branch fields with general-purpose parameters
* Fix last_synced time
* Render discrete form fields for backend parameters
* Enable dynamic edit form for DataSource
* Register DataBackend classes in application registry
* Add search indexers for DataSource, DataFile
* Add single & bulk delete views for DataFile
* Add model documentation
* Convert DataSource to a primary model
* Introduce pre_sync & post_sync signals
* Clean up migrations
* Rename url to source_url
* Clean up filtersets
* Add API & filterset tests
* Add view tests
* Add initSelect() to HTMX refresh handler
* Render DataSourceForm fieldsets dynamically
* Update compiled static resources
* Initial work on new search backend
* Clean up search backends
* Return only the most relevant result per object
* Clear any pre-existing cached entries on cache()
* #6003: Implement global search functionality for custom field values
* Tweak field weights & document guidance
* Extend search() to accept a lookup type
* Move get_registry() out of SearchBackend
* Enforce object permissions when returning search results
* Add indexers for remaining models
* Avoid calling remove() on non-cacheable objects
* Use new search backend by default
* Extend search backend to filter by object type
* Clean up search view form
* Enable specifying lookup logic
* Add indexes for value field
* Remove object type selector from search bar
* Introduce SearchTable and enable HTMX for results
* Enable pagination
* Remove legacy search backend
* Cleanup
* Use a UUID for CachedValue primary key
* Refactoring search methods
* Define max search results limit
* Extend reindex command to support specifying particular models
* Add clear() and size to SearchBackend
* Optimize bulk caching performance
* Highlight matched portion of field value
* Performance improvements for reindexing
* Started on search tests
* Cleanup & docs
* Documentation updates
* Clean up SearchIndex
* Flatten search registry to register by app_label.model_name
* Clean up search backend classes
* Clean up RestrictedGenericForeignKey and RestrictedPrefetch
* Resolve migrations conflict
* Added JobResult form filtersets
* Change housekeeping cleanup delete from `_raw_delete` to `delete` to make sure scheduled tasks are cancelled
* Change default sort of JobResult table to -created
* Added `delete` override to `JobResult` to remove scheduled tasks from RQ when a JobResult is deleted
* Updated js/css dist files. Will need to be redone when develop is merged to feature.
