skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Django Remote Submission

Abstract

The Django Remote Submission (DRS) is a Django (Django, n.d.) application to manage long running job submission, including starting the job, saving logs, and storing results. It is an independent project available as a standalone pypi package (PyPi, n.d.). It can be easily integrated in any Django project. The source code is freely available as a GitHub repository (django-remote-submission, n.d.). To run the jobs in background, DRS takes advantage of Celery (Celery, n.d.), a powerful asynchronous job queue used for running tasks in the background, and the Redis Server (Redis, n.d.), an in-memory data structure store. Celery uses brokers to pass messages between a Django Project and the Celery workers. Redis is the message broker of DRS. In addition DRS provides real time monitoring of the progress of Jobs and associated logs. Through the Django Channels project (Channels, n.d.), and the usage of Web Sockets, it is possible to asynchronously display the Job Status and the live Job output (standard output and standard error) on a web page.

Authors:
ORCiD logo [1];  [1]; ORCiD logo [1]
  1. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Publication Date:
Research Org.:
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Sponsoring Org.:
USDOE
OSTI Identifier:
1394387
Grant/Contract Number:
AC05-00OR22725
Resource Type:
Journal Article: Accepted Manuscript
Journal Name:
Journal of Open Source Software
Additional Journal Information:
Journal Volume: 2; Journal Issue: 16; Journal ID: ISSN 2475-9066
Publisher:
Open Source Initiative - NumFOCUS
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING

Citation Formats

Doucet, Mathieu, Hobson, Tanner C., and Ferraz Leal, Ricardo Miguel. Django Remote Submission. United States: N. p., 2017. Web. doi:10.21105/joss.00366.
Doucet, Mathieu, Hobson, Tanner C., & Ferraz Leal, Ricardo Miguel. Django Remote Submission. United States. doi:10.21105/joss.00366.
Doucet, Mathieu, Hobson, Tanner C., and Ferraz Leal, Ricardo Miguel. 2017. "Django Remote Submission". United States. doi:10.21105/joss.00366. https://www.osti.gov/servlets/purl/1394387.
@article{osti_1394387,
title = {Django Remote Submission},
author = {Doucet, Mathieu and Hobson, Tanner C. and Ferraz Leal, Ricardo Miguel},
abstractNote = {The Django Remote Submission (DRS) is a Django (Django, n.d.) application to manage long running job submission, including starting the job, saving logs, and storing results. It is an independent project available as a standalone pypi package (PyPi, n.d.). It can be easily integrated in any Django project. The source code is freely available as a GitHub repository (django-remote-submission, n.d.). To run the jobs in background, DRS takes advantage of Celery (Celery, n.d.), a powerful asynchronous job queue used for running tasks in the background, and the Redis Server (Redis, n.d.), an in-memory data structure store. Celery uses brokers to pass messages between a Django Project and the Celery workers. Redis is the message broker of DRS. In addition DRS provides real time monitoring of the progress of Jobs and associated logs. Through the Django Channels project (Channels, n.d.), and the usage of Web Sockets, it is possible to asynchronously display the Job Status and the live Job output (standard output and standard error) on a web page.},
doi = {10.21105/joss.00366},
journal = {Journal of Open Source Software},
number = 16,
volume = 2,
place = {United States},
year = 2017,
month = 8
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Save / Share:
  • This U.S. Environmental Protection Agency notice establishes final rules requiring manufacturers of 1981 and earlier model year motor vehicles and motor vehicle engines to submit altitude adjustment instructions for in-use motor vehicles and motor vehicle engines within one year of the effective date. Similar instructions must also be submitted for 1982 and later model year motor vehicles and motor vehicle engines within 30 days from the date on which a certificate of conformity is issued by EPA. These rules are intended to improve the emission control performance of 1981 and later model year light-duty vehicles and light-duty trucks and 1980more » and later model year motorcycles that are not designed for operation at high altitude (over 4000 ft above sea level). The rules will become effective on 11/7/80.« less
  • PanDA (Production and Distributed Analysis) is the workload management system of the ATLAS experiment, used to run managed production and user analysis jobs on the grid. As a late-binding, pilot-based system, the maintenance of a smooth and steady stream of pilot jobs to all grid sites is critical for PanDA operation. The ATLAS Computing Facility (ACF) at BNL, as the ATLAS Tier1 center in the US, operates the pilot submission systems for the US. This is done using the PanDA 'AutoPilot' scheduler component which submits pilot jobs via Condor-G, a grid job scheduling system developed at the University of Wisconsin-Madison.more » In this paper, we discuss the operation and performance of the Condor-G pilot submission at BNL, with emphasis on the challenges and issues encountered in the real grid production environment. With the close collaboration of Condor and PanDA teams, the scalability and stability of the overall system has been greatly improved over the last year. We review improvements made to Condor-G resulting from this collaboration, including isolation of site-based issues by running a separate Gridmanager for each remote site, introduction of the 'Nonessential' job attribute to allow Condor to optimize its behavior for the specific character of pilot jobs, better understanding and handling of the Gridmonitor process, as well as better scheduling in the PanDA pilot scheduler component. We will also cover the monitoring of the health of the system.« less
  • Abstract not provided.
  • Abstract not provided.