NEWT is a collection of REST based services that allow scientists, programmers, staff, and the public to write web apps for HPC. These are services designed to make HPC computing and data resources highly usable via a web browser. Web browser interaction is often mediated through scripts and web apps that poll, aggregate, and mashup the content from the restful API's URLs. As such the API is not really a set of web pages, but a set of well defined URLs that web apps can invoke which act as information sources and sinks.

We take a pragmatic approach to building HPC-to-web methods, acknowledging a restricted vocabulary of methods gathered from many years of watching how science teams use compute resources and data stores. Starting, stopping, and monitoring batch jobs, WAN data movement, file and data management, and UNIX-like utility functions are the functions NEWT addresses. The API is built around easy-to-use, concise, programmable interfaces. NEWT provides LDAP/PKI authentication mechanisms for read-write management of jobs and data.

What does the API do?

  • Authentication - Log in to the NEWT service with your username and password
  • Accounting Information - Get information about your account through the NERSC Information Management (NIM) system
  • Files - Upload, download files. Browse directories
  • Commands - Run UNIX commands and utilities
  • Jobs - Submit jobs to the batch system
  • Store - Persistent storage layer for remote client applications

API Overview

Transactions against NEWT take the form

For generic resources (account, auth, store):
VERB newt_base_url/resource_type/path?arg1=val1&arg2=val2

OR

For system specific resources (status, file, command, job, queue):
VERB newt_base_url/resource_type/system/path?arg1=val1&arg2=val2

The API is based on a set of actions (verbs) that categorize the different services. Each verb may relate to different actions depending on the METHOD in the HTTP headers. All transactions take place over SSL (https). Most actions require the user to be authenticated first. Data is returned by default in JSON format

VERBHTTP verbs - One of {GET, POST, PUT, DELETE}
newt_base_urlbase URL e.g. https://newt.nersc.gov/newt
resource_typeOne of {auth, account, store, status, file, command, job, queue}
systemOne of {edison, cori, pdsf, archive, datatran}
pathPath to end resource
?arg1=val1&arg2=val2arguments for the request

The following table describes the NERSC system specific resources available through NEWT:
System StatusFilesBatch QueueRun CommandsRun Jobs
edison
cori
datatran
archive
pdsf

The NEWT API

This section describes the currently implemented methods in NEWT, and includes worked examples to test existing functionality. It assumes that your web application will be running within the NERSC domain (though this is not required). Examples require a NERSC login for them to function.

If you're not a NERSC user and have questions please email us.

First Steps

Your NEWT web application can start as a simple web page. Internally NEWT uses jQuery to handle some AJAX data movement. We recommend you do to, but the API is pure REST so that's really up to you. Every NEWT application starts with including these lines in the <head> of your web page.

<script src="https://newt.nersc.gov/js/jquery-1.7.2.js"></script>
<script src="https://newt.nersc.gov/js/newt.js"></script>

Authentication

Including the lines above in your web page means that authentication is taken care of automatically. You should see an authentication status bar/form slide down at the top of your web page.

This will transparently add functionality for for logging in, logging out, and checking authentication status into your application. It also defines a javascript variable named newt_base_url which is the base URL for many of the examples below. Everything below assumes that NEWT is installed at https://newt.nersc.gov/newt.

API Reference Resource Inventory

Authentication Status

MethodGET [newt_base_url]/login/
OutputJSON { "auth": boolean, "username": string, "session_lifetime": seconds, "newt_sessionid": string}
SemanticsGet current auth status, username and number of seconds in session

Example: https://newt.nersc.gov/newt/login/

Authentication Login

MethodPOST [newt_base_url]/login
Inputusername="my_username"
password="my_password"
OutputJSON { "auth": boolean, "username": string, "session_lifetime": seconds, "newt_sessionid": string }
Set-Cookie response header contains session ID
SemanticsPosts authentication information. If authentication is successful, the session cookie is set. Returns JSON string with authentication status

Example: https://newt.nersc.gov/newt/login/?next=/newt/

<form method="post" 
action="https://newt.nersc.gov/newt/login/"> <div> <div class="form-row"> <label for="id_username">Username</label> <input id="id_username" type="text" name="username" /> </div> <div class="form-row"> <label for="id_password">Password</label> <input type="password" name="password" id="id_password" /> </div> </div> <div class="submit-row"> <input type="submit" value="login" /> <input type="hidden" name="next" value="" /> </div> </form>

Authentication Logout

MethodGET [newt_base_url]/logout
OutputJSON { "auth": boolean, "username": string, "session_lifetime": seconds}
SemanticsLog the user out and returns current authentication status

Example: https://newt.nersc.gov/newt/logout/

Status Get System Status

MethodGET [newt_base_url]/status/[machine]
OutputJSON { "status": string, "machine": string }
SemanticsGet System Availability Status. The returned status attribute will either be "up", "down" or "unknown"

Example: https://newt.nersc.gov/newt/status/cori

Status Get Status for All Systems

MethodGET [newt_base_url]/status/
OutputJSON Array [{ "status": string, "system": string }, ... ]
SemanticsGet System Availability Status. The returned status attribute will either be "up", "down" or "unknown"

Example: https://newt.nersc.gov/newt/status/

Status Get MOTD

MethodGET [newt_base_url]/status/motd
OutputMessage of the day (text)
SemanticsGet a plain text summary of the current NERSC status (Message of the Day).

Example: https://newt.nersc.gov/newt/status/motd

Files Directory Listing

MethodGET [newt_base_url]/file/[machine]/[path]/
OutputJSON Array [{"perms": string, "hardlinks": string, "user": string, "group": string, "size": string, "date": string, "name": string}, ... ]
SemanticsGet directory listing if [path] is a directory; get listing for individual file if [path] is a file.

Example: https://newt.nersc.gov/newt/file/cori/tmp/

See also the ajax directory listing demo

Files Download a File

MethodGET [newt_base_url]/file/[machine]/[path]?view=read
OutputBinary file located on [machine] at [path]
SemanticsReturn the file to the user. Note the view paramater must be set to “read” otherwise it will return a directory listing

Example: https://newt.nersc.gov/newt/file/cori/etc/motd?view=read

Files Create/Update a File

MethodPUT [newt_base_url]/file/[machine]/[path]
Input:file contents
OutputJSON string: {status: "OK", location: string}
SemanticsUpdate/create file with given contents. Returns request status and location of file.

Example:

Files Upload a File

MethodPOST [newt_base_url]/file/[machine]/[path]/
Output200 response code if successful
SemanticsUpload a file on [machine] at the give [path]

Example:

<form action="https://newt.nersc.gov/newt/file/cori/tmp/" method="post" enctype="multipart/form-data">
<label for="id_file">File:</label>
<input type="file" name="file" id="id_file" />
<input type="submit" value="upload" /> </form>

Jobs Run a command

MethodPOST [newt_base_url]/command/[machine]
Inputexecutable="/path/to/exec options" # full command that needs to be run
Optional parameters:
loginenv=true # setting to true initializes a full bash login env before running
OutputJSON { "status": ["OK" | "ERROR"], "output": string, "error": string }
SemanticsRun a command (synchronously) on the compute system and get back output. Optionally setting "loginenv=true" will initialize a full bash login env before running (this could result in a small additional delay). Also note that "status" can only capture higher level failures, and not failures within the command. Make sure you check "output" and "error" for the results.

Example:

<form action="https://newt.nersc.gov/newt/command/cori/" method="post">
<input type="text" name="executable" value="/bin/ls" />
<input type="submit" value="Run" /> </form>

Jobs View Queue

MethodGET [newt_base_url]/queue/[machine]/<sacct>
Input Optional Parameters: (add ?param1=val1&param2=val2 etc. to URL)
index=N # index for first record
limit=N # number of records
<key>=<value> # restrict records to ones with key=value eg. queue=regular
OutputJSON Array: [{"jobid": string, "name": string, "user": string, "timeuse": string, "status": string, "queue": string}, ... ]
SemanticsGet the results of a sqs or qstat on a given machine as a JSON array. Note that the default (without the optional /sacct suffix) may not include completed jobs since they are no longer in the queue. Append /sacct (see example) to your URL string for completed jobs (with 24 hours) as well.

Example: https://newt.nersc.gov/newt/queue/cori/

SACCT Example (Completed Job info): https://newt.nersc.gov/newt/queue/cori/sacct

Filter Example: https://newt.nersc.gov/newt/queue/cori/?index=1&limit=3&status=R

See also the ajax view queue demo

Jobs Submit Job to Queue

MethodPOST [newt_base_url]/queue/[machine]/
Input jobfile=/path/to/script # An existing batch submit file on target host
jobscript="#Script contents" # Only set one of jobscript or jobfile
OutputJSON String: { "status": ["OK" | "ERROR"], "error": string, "jobid" : string }
SemanticsSubmit a script to the batch system (qsub). You can either specify and existing batch script or post a script that will be submitted. It returns a jobid in the batch queue

Example A: Run jobfile on cori:

<form action="https://newt.nersc.gov/newt/queue/cori/" method="post">
<input size=40 name="jobfile" type="text" />
<input type="submit" value="submit" />
</form>


Example B: Submit jobscript to cori:

<form action="https://newt.nersc.gov/newt/queue/cori/" method="post">
<textarea rows=3 cols=40 name="jobscript">
#!/bin/bash -l
#SBATCH -N 1
srun -n 32 /bin/hostname
</textarea>
<input type="submit" value="submit" />
</form>


Jobs View Job in Queue

MethodGET [newt_base_url]/queue/[machine]/[jobid]
OutputJSON String: {"jobid": string, "name": string, "user": string, "timeuse": string, "status": string, "queue": string}
SemanticsGet the results of a "squeue or qstat jobid" on a given machine as a JSON string. Note that this may not include completed jobs since they are no longer in the queue. Append /sacct (see below) to your string for completed jobs (with 24 hours) as well.

Jobs View Job Account Information (including recently completed jobs)

MethodGET [newt_base_url]/queue/[machine]/[jobid]/sacct
OutputJSON String: {"jobid": string, "name": string, "user": string, "timeuse": string, "status": string, "queue": string}
SemanticsGet the results of a "sacct jobid" on a given machine as a JSON string. This includes recently (within 24 hours) completed jobs

Jobs Delete Job from Queue

MethodDELETE [newt_base_url]/queue/[machine]/[jobid]
OutputJSON String: {"status": ["OK | ERROR"], "output": string, "error": string}
SemanticsGet a job from the batch queue

Liststore

This application is a simple frontend to a shared JSON object storage database. It allows applications to store JSON data in a persistent fashion.

Liststore Get Available Stores

MethodGET [newt_base_url]/store/
OutputJSON ["Store1", "Store2"]
SemanticsGet a list of available stores.

Example: https://newt.nersc.gov/newt/store/

Liststore Create Store

MethodPOST [newt_base_url]/store/[store_name]
InputOptional: JSON {"doc1":["one", "two", "three"], "doc2":{"sam":"snead", "tiger":"woods"}
Set Header 'content-type':'application/json; charset=utf-8'
OutputJSON String: {"status": ["OK | ERROR"], "output": string, "error": string}
SemanticsCreate a store (with optional document), redirect to the newly created URL.

Example:

<form id="dbcreate" action="https://newt.nersc.gov/newt/store/" method='post'>
<input type="submit" value="Create" />
</form>

Liststore Delete Store

MethodDELETE [newt_base_url]/store/[store_name]/
OutputJSON String: {"status": ["OK | ERROR"], "output": string, "error": string}
SemanticsDelete a store.

Liststore Get Store Contents

MethodGET [newt_base_url]/store/[store_name]/
OutputJSON Array [ ]
SemanticsGet all documents in a store.

Example: https://newt.nersc.gov/newt/store/dbtest/

Liststore Query Store

MethodGET [newt_base_url]/store/[store_name]/?query={"key":"val"}
OutputJSON Array [ ]
SemanticsGet results of a query, using mongo query syntax.

Liststore Get Store Permissions

MethodGET [newt_base_url]/store/[store_name]/admin
OutputJSON { "username": string, "_id": string, "writers": [], "collection": string, "readers": [] }
SemanticsGet permissions for a store.

Example: https://newt.nersc.gov/newt/store/dbtest/admin

Liststore Update Store Permissions

MethodPUT [newt_base_url]/store/[store_name]/admin
InputJSON {"readers": ["group1", "group2"], "writers": ["group1"]}
Output
SemanticsUpdate list of readers and writers that can access store. Readers and writers can be any valid UNIX group

ListstoreCreate Document

MethodPOST [newt_base_url]/store/[store_name]
InputJSON {"doc1":["one", "two", "three"], "doc2":{"sam":"snead", "tiger":"woods"}
Set Header 'content-type':'application/json; charset=utf-8'
OutputJSON String: {"_id": string}
SemanticsCreate a document, return document ID.

Liststore Get Document Contents

MethodGET [newt_base_url]/store/[store_name]/[document_id]
OutputJSON {"sam":"snead", "tiger":"woods"}
SemanticsGet contents of a document.

Liststore Update Document

MethodPUT [newt_base_url]/store/[store_name]/[document_id]
InputJSON {"sam":"snead", "tiger":"woods"}
OutputJSON String: {"status": ["OK | ERROR"], "output": string, "error": string}
SemanticsUpdate contents of a document, for the given fields.

Account Get Account information from IRIS

MethodPOST [newt_base_url]/account/iris
Inputquery: function_name(param: value){field1,field2,...}
OutputJSON String: {"data": {JSON}
SemanticsGet the information from IRIS for a given resource as a JSON object. We will be updating this section with more details and example API calls, but please contact NERSC support staff if you wish to use NEWT with the new IRIS API
Example // Escape the quotes as shown below to makes sure it parses as graphql:

$.newt_ajax({
url: "/account/iris",
type: "POST",
data: {"query": 'user(username: \\\"myuser\\\"){uid, email}'},
success: function(out) { console.log(out.data) }
}
})


// Outputs:

{"newt":{"users":[{"uid":12345,"email":"myuser@acme.com"}]}}

Please refer to NEWT IRIS docs for more detailed information on how to invoke this API