NEWT is a collection of REST based services that allow scientists, programmers, staff, and the public to write web apps for HPC. These are services designed to make HPC computing and data resources highly usable via a web browser. Web browser interaction is often mediated through scripts and web apps that poll, aggregate, and mashup the content from the restful API's URLs. As such the API is not really a set of web pages, but a set of well defined URLs that web apps can invoke which act as information sources and sinks.

We take a pragmatic approach to building HPC-to-web methods, acknowledging a restricted vocabulary of methods gathered from many years of watching how science teams use compute resources and data stores. Starting, stopping, and monitoring batch jobs, WAN data movement, file and data management, and UNIX-like utility functions are the functions NEWT addresses. The API is built around easy-to-use, concise, programmable interfaces. NEWT provides LDAP/PKI authentication mechanisms for read-write management of jobs and data.

What does the API do?

Authentication - Log in to the NEWT service with your username and password
Accounting Information - Get information about your account through the NERSC Information Management (NIM) system
Files - Upload, download files. Browse directories
Commands - Run UNIX commands and utilities
Jobs - Submit jobs to the batch system

API Overview

Transactions against NEWT take the form

For generic resources (account, auth, store):
VERB newt_base_url/resource_type/path?arg1=val1&arg2=val2

For system specific resources (status, file, command, job, queue):
VERB newt_base_url/resource_type/system/path?arg1=val1&arg2=val2

The API is based on a set of actions (verbs) that categorize the different services. Each verb may relate to different actions depending on the METHOD in the HTTP headers. All transactions take place over SSL (https). Most actions require the user to be authenticated first. Data is returned by default in JSON format

VERB HTTP verbs - One of {GET, POST, PUT, DELETE}

newt_base_url base URL e.g. https://newt.nersc.gov/newt

resource_type One of {auth, account, store, status, file, command, job, queue}

system One of {edison, cori, pdsf, archive, datatran}

path Path to end resource

?arg1=val1&arg2=val2 arguments for the request

The following table describes the NERSC system specific resources available through NEWT:

	System Status	Files	Batch Queue	Run Commands	Run Jobs
edison
cori
datatran
archive
pdsf

The NEWT API

This section describes the currently implemented methods in NEWT, and includes worked examples to test existing functionality. It assumes that your web application will be running within the NERSC domain (though this is not required). Examples require a NERSC login for them to function.

If you're not a NERSC user and have questions please email us.

First Steps

Your NEWT web application can start as a simple web page. Internally NEWT uses jQuery to handle some AJAX data movement. We recommend you do to, but the API is pure REST so that's really up to you. Every NEWT application starts with including these lines in the <head> of your web page.

<script src="https://newt.nersc.gov/js/jquery-1.7.2.js"></script>

<script src="https://newt.nersc.gov/js/newt.js"></script>

Authentication

Including the lines above in your web page means that authentication is taken care of automatically. You should see an authentication status bar/form slide down at the top of your web page.

This will transparently add functionality for for logging in, logging out, and checking authentication status into your application. It also defines a javascript variable named newt_base_url which is the base URL for many of the examples below. Everything below assumes that NEWT is installed at https://newt.nersc.gov/newt.

API Reference Resource Inventory

Authentication Status

Method GET [newt_base_url]/login/

Output JSON { "auth": boolean, "username": string, "session_lifetime": seconds, "newt_sessionid": string}

Semantics Get current auth status, username and number of seconds in session

Example: https://newt.nersc.gov/newt/login/

Authentication Login

Method POST [newt_base_url]/login

Input username="my_username"
password="my_password"

Output JSON { "auth": boolean, "username": string, "session_lifetime": seconds, "newt_sessionid": string }
Set-Cookie response header contains session ID

Semantics Posts authentication information. If authentication is successful, the session cookie is set. Returns JSON string with authentication status

Example: https://newt.nersc.gov/newt/login/?next=/newt/

<form method="post" 
action="https://newt.nersc.gov/newt/login/">
<div>
  <div class="form-row">
    <label for="id_username">Username</label>
    <input id="id_username" type="text" name="username" />
  </div>
  <div class="form-row">
    <label for="id_password">Password</label>
    <input type="password" name="password" id="id_password" />
  </div>
</div>

  <div class="submit-row">
    <input type="submit" value="login" />
    <input type="hidden" name="next" value="" />
  </div>  

</form>

Authentication Logout

Method GET [newt_base_url]/logout

Output JSON { "auth": boolean, "username": string, "session_lifetime": seconds}

Semantics Log the user out and returns current authentication status

Example: https://newt.nersc.gov/newt/logout/

Status Get System Status

Method GET [newt_base_url]/status/[machine]

Output JSON { "status": string, "machine": string }

Semantics Get System Availability Status. The returned status attribute will either be "up", "down" or "unknown"

Example: https://newt.nersc.gov/newt/status/cori

Status Get Status for All Systems

Method GET [newt_base_url]/status/

Output JSON Array [{ "status": string, "system": string }, ... ]

Semantics Get System Availability Status. The returned status attribute will either be "up", "down" or "unknown"

Example: https://newt.nersc.gov/newt/status/

Status Get MOTD

Method GET [newt_base_url]/status/motd

Output Message of the day (text)

Semantics Get a plain text summary of the current NERSC status (Message of the Day).

Example: https://newt.nersc.gov/newt/status/motd

Files Directory Listing

Method GET [newt_base_url]/file/[machine]/[path]/

Output JSON Array [{"perms": string, "hardlinks": string, "user": string, "group": string, "size": string, "date": string, "name": string}, ... ]

Semantics Get directory listing if [path] is a directory; get listing for individual file if [path] is a file.

Example: https://newt.nersc.gov/newt/file/cori/tmp/

See also the ajax directory listing demo

Files Download a File

Method GET [newt_base_url]/file/[machine]/[path]?view=read

Output Binary file located on [machine] at [path]

Semantics Return the file to the user. Note the view paramater must be set to “read” otherwise it will return a directory listing

Example: https://newt.nersc.gov/newt/file/cori/etc/motd?view=read

Files Create/Update a File

Method PUT [newt_base_url]/file/[machine]/[path]

Input: file contents

Output JSON string: {status: "OK", location: string}

Semantics Update/create file with given contents. Returns request status and location of file.

Example:

Files Upload a File

Method POST [newt_base_url]/file/[machine]/[path]/

Output 200 response code if successful

Semantics Upload a file on [machine] at the give [path]

Example:

<form action="https://newt.nersc.gov/newt/file/cori/tmp/" method="post" 
    enctype="multipart/form-data">

  <label for="id_file">File:</label>

  <input type="file" name="file" id="id_file" />

  <input type="submit" value="upload" />
</form>

Jobs Run a command

Method POST [newt_base_url]/command/[machine]

Input executable="/path/to/exec options" # full command that needs to be run
Optional parameters:
loginenv=true # setting to true initializes a full bash login env before running

Output JSON { "status": ["OK" | "ERROR"], "output": string, "error": string }

Semantics Run a command (synchronously) on the compute system and get back output. Optionally setting "loginenv=true" will initialize a full bash login env before running (this could result in a small additional delay). Also note that "status" can only capture higher level failures, and not failures within the command. Make sure you check "output" and "error" for the results.

Example:

<form action="https://newt.nersc.gov/newt/command/cori/" method="post">

  <input type="text" name="executable" value="/bin/ls" />

  <input type="submit" value="Run" />
</form>

Jobs View Queue

Method GET [newt_base_url]/queue/[machine]/<sacct>

Input Optional Parameters: (add ?param1=val1&param2=val2 etc. to URL)
index=N # index for first record limit=N # number of records <key>=<value> # restrict records to ones with key=value eg. queue=regular

Output JSON Array: [{"jobid": string, "name": string, "user": string, "timeuse": string, "status": string, "queue": string}, ... ]

Semantics Get the results of a sqs or qstat on a given machine as a JSON array. Note that the default (without the optional /sacct suffix) may not include completed jobs since they are no longer in the queue. Append /sacct (see example) to your URL string for completed jobs (with 24 hours) as well.

Example: https://newt.nersc.gov/newt/queue/cori/

SACCT Example (Completed Job info): https://newt.nersc.gov/newt/queue/cori/sacct

Filter Example: https://newt.nersc.gov/newt/queue/cori/?index=1&limit=3&status=R

Jobs Submit Job to Queue

Method POST [newt_base_url]/queue/[machine]/

Input jobfile=/path/to/script # An existing batch submit file on target host
jobscript="#Script contents" # Only set one of jobscript or jobfile

Output JSON String: { "status": ["OK" | "ERROR"], "error": string, "jobid" : string }

Semantics Submit a script to the batch system (qsub). You can either specify and existing batch script or post a script that will be submitted. It returns a jobid in the batch queue

Example A: Run jobfile on cori:


	<form action="https://newt.nersc.gov/newt/queue/cori/" method="post">

  	  <input size=40 name="jobfile" type="text" />

	  <input type="submit" value="submit" /> 

	</form>

Example B: Submit jobscript to cori:


	<form action="https://newt.nersc.gov/newt/queue/cori/" method="post">

	  <textarea rows=3 cols=40 name="jobscript">

#!/bin/bash -l

#SBATCH -N 1

srun -n 32  /bin/hostname

	  </textarea>

	  <input type="submit" value="submit" /> 

	</form>

Jobs View Job in Queue

Method GET [newt_base_url]/queue/[machine]/[jobid]

Output JSON String: {"jobid": string, "name": string, "user": string, "timeuse": string, "status": string, "queue": string}

Semantics Get the results of a "squeue or qstat jobid" on a given machine as a JSON string. Note that this may not include completed jobs since they are no longer in the queue. Append /sacct (see below) to your string for completed jobs (with 24 hours) as well.

Jobs View Job Account Information (including recently completed jobs)

Method GET [newt_base_url]/queue/[machine]/[jobid]/sacct

Output JSON String: {"jobid": string, "name": string, "user": string, "timeuse": string, "status": string, "queue": string}

Semantics Get the results of a "sacct jobid" on a given machine as a JSON string. This includes recently (within 24 hours) completed jobs

Jobs Delete Job from Queue

Method DELETE [newt_base_url]/queue/[machine]/[jobid]

Output JSON String: {"status": ["OK | ERROR"], "output": string, "error": string}

Semantics Get a job from the batch queue

Account Get Account information from IRIS

Method POST [newt_base_url]/account/iris

Input query: function_name(param: value){field1,field2,...}

Output JSON String: {"data": {JSON}

Semantics Get the information from IRIS for a given resource as a JSON object. We will be updating this section with more details and example API calls, but please contact NERSC support staff if you wish to use NEWT with the new IRIS API

Example $.newt_ajax({ url: "/account/iris", type: "POST", data: {"query": 'user(username: "myuser"){uid, email}'}, success: function(out) { console.log(out.data) } } }) // Outputs: {"newt":{"users":[{"uid":12345,"email":"myuser@acme.com"}]}}

Please refer to NEWT IRIS docs for more detailed information on how to invoke this API

VERB	HTTP verbs - One of {GET, POST, PUT, DELETE}
newt_base_url	base URL e.g. https://newt.nersc.gov/newt
resource_type	One of {auth, account, store, status, file, command, job, queue}
system	One of {edison, cori, pdsf, archive, datatran}
path	Path to end resource
?arg1=val1&arg2=val2	arguments for the request

Method	`GET [newt_base_url]/login/`
Output	`JSON { "auth": boolean, "username": string, "session_lifetime": seconds, "newt_sessionid": string}`
Semantics	Get current auth status, username and number of seconds in session

Method	`POST [newt_base_url]/login`
Input	username="my_username" password="my_password"
Output	`JSON { "auth": boolean, "username": string, "session_lifetime": seconds, "newt_sessionid": string }` `Set-Cookie` response header contains session ID
Semantics	Posts authentication information. If authentication is successful, the session cookie is set. Returns JSON string with authentication status

Method	`GET [newt_base_url]/logout`
Output	`JSON { "auth": boolean, "username": string, "session_lifetime": seconds}`
Semantics	Log the user out and returns current authentication status

Method	`GET [newt_base_url]/status/[machine]`
Output	`JSON { "status": string, "machine": string }`
Semantics	Get System Availability Status. The returned `status` attribute will either be "up", "down" or "unknown"

Method	`GET [newt_base_url]/status/`
Output	`JSON Array [{ "status": string, "system": string }, ... ]`
Semantics	Get System Availability Status. The returned `status` attribute will either be "up", "down" or "unknown"

Method	`GET [newt_base_url]/status/motd`
Output	Message of the day (text)
Semantics	Get a plain text summary of the current NERSC status (Message of the Day).

Method	`GET [newt_base_url]/file/[machine]/[path]/`
Output	`JSON Array [{"perms": string, "hardlinks": string, "user": string, "group": string, "size": string, "date": string, "name": string}, ... ]`
Semantics	Get directory listing if [path] is a directory; get listing for individual file if [path] is a file.

Method	`PUT [newt_base_url]/file/[machine]/[path]`
Input:	`file contents`
Output	`JSON string: {status: "OK", location: string}`
Semantics	Update/create file with given contents. Returns request status and location of file.

Method	`POST [newt_base_url]/file/[machine]/[path]/`
Output	200 response code if successful
Semantics	Upload a file on [machine] at the give [path]

Method	`POST [newt_base_url]/command/[machine]`
Input	`executable="/path/to/exec options" # full command that needs to be run` Optional parameters: `loginenv=true # setting to true initializes a full bash login env before running`
Output	`JSON { "status": ["OK" \| "ERROR"], "output": string, "error": string }`
Semantics	Run a command (synchronously) on the compute system and get back output. Optionally setting "loginenv=true" will initialize a full bash login env before running (this could result in a small additional delay). Also note that "status" can only capture higher level failures, and not failures within the command. Make sure you check "output" and "error" for the results.

Method	`GET [newt_base_url]/queue/[machine]/<sacct>`
Input	Optional Parameters: (add ?param1=val1&param2=val2 etc. to URL) `index=N # index for first record limit=N # number of records <key>=<value> # restrict records to ones with key=value eg. queue=regular`
Output	`JSON Array: [{"jobid": string, "name": string, "user": string, "timeuse": string, "status": string, "queue": string}, ... ]`
Semantics	Get the results of a sqs or qstat on a given machine as a JSON array. Note that the default (without the optional /sacct suffix) may not include completed jobs since they are no longer in the queue. Append /sacct (see example) to your URL string for completed jobs (with 24 hours) as well.

Method	`POST [newt_base_url]/queue/[machine]/`
Input	`jobfile=/path/to/script` # An existing batch submit file on target host `jobscript="#Script contents"` # Only set one of `jobscript` or `jobfile`
Output	`JSON String: { "status": ["OK" \| "ERROR"], "error": string, "jobid" : string }`
Semantics	Submit a script to the batch system (qsub). You can either specify and existing batch script or post a script that will be submitted. It returns a jobid in the batch queue

Method	`GET [newt_base_url]/queue/[machine]/[jobid]`
Output	`JSON String: {"jobid": string, "name": string, "user": string, "timeuse": string, "status": string, "queue": string}`
Semantics	Get the results of a "squeue or qstat jobid" on a given machine as a JSON string. Note that this may not include completed jobs since they are no longer in the queue. Append /sacct (see below) to your string for completed jobs (with 24 hours) as well.

Method	`DELETE [newt_base_url]/queue/[machine]/[jobid]`
Output	`JSON String: {"status": ["OK \| ERROR"], "output": string, "error": string}`
Semantics	Get a job from the batch queue

Method	`POST [newt_base_url]/account/iris`
Input	`query: function_name(param: value){field1,field2,...}`
Output	`JSON String: {"data": {JSON}`
Semantics	Get the information from IRIS for a given resource as a JSON object. We will be updating this section with more details and example API calls, but please contact NERSC support staff if you wish to use NEWT with the new IRIS API
Example	`$.newt_ajax({ url: "/account/iris", type: "POST", data: {"query": 'user(username: "myuser"){uid, email}'}, success: function(out) { console.log(out.data) } } }) // Outputs: {"newt":{"users":[{"uid":12345,"email":"myuser@acme.com"}]}}`