NEWT is a collection of REST based services that allow scientists, programmers, staff, and the public to write web apps for HPC. These are services designed to make HPC computing and data resources highly usable via a web browser. Web browser interaction is often mediated through scripts and web apps that poll, aggregate, and mashup the content from the restful API's URLs. As such the API is not really a set of web pages, but a set of well defined URLs that web apps can invoke which act as information sources and sinks.

We take a pragmatic approach to building HPC-to-web methods, acknowledging a restricted vocabulary of methods gathered from many years of watching how science teams use compute resources and data stores. Starting, stopping, and monitoring batch jobs, WAN data movement, file and data management, and UNIX-like utility functions are the functions NEWT addresses. The API is built around easy-to-use, concise, programmable interfaces. NEWT provides LDAP/PKI authentication mechanisms for read-write management of jobs and data.

What does the API do?

  • Authentication - Log in to the NEWT service with your username and password
  • Accounting Information - Get information about your account through the NERSC Information Management (NIM) system
  • Files - Upload, download files. Browse directories
  • Commands - Run UNIX commands and utilities
  • Jobs - Submit jobs to the batch system
  • Store - Persistent storage layer for remote client applications

API Overview

Transactions against NEWT take the form

For generic resources (account, auth, store):
VERB newt_base_url/resource_type/path?arg1=val1&arg2=val2

OR

For system specific resources (status, file, command, job, queue):
VERB newt_base_url/resource_type/system/path?arg1=val1&arg2=val2

The API is based on a set of actions (verbs) that categorize the different services. Each verb may relate to different actions depending on the METHOD in the HTTP headers. All transactions take place over SSL (https). Most actions require the user to be authenticated first. Data is returned by default in JSON format

VERBHTTP verbs - One of {GET, POST, PUT, DELETE}
newt_base_urlbase URL e.g. https://newt.nersc.gov/newt
resource_typeOne of {auth, account, store, status, file, command, job, queue}
systemOne of {edison, hopper, carver, pdsf, archive, datatran}
pathPath to end resource
?arg1=val1&arg2=val2arguments for the request

The following table describes the NERSC system specific resources available through NEWT:
System StatusFilesBatch QueueRun CommandsRun Jobs
edison
hopper
carver
datatran
archive
pdsf

The NEWT API

This section describes the currently implemented methods in NEWT, and includes worked examples to test existing functionality. It assumes that your web application will be running within the NERSC domain (though this is not required). Examples require a NERSC login for them to function.

If you're not a NERSC user and have questions please email us.

First Steps

Your NEWT web application can start as a simple web page. Internally NEWT uses jQuery to handle some AJAX data movement. We recommend you do to, but the API is pure REST so that's really up to you. Every NEWT application starts with including these lines in the <head> of your web page.

<script src="https://newt.nersc.gov/js/jquery-1.4.2.js"></script>
<script src="https://newt.nersc.gov/js/newt.js"></script>

Authentication

Including the lines above in your web page means that authentication is taken care of automatically. You should see an authentication status bar/form slide down at the top of your web page.

This will transparently add functionality for for logging in, logging out, and checking authentication status into your application. It also defines a javascript variable named newt_base_url which is the base URL for many of the examples below. Everything below assumes that NEWT is installed at https://newt.nersc.gov/newt.

API Reference Resource Inventory

Authentication Status

MethodGET [newt_base_url]/login/
OutputJSON { "auth": boolean, "username": string, "session_lifetime": seconds, "newt_sessionid": string}
SemanticsGet current auth status, username and number of seconds in session

Example: https://newt.nersc.gov/newt/login/

Authentication Login

MethodPOST [newt_base_url]/login
Inputusername="my_username"
password="my_password"
OutputJSON { "auth": boolean, "username": string, "session_lifetime": seconds, "newt_sessionid": string }
Set-Cookie response header contains session ID
SemanticsPosts authentication information. If authentication is successful, the session cookie is set. Returns JSON string with authentication status

Example: https://newt.nersc.gov/newt/login/?next=/newt/

<form method="post" 
action="https://newt.nersc.gov/newt/login/"> <div> <div class="form-row"> <label for="id_username">Username</label> <input id="id_username" type="text" name="username" /> </div> <div class="form-row"> <label for="id_password">Password</label> <input type="password" name="password" id="id_password" /> </div> </div> <div class="submit-row"> <input type="submit" value="login" /> <input type="hidden" name="next" value="" /> </div> </form>

Authentication Logout

MethodGET [newt_base_url]/logout
OutputJSON { "auth": boolean, "username": string, "session_lifetime": seconds}
SemanticsLog the user out and returns current authentication status

Example: https://newt.nersc.gov/newt/logout/

Status Get System Status

MethodGET [newt_base_url]/status/[machine]
OutputJSON { "status": string, "machine": string }
SemanticsGet System Availability Status. The returned status attribute will either be "up", "down" or "unknown"

Example: https://newt.nersc.gov/newt/status/hopper

Status Get Status for All Systems

MethodGET [newt_base_url]/status/
OutputJSON Array [{ "status": string, "system": string }, ... ]
SemanticsGet System Availability Status. The returned status attribute will either be "up", "down" or "unknown"

Example: https://newt.nersc.gov/newt/status/

Status Get MOTD

MethodGET [newt_base_url]/status/motd
OutputMessage of the day (text)
SemanticsGet a plain text summary of the current NERSC status (Message of the Day).

Example: https://newt.nersc.gov/newt/status/motd

Files Directory Listing

MethodGET [newt_base_url]/file/[machine]/[path]/
OutputJSON Array [{"perms": string, "hardlinks": string, "user": string, "group": string, "size": string, "date": string, "name": string}, ... ]
SemanticsGet directory listing if [path] is a directory; get listing for individual file if [path] is a file.

Example: https://newt.nersc.gov/newt/file/carver/tmp/

See also the ajax directory listing demo

Files Download a File

MethodGET [newt_base_url]/file/[machine]/[path]?view=read
OutputBinary file located on [machine] at [path]
SemanticsReturn the file to the user. Note the view paramater must be set to “read” otherwise it will return a directory listing

Example: https://newt.nersc.gov/newt/file/carver/etc/motd?view=read

Files Create/Update a File

MethodPUT [newt_base_url]/file/[machine]/[path]
Input:file contents
OutputJSON string: {status: "OK", location: string}
SemanticsUpdate/create file with given contents. Returns request status and location of file.

Example:

Files Upload a File

MethodPOST [newt_base_url]/file/[machine]/[path]/
Output200 response code if successful
SemanticsUpload a file on [machine] at the give [path]

Example:

<form action="https://newt.nersc.gov/newt/file/carver/tmp/" method="post" enctype="multipart/form-data">
<label for="id_file">File:</label>
<input type="file" name="file" id="id_file" />
<input type="submit" value="upload" /> </form>

Jobs Run a command

MethodPOST [newt_base_url]/command/[machine]
Inputexecutable="/path/to/exec options" # full command that needs to be run
Optional parameters:
loginenv=true # setting to true initializes a full bash login env before running
OutputJSON { "status": ["OK" | "ERROR"], "output": string, "error": string }
SemanticsRun a command (synchronously) on the compute system and get back output. Optionally setting "loginenv=true" will initialize a full bash login env before running (this could result in a small additional delay). Also note that "status" can only capture higher level failures, and not failures within the command. Make sure you check "output" and "error" for the results.

Example:

<form action="https://newt.nersc.gov/newt/command/hopper/" method="post">
<input type="text" name="executable" value="/bin/ls" />
<input type="submit" value="Run" /> </form>

Jobs View Queue

MethodGET [newt_base_url]/queue/[machine]/
Input Optional Parameters: (add ?param1=val1&param2=val2 etc. to URL)
index=N # index for first record
limit=N # number of records
<key>=<value> # restrict records to ones with key=value eg. queue=regular
OutputJSON Array: [{"jobid": string, "name": string, "user": string, "timeuse": string, "status": string, "queue": string}, ... ]
SemanticsGet the results of a qstat on a given machine as a JSON array

Example: https://newt.nersc.gov/newt/queue/carver/

Filter Example: https://newt.nersc.gov/newt/queue/carver/?index=1&limit=3&status=R

See also the ajax view queue demo

Jobs Submit Job to Queue

MethodPOST [newt_base_url]/queue/[machine]/
Input jobfile=/path/to/script # An existing batch submit file on target host
jobscript="#Script contents" # Only set one of jobscript or jobfile
OutputJSON String: { "status": ["OK" | "ERROR"], "error": string, "jobid" : string }
SemanticsSubmit a script to the batch system (qsub). You can either specify and existing batch script or post a script that will be submitted. It returns a jobid in the batch queue

Example A: Run jobfile on Carver:

<form action="https://newt.nersc.gov/newt/queue/carver/" method="post">
<input size=40 name="jobfile" type="text" />
<input type="submit" value="submit" />
</form>


Example B: Submit jobscript to Carver:

<form action="https://newt.nersc.gov/newt/queue/carver/" method="post">
<textarea rows=3 cols=40 name="jobscript">
#PBS -l nodes=1:ppn=8
mpirun -n 8 /bin/hostname
</textarea>
<input type="submit" value="submit" />
</form>


Jobs View Job in Queue

MethodGET [newt_base_url]/queue/[machine]/[jobid]
OutputJSON String: {"jobid": string, "name": string, "user": string, "timeuse": string, "status": string, "queue": string}
SemanticsGet the results of a "qstat jobid" on a given machine as a JSON string

Jobs Delete Job from Queue

MethodDELETE [newt_base_url]/queue/[machine]/[jobid]
OutputJSON String: {"status": ["OK | ERROR"], "output": string, "error": string}
SemanticsGet a job from the batch queue

Liststore

This application is a simple frontend to a shared JSON object storage database. It allows applications to store JSON data in a persistent fashion.

Liststore Get Available Stores

MethodGET [newt_base_url]/store/
OutputJSON ["Store1", "Store2"]
SemanticsGet a list of available stores.

Example: https://newt.nersc.gov/newt/store/

Liststore Create Store

MethodPOST [newt_base_url]/store/[store_name]
InputOptional: JSON {"doc1":["one", "two", "three"], "doc2":{"sam":"snead", "tiger":"woods"}
Set Header 'content-type':'application/json; charset=utf-8'
OutputJSON String: {"status": ["OK | ERROR"], "output": string, "error": string}
SemanticsCreate a store (with optional document), redirect to the newly created URL.

Example:

<form id="dbcreate" action="https://newt.nersc.gov/newt/store/" method='post'>
<input type="submit" value="Create" />
</form>

Liststore Delete Store

MethodDELETE [newt_base_url]/store/[store_name]/
OutputJSON String: {"status": ["OK | ERROR"], "output": string, "error": string}
SemanticsDelete a store.

Liststore Get Store Contents

MethodGET [newt_base_url]/store/[store_name]/
OutputJSON Array [ ]
SemanticsGet all documents in a store.

Example: https://newt.nersc.gov/newt/store/dbtest/

Liststore Query Store

MethodGET [newt_base_url]/store/[store_name]/?query={"key":"val"}
OutputJSON Array [ ]
SemanticsGet results of a query, using mongo query syntax.

Liststore Get Store Permissions

MethodGET [newt_base_url]/store/[store_name]/admin
OutputJSON { "username": string, "_id": string, "writers": [], "collection": string, "readers": [] }
SemanticsGet permissions for a store.

Example: https://newt.nersc.gov/newt/store/dbtest/admin

Liststore Update Store Permissions

MethodPUT [newt_base_url]/store/[store_name]/admin
InputJSON {"readers": ["group1", "group2"], "writers": ["group1"]}
Output
SemanticsUpdate list of readers and writers that can access store. Readers and writers can be any valid UNIX group

ListstoreCreate Document

MethodPOST [newt_base_url]/store/[store_name]
InputJSON {"doc1":["one", "two", "three"], "doc2":{"sam":"snead", "tiger":"woods"}
Set Header 'content-type':'application/json; charset=utf-8'
OutputJSON String: {"_id": string}
SemanticsCreate a document, return document ID.

Liststore Get Document Contents

MethodGET [newt_base_url]/store/[store_name]/[document_id]
OutputJSON {"sam":"snead", "tiger":"woods"}
SemanticsGet contents of a document.

Liststore Update Document

MethodPUT [newt_base_url]/store/[store_name]/[document_id]
InputJSON {"sam":"snead", "tiger":"woods"}
OutputJSON String: {"status": ["OK | ERROR"], "output": string, "error": string}
SemanticsUpdate contents of a document, for the given fields.

Account Get Account information from NIM

MethodGET [newt_base_url]/account/[path to NIM resource]
OutputJSON String
SemanticsGet the information from NIM for a given resource as a JSON array. For more information on how to construct a resource path refer to the Account section

Example: https://newt.nersc.gov/newt/account/user/shreyas

Account

The NIM (NERSC Information Management system) database provides accounting information for all NERSC users and resources. The design of the NIM REST API allows for a combination of verb, service, attribute and qualified path to determine the URL that can be used to query any object in the database. The object and qualified path must exist else it will return a parsing error

This document lists the various NIM account objects and how they can be accessed through NEWT. This is a read-only interface to NIM and is only used with the GET verb to access objects.

Objects

  • user[s]
  • repo[s]
  • group[s]
  • host[s]
  • person[s]
  • project[s]
  • organization[s]
  • program[s]
  • scicat[s]
  • class[es]
  • image[s]
  • item[s]
  • state
  • running
  • client
  • ldap

Attributes

  • id
  • allowed
  • restricted
  • photo

Example NIM Queries

The URL https://newt.nersc.gov/newt/account should be prepended to the service and paths below. Output is delivered in JSON format.

VerbPathDescriptionLinkNotes
GET/items
List all service_query entries
https://newt.nersc.gov/newt/account/items
GET/hosts
List all NIM hosts
https://newt.nersc.gov/newt/account/hosts
GET/host/id/46
Show a host via host_id
https://newt.nersc.gov/newt/account/host/id/46
GET/host/hopper
Show a host via hostname
https://newt.nersc.gov/newt/account/host/hopper
case independent
GET/users
List all NIM users
https://newt.nersc.gov/newt/account/users
GET/user/id/12625
Show a user via user_id or uid https://newt.nersc.gov/newt/account/user/id/12625
GET/user/rkowen
Show a user via uname
https://newt.nersc.gov/newt/account/user/rkowen
case dependent
GET/groups
List all NIM groups
https://newt.nersc.gov/newt/account/groups
GET/group/id/105Show a group via gid https://newt.nersc.gov/newt/account/group/id/105
GET/group/mpccc
Show a group via gname
https://newt.nersc.gov/newt/account/group/mpccccase dependent
GET/repos
List all NIM repos
https://newt.nersc.gov/newt/account/repos
GET/repo/id/105
Show repos associated to a repo via rid
https://newt.nersc.gov/newt/account/repo/id/105
GET/repo/mpccc
Show repos associated to a repo via rname https://newt.nersc.gov/newt/account/repo/mpccc
case dependent
GET/usage/repos
Show all the repos and usage allocations https://newt.nersc.gov/newt/account/usage/repos
very large
GET/usage/repo/mpccc
Show usage associated with a repo https://newt.nersc.gov/newt/account/usage/repo/mpccc
case dependent
GET/usage/repo/mpccc/users
Show the repo and user allocation and usage https://newt.nersc.gov/newt/account/usage/repo/mpccc/users
case dependent
GET/usage/user/rkowen
Show allocation and usage for the given user https://newt.nersc.gov/newt/account/usage/user/rkowen
case dependent
GET/classesList all the Classes https://newt.nersc.gov/newt/account/classes
GET/class/id/23511Show a class via class_id https://newt.nersc.gov/newt/account/class/id/23511
GET/class/alccShow a class via classname https://newt.nersc.gov/newt/account/class/alcccase independent
GET/imagesList all the images https://newt.nersc.gov/newt/account/images
GET/image/id/6Show image via image_id https://newt.nersc.gov/newt/account/image/id/6
GET/image/lbl1.gifShow image via title/filename https://newt.nersc.gov/newt/account/image/lbl1.gif case independent
GET/image/id/6/photo
Return image via image_id
https://newt.nersc.gov/newt/account/image/id/6/photo
GET/image/lbl1.gif/photo
Return image via title/filename
https://newt.nersc.gov/newt/account/image/lbl1.gif/photo
GET/officesList all the offices https://newt.nersc.gov/newt/account/offices
GET/office/id/10015Show an office via office_id https://newt.nersc.gov/newt/account/office/id/10015
GET/office/CCCShow an office via offname https://newt.nersc.gov/newt/account/office/ccccase independent
GET/organizations
List all the organizations
https://newt.nersc.gov/newt/account/organizations
GET/organization/id/599
Show an organization via org_id
https://newt.nersc.gov/newt/account/organization/id/599
GET/organization/nersc
Show an organization via label
https://newt.nersc.gov/newt/account/organization/nersccase independent
GET/organization/id/599/photoReturn an image via org_id
https://newt.nersc.gov/newt/account/organization/id/599/photo
GET/organization/nersc/photoReturn an image via label
https://newt.nersc.gov/newt/account/organization/nersc/photo
GET/persons
List all persons https://newt.nersc.gov/newt/account/persons
GET/person/id/6410
Show a person via person_id
https://newt.nersc.gov/newt/account/person/id/6410
GET/person/r.k. owen
Show a person via fullname
https://newt.nersc.gov/newt/account/person/r.k. owencase independent
GET/programsList all programs
https://newt.nersc.gov/newt/account/programs
GET/program/id/12882Show a program via program_id
https://newt.nersc.gov/newt/account/program/id/12882
GET/program/ccc
Show a program via progname
https://newt.nersc.gov/newt/account/program/ccccase independent
GET/projects
List all projects
https://newt.nersc.gov/newt/account/projects
GET/project/id/14941
Show a project via project_id
https://newt.nersc.gov/newt/account/project/id/14941
GET/project/testmpp
Show a project via projname
https://newt.nersc.gov/newt/account/project/testmppcase dependent
GET/project/id/14941/photoReturn an image via project_id
https://newt.nersc.gov/newt/account/project/id/14941/photo
GET/project/testmpp/photoReturn an image via projname
https://newt.nersc.gov/newt/account/project/testmpp/photo
GET/scicats
List all science categories
https://newt.nersc.gov/newt/account/scicats
GET/scicat/id/13487
Show a science category via scicat_id
https://newt.nersc.gov/newt/account/scicat/id/13487
GET/scicat/other
Show a science category via scicatname
https://newt.nersc.gov/newt/account/scicat/othercase independent
GET/hosts/users
List all hosts and their users
https://newt.nersc.gov/newt/account/hosts/users
very large
GET/host/id/46/usersShow all users for a given host
https://newt.nersc.gov/newt/account/host/id/46/users
GET/host/hopper/users
Show all users for a given host
https://newt.nersc.gov/newt/account/host/hopper/users
GET/host/hopper/user/rkowen
Show specific host and user info https://newt.nersc.gov/newt/account/host/hopper/user/rkowen
and /id permutations
GET/users/hosts
List all users and the hosts they belong to https://newt.nersc.gov/newt/account/users/hosts
very large
GET/user/id/12625/hosts
Show the hosts for a given user
https://newt.nersc.gov/newt/account/user/id/12625/hosts
GET/user/rkowen/hostsShow the hosts for a given user
https://newt.nersc.gov/newt/account/user/rkowen/hosts
GET/user/rkowen/host/hopperShow specific user and host info
https://newt.nersc.gov/newt/account/user/rkowen/host/hopper
and /id permutations
GET/users/groups
List all users and their groups
https://newt.nersc.gov/newt/account/users/groups
fairly large
GET/user/id/12625/groups
Show a user's groups via user_id https://newt.nersc.gov/newt/account/user/id/12625/groups
GET/user/rkowen/groups
Show a user's groups via uname https://newt.nersc.gov/newt/account/user/rkowen/groups
GET/user/rkowen/persons
Show the user's person info
https://newt.nersc.gov/newt/account/user/rkowen/persons
GET/persons/usersList all persons and sub-user info https://newt.nersc.gov/newt/account/persons/users
GET/person/id/9345/users
Show a person and the sub-user info
https://newt.nersc.gov/newt/account/person/id/9345/users
GET/groups/users
List all groups and their users
https://newt.nersc.gov/newt/account/groups/usersfairly large
GET/group/id/105/users
Show a group's users via gid
https://newt.nersc.gov/newt/account/group/id/105/users
GET/group/mpccc/users
Show a group's users via gname
https://newt.nersc.gov/newt/account/group/mpccc/users
GET/offices/programsList all offices and sub-programs
https://newt.nersc.gov/newt/account/offices/programs
GET/office/ber/programs
Show an office and sub-programs
https://newt.nersc.gov/newt/account/office/ber/programs
GET/organizations/projects
List all organizations and sub-projects
https://newt.nersc.gov/newt/account/organizations/projects
GET/organization/nersc/projectsShow an organization and sub-projects
https://newt.nersc.gov/newt/account/organization/nersc/projects
GET/scicats/projects
List all science categories and sub-projects
https://newt.nersc.gov/newt/account/scicats/projects
GET/scicat/other/projects
Show a science category and sub-projects
https://newt.nersc.gov/newt/account/scicat/other/projects
GET/programs/projects
List all programs and sub-projects
https://newt.nersc.gov/newt/account/programs/projects
GET/program/ccc/projects
Show a program and sub-projects
https://newt.nersc.gov/newt/account/program/ccc/projects
GET/classes/projectsList all classes and sub-projects
https://newt.nersc.gov/newt/account/classes/projects
GET/class/alcc/projects
Show a class and sub-projects
https://newt.nersc.gov/newt/account/class/alcc/projects

Note that the "mixed" queries near the bottom of the table for a specific item with sub-items the item can be specified by name or id the same way as those queries that return a single item near the top of the table.