NEWT is a collection of REST based services that allow scientists, programmers, staff, and the public to write web apps for HPC. These are services designed to make HPC computing and data resources highly usable via a web browser. Web browser interaction is often mediated through scripts and web apps that poll, aggregate, and mashup the content from the restful API's URLs. As such the API is not really a set of web pages, but a set of well defined URLs that web apps can invoke which act as information sources and sinks.

We take a pragmatic approach to building HPC-to-web methods, acknowledging a restricted vocabulary of methods gathered from many years of watching how science teams use compute resources and data stores. Starting, stopping, and monitoring batch jobs, WAN data movement, file and data management, and UNIX-like utility functions are the functions NEWT addresses. The API is built around easy-to-use, concise, programmable interfaces. NEWT provides LDAP/PKI authentication mechanisms for read-write management of jobs and data.

What does the API do?

  • Authentication - Log in to the NEWT service with your username and password
  • Accounting Information - Get information about your account through the NERSC Information Management (NIM) system
  • Files - Upload, download files. Browse directories
  • Commands - Run UNIX commands and utilities
  • Jobs - Submit jobs to the batch system
  • Store - Persistent storage layer for remote client applications

API Overview

Transactions against NEWT take the form

For generic resources (account, auth, store):
VERB newt_base_url/resource_type/path?arg1=val1&arg2=val2


For system specific resources (status, file, command, job, queue):
VERB newt_base_url/resource_type/system/path?arg1=val1&arg2=val2

The API is based on a set of actions (verbs) that categorize the different services. Each verb may relate to different actions depending on the METHOD in the HTTP headers. All transactions take place over SSL (https). Most actions require the user to be authenticated first. Data is returned by default in JSON format

newt_base_urlbase URL e.g.
resource_typeOne of {auth, account, store, status, file, command, job, queue}
systemOne of {edison, hopper, carver, pdsf, archive, datatran}
pathPath to end resource
?arg1=val1&arg2=val2arguments for the request

The following table describes the NERSC system specific resources available through NEWT:
System StatusFilesBatch QueueRun CommandsRun Jobs


This section describes the currently implemented methods in NEWT, and includes worked examples to test existing functionality. It assumes that your web application will be running within the NERSC domain (though this is not required). Examples require a NERSC login for them to function.

If you're not a NERSC user and have questions please email us.

First Steps

Your NEWT web application can start as a simple web page. Internally NEWT uses jQuery to handle some AJAX data movement. We recommend you do to, but the API is pure REST so that's really up to you. Every NEWT application starts with including these lines in the <head> of your web page.

<script src=""></script>
<script src=""></script>


Including the lines above in your web page means that authentication is taken care of automatically. You should see an authentication status bar/form slide down at the top of your web page.

This will transparently add functionality for for logging in, logging out, and checking authentication status into your application. It also defines a javascript variable named newt_base_url which is the base URL for many of the examples below. Everything below assumes that NEWT is installed at

API Reference Resource Inventory

Authentication Status

MethodGET [newt_base_url]/login/
OutputJSON { "auth": boolean, "username": string, "session_lifetime": seconds, "newt_sessionid": string}
SemanticsGet current auth status, username and number of seconds in session


Authentication Login

MethodPOST [newt_base_url]/login
OutputJSON { "auth": boolean, "username": string, "session_lifetime": seconds, "newt_sessionid": string }
Set-Cookie response header contains session ID
SemanticsPosts authentication information. If authentication is successful, the session cookie is set. Returns JSON string with authentication status


<form method="post" 
action=""> <div> <div class="form-row"> <label for="id_username">Username</label> <input id="id_username" type="text" name="username" /> </div> <div class="form-row"> <label for="id_password">Password</label> <input type="password" name="password" id="id_password" /> </div> </div> <div class="submit-row"> <input type="submit" value="login" /> <input type="hidden" name="next" value="" /> </div> </form>

Authentication Logout

MethodGET [newt_base_url]/logout
OutputJSON { "auth": boolean, "username": string, "session_lifetime": seconds}
SemanticsLog the user out and returns current authentication status


Status Get System Status

MethodGET [newt_base_url]/status/[machine]
OutputJSON { "status": string, "machine": string }
SemanticsGet System Availability Status. The returned status attribute will either be "up", "down" or "unknown"


Status Get Status for All Systems

MethodGET [newt_base_url]/status/
OutputJSON Array [{ "status": string, "system": string }, ... ]
SemanticsGet System Availability Status. The returned status attribute will either be "up", "down" or "unknown"


Status Get MOTD

MethodGET [newt_base_url]/status/motd
OutputMessage of the day (text)
SemanticsGet a plain text summary of the current NERSC status (Message of the Day).


Files Directory Listing

MethodGET [newt_base_url]/file/[machine]/[path]/
OutputJSON Array [{"perms": string, "hardlinks": string, "user": string, "group": string, "size": string, "date": string, "name": string}, ... ]
SemanticsGet directory listing if [path] is a directory; get listing for individual file if [path] is a file.


See also the ajax directory listing demo

Files Download a File

MethodGET [newt_base_url]/file/[machine]/[path]?view=read
OutputBinary file located on [machine] at [path]
SemanticsReturn the file to the user. Note the view paramater must be set to “read” otherwise it will return a directory listing


Files Create/Update a File

MethodPUT [newt_base_url]/file/[machine]/[path]
Input:file contents
OutputJSON string: {status: "OK", location: string}
SemanticsUpdate/create file with given contents. Returns request status and location of file.


Files Upload a File

MethodPOST [newt_base_url]/file/[machine]/[path]/
Output200 response code if successful
SemanticsUpload a file on [machine] at the give [path]


<form action="" method="post" enctype="multipart/form-data">
<label for="id_file">File:</label>
<input type="file" name="file" id="id_file" />
<input type="submit" value="upload" /> </form>

Jobs Run a command

MethodPOST [newt_base_url]/command/[machine]
Inputexecutable="/path/to/exec options" # full command that needs to be run
Optional parameters:
loginenv=true # setting to true initializes a full bash login env before running
OutputJSON { "status": ["OK" | "ERROR"], "output": string, "error": string }
SemanticsRun a command (synchronously) on the compute system and get back output. Optionally setting "loginenv=true" will initialize a full bash login env before running (this could result in a small additional delay). Also note that "status" can only capture higher level failures, and not failures within the command. Make sure you check "output" and "error" for the results.


<form action="" method="post">
<input type="text" name="executable" value="/bin/ls" />
<input type="submit" value="Run" /> </form>

Jobs View Queue

MethodGET [newt_base_url]/queue/[machine]/
Input Optional Parameters: (add ?param1=val1&param2=val2 etc. to URL)
index=N # index for first record
limit=N # number of records
<key>=<value> # restrict records to ones with key=value eg. queue=regular
OutputJSON Array: [{"jobid": string, "name": string, "user": string, "timeuse": string, "status": string, "queue": string}, ... ]
SemanticsGet the results of a qstat on a given machine as a JSON array


Filter Example:

See also the ajax view queue demo

Jobs Submit Job to Queue

MethodPOST [newt_base_url]/queue/[machine]/
Input jobfile=/path/to/script # An existing batch submit file on target host
jobscript="#Script contents" # Only set one of jobscript or jobfile
OutputJSON String: { "status": ["OK" | "ERROR"], "error": string, "jobid" : string }
SemanticsSubmit a script to the batch system (qsub). You can either specify and existing batch script or post a script that will be submitted. It returns a jobid in the batch queue

Example A: Run jobfile on Carver:

<form action="" method="post">
<input size=40 name="jobfile" type="text" />
<input type="submit" value="submit" />

Example B: Submit jobscript to Carver:

<form action="" method="post">
<textarea rows=3 cols=40 name="jobscript">
#PBS -l nodes=1:ppn=8
mpirun -n 8 /bin/hostname
<input type="submit" value="submit" />

Jobs View Job in Queue

MethodGET [newt_base_url]/queue/[machine]/[jobid]
OutputJSON String: {"jobid": string, "name": string, "user": string, "timeuse": string, "status": string, "queue": string}
SemanticsGet the results of a "qstat jobid" on a given machine as a JSON string

Jobs Delete Job from Queue

MethodDELETE [newt_base_url]/queue/[machine]/[jobid]
OutputJSON String: {"status": ["OK | ERROR"], "output": string, "error": string}
SemanticsGet a job from the batch queue


This application is a simple frontend to a shared JSON object storage database. It allows applications to store JSON data in a persistent fashion.

Liststore Get Available Stores

MethodGET [newt_base_url]/store/
OutputJSON ["Store1", "Store2"]
SemanticsGet a list of available stores.


Liststore Create Store

MethodPOST [newt_base_url]/store/[store_name]
InputOptional: JSON {"doc1":["one", "two", "three"], "doc2":{"sam":"snead", "tiger":"woods"}
Set Header 'content-type':'application/json; charset=utf-8'
OutputJSON String: {"status": ["OK | ERROR"], "output": string, "error": string}
SemanticsCreate a store (with optional document), redirect to the newly created URL.


<form id="dbcreate" action="" method='post'>
<input type="submit" value="Create" />

Liststore Delete Store

MethodDELETE [newt_base_url]/store/[store_name]/
OutputJSON String: {"status": ["OK | ERROR"], "output": string, "error": string}
SemanticsDelete a store.

Liststore Get Store Contents

MethodGET [newt_base_url]/store/[store_name]/
OutputJSON Array [ ]
SemanticsGet all documents in a store.


Liststore Query Store

MethodGET [newt_base_url]/store/[store_name]/?query={"key":"val"}
OutputJSON Array [ ]
SemanticsGet results of a query, using mongo query syntax.

Liststore Get Store Permissions

MethodGET [newt_base_url]/store/[store_name]/admin
OutputJSON { "username": string, "_id": string, "writers": [], "collection": string, "readers": [] }
SemanticsGet permissions for a store.


Liststore Update Store Permissions

MethodPUT [newt_base_url]/store/[store_name]/admin
InputJSON {"readers": ["group1", "group2"], "writers": ["group1"]}
SemanticsUpdate list of readers and writers that can access store. Readers and writers can be any valid UNIX group

ListstoreCreate Document

MethodPOST [newt_base_url]/store/[store_name]
InputJSON {"doc1":["one", "two", "three"], "doc2":{"sam":"snead", "tiger":"woods"}
Set Header 'content-type':'application/json; charset=utf-8'
OutputJSON String: {"_id": string}
SemanticsCreate a document, return document ID.

Liststore Get Document Contents

MethodGET [newt_base_url]/store/[store_name]/[document_id]
OutputJSON {"sam":"snead", "tiger":"woods"}
SemanticsGet contents of a document.

Liststore Update Document

MethodPUT [newt_base_url]/store/[store_name]/[document_id]
InputJSON {"sam":"snead", "tiger":"woods"}
OutputJSON String: {"status": ["OK | ERROR"], "output": string, "error": string}
SemanticsUpdate contents of a document, for the given fields.

Account Get Account information from NIM

MethodGET [newt_base_url]/account/[path to NIM resource]
OutputJSON String
SemanticsGet the information from NIM for a given resource as a JSON array. For more information on how to construct a resource path refer to the Account section



The NIM (NERSC Information Management system) database provides accounting information for all NERSC users and resources. The design of the NIM REST API allows for a combination of verb, service, attribute and qualified path to determine the URL that can be used to query any object in the database. The object and qualified path must exist else it will return a parsing error

This document lists the various NIM account objects and how they can be accessed through NEWT. This is a read-only interface to NIM and is only used with the GET verb to access objects.


  • user[s]
  • repo[s]
  • group[s]
  • host[s]
  • person[s]
  • project[s]
  • organization[s]
  • program[s]
  • scicat[s]
  • class[es]
  • image[s]
  • item[s]
  • state
  • running
  • client
  • ldap


  • id
  • allowed
  • restricted
  • photo

Example NIM Queries

The URL should be prepended to the service and paths below. Output is delivered in JSON format.

List all service_query entries
List all NIM hosts
Show a host via host_id
Show a host via hostname
case independent
List all NIM users
Show a user via user_id or uid
Show a user via uname
case dependent
List all NIM groups
GET/group/id/105Show a group via gid
Show a group via gname dependent
List all NIM repos
Show repos associated to a repo via rid
Show repos associated to a repo via rname
case dependent
Show all the repos and usage allocations
very large
Show usage associated with a repo
case dependent
Show the repo and user allocation and usage
case dependent
Show allocation and usage for the given user
case dependent
GET/classesList all the Classes
GET/class/id/23511Show a class via class_id
GET/class/alccShow a class via classname independent
GET/imagesList all the images
GET/image/id/6Show image via image_id
GET/image/lbl1.gifShow image via title/filename case independent
Return image via image_id
Return image via title/filename
GET/officesList all the offices
GET/office/id/10015Show an office via office_id
GET/office/CCCShow an office via offname independent
List all the organizations
Show an organization via org_id
Show an organization via label independent
GET/organization/id/599/photoReturn an image via org_id
GET/organization/nersc/photoReturn an image via label
List all persons
Show a person via person_id
GET/person/r.k. owen
Show a person via fullname owencase independent
GET/programsList all programs
GET/program/id/12882Show a program via program_id
Show a program via progname independent
List all projects
Show a project via project_id
Show a project via projname dependent
GET/project/id/14941/photoReturn an image via project_id
GET/project/testmpp/photoReturn an image via projname
List all science categories
Show a science category via scicat_id
Show a science category via scicatname independent
List all hosts and their users
very large
GET/host/id/46/usersShow all users for a given host
Show all users for a given host
Show specific host and user info
and /id permutations
List all users and the hosts they belong to
very large
Show the hosts for a given user
GET/user/rkowen/hostsShow the hosts for a given user
GET/user/rkowen/host/hopperShow specific user and host info
and /id permutations
List all users and their groups
fairly large
Show a user's groups via user_id
Show a user's groups via uname
Show the user's person info
GET/persons/usersList all persons and sub-user info
Show a person and the sub-user info
List all groups and their users large
Show a group's users via gid
Show a group's users via gname
GET/offices/programsList all offices and sub-programs
Show an office and sub-programs
List all organizations and sub-projects
GET/organization/nersc/projectsShow an organization and sub-projects
List all science categories and sub-projects
Show a science category and sub-projects
List all programs and sub-projects
Show a program and sub-projects
GET/classes/projectsList all classes and sub-projects
Show a class and sub-projects

Note that the "mixed" queries near the bottom of the table for a specific item with sub-items the item can be specified by name or id the same way as those queries that return a single item near the top of the table.