Programmatic Submission

Introduction

Submissions of different data objects can be made programmatically using XML or JSON file formats.

A receipt XML/JSON with accession numbers is provided upon successful submission. Any validation or other errors will be written in the receipt as well.

Production and test services

There are two programmatic Webin submission services. One for test submissions and another for production submissions:

The test service is recreated from the full content of the production service every day at 03.00 GMT/BST. Therefore, any submissions made to the test service will be removed by the following day.

When you are using the test service the receipt XML will contain the following message:

This submission is a TEST submission and will be discarded within 24 hours.

It is advisable to first test your submissions using the Webin test service before establising an automated submission pipeline.

Upload data files

Data files must be uploaded into a submitter specific private Webin file upload area before they can be submitted. Always keep a local copy of the uploaded files until the files have been successfully submitted and archived. The Webin file upload area is a temporary transit area which is not backed up and subject to a fair use policy.

Once uploaded, data files are submitted by referring to them from Run JSON/XML or Analysis JSON/XML. Run JSON/XML is used for used for sequence read submissions while Analysis JSON/XML is used for other types of submissions (e.g. genome assemblies).

More information of uploading data files and our the fair use policy is available here.

Submission protocol

Submissions are made through the secure HTTPS protocol using POST multipart/form-data according to RFC1867.

Authentication method

Webin username and password must be provided using basic HTTP authentication.

When using curl the username and password are provided using the -u option:

curl -u username:password

Webin REST V2 API

Programmatic submissions can be made using the Webin Rest V2 service. This has been developed to address issues such as timeout errors during submissions. The Webin REST V1 submission endpoint may time out while the submission still is being processed. If this happens the submitter will not receive a submission receipt.The Webin REST version 2 (V2) prevents this error and has a number of other improvements.

The Webin REST V2 API supports two types of submissions:

  • synchronous submissions that use an endpoint that returns a submission receipt

  • asynchronous submissions that use an endpoint that returns a submission accession that is used to poll for the submission receipt

The synchronous submission endpoint timeouts gracefully so that the submission processing is terminated if the submission processing takes more than 1 minute. The asynchronous submission endpoint supports larger submissions and gracefully terminated the submission if the processing takes more than 10 minutes.

These endpoints are further explained below:

Service URL: https://www.ebi.ac.uk/ena/submit/webin-v2/

API Mapping

Use

/submit

Processes submissions and returns the submission receipt. This method supports
XML or JSON documents of up to 15 MB. If the submission processing takes more than 1 minute
then the submission processing is gracefully terminated with a timeout error.

/submit/queue

Queues submissions for processing and returns a submission ID and a submission polling
link. The submission processing status and the receipt are retrieved for the submission ID
using the poll link. If the submission processing takes more than 10 minutes
then the submission processing is gracefully terminated with a timeout error.

/submit/poll

Returns the submission processing status and receipt (if available) for a queued
submission.

How to use Webin REST V2

In contrast to Webin REST V1 where individual objects (submission, runs, samples, etc.) are submitted as separate files using multipart file upload, in Webin REST V2 they are submitted as a single file in the request body. Additionally, Webin REST V2 supports both JSON and XML formats while Webin REST V1 only supports the XML format. Please refer to the Webin Rest V1 documentation to find out more.

Webin REST V2 accepts submissions to the production and test servers.

The web API can be viewed and tested using the Swagger web interface accessible through the above links. Some users use cURL while others build their own API to access these endpoints.

Submission Format

The type of each submitted JSON/XML file must be specified at time of submission.

When using curl, each JSON/XML file is submitted using the -T option:

-T "file=@FILENAME"

where the file referenced can contain a combination of the datatypes documented below:

The JSON or XML file can be submitted in the request body as a single document. For example, a PROJECT and SAMPLE object can be submitted like this:

 {
    "submission": {
      "alias": "submissionAliasName",
      "accession": "",
      "actions": [
          {
                "type": "ADD"
          },
          {
                "type": "HOLD",
                "holdUntilDate": "2025-01-12"
            }
        ],
        "attributes": [
            {
                "tag": "test_tag",
                "value": "test_val"
            },
            {
                "tag": "test_tag_1",
                "value": "test_val_1"
            }
        ]
    },
    "projects": [
        {
            "alias": "comparative-analysis",
            "name": "Human Gut Microbiota Study",
            "title": "Exploration of the diversity human gastric microbiota",
            "description": "The genome sequences of gut microbes were obtained using...",
            "sequencingProject": {},
            "attributes": [
                {
                    "tag": "testTag",
                    "value": "testValue"
                }
            ],
            "project_links": [
                {
                    "xrefLink": {
                        "db": "PUBMED",
                        "id": "25035323"
                    }
                }
            ]
        }
    ],
    "samples": [
        {
            "alias": "stomach_microbiota",
            "title": "human gastric microbiota, mucosal",
            "organism": {
                "taxonId": "1284369"
            },
            "attributes": [
                {
                    "tag": "investigation type",
                    "value": "mimarks-survey"
                },
                {
                    "tag": "sequencing method",
                    "value": "pyrosequencing"
                },
                {
                    "tag": "collection date",
                    "value": "2010-01-20"
                },
                {
                    "tag": "host body site",
                    "value": "Mucosa of stomach"
                },
                {
                    "tag": "human-associated environmental package",
                    "value": "human-associated"
                },
                {
                    "tag": "geographic location (latitude)",
                    "value": "1.81",
                    "unit": "DD"
                },
                {
                    "tag": "geographic location (longitude)",
                    "value": "-78.76",
                    "unit": "DD"
                },
                {
                    "tag": "geographic location (country and/or sea)",
                    "value": "Colombia"
                },
                {
                    "tag": "geographic location (region and locality)",
                    "value": "Tumaco"
                },
                {
                    "tag": "environment (biome)",
                    "value": "coast"
                },
                {
                "tag": "environment (feature)",
                "value": "human-associated habitat"
                },
                {
                    "tag": "project name",
                    "value": "Human microbiota"
                },
                {
                    "tag": "environment (material)",
                    "value": "gastric biopsy"
                },
                {
                    "tag": "ena-checklist",
                    "value": "ERC000014"
                }
            ]
        }
    ]
}

Release date for data objects

When the HOLD action is provided with a date then all submitted studies, projects and samples will become immediately public on that date. Run, experiment and analysis objects will become public on the date when studies or projects they refer to are made public.

A release date can be provided for studies by using the HOLD action together with the ADD action:

 "submission":{
 "alias":"submissionAliasName",
 "accession":"",
 "actions":[
    {
        "type":"ADD"
    },
    {
        "type":"HOLD",
        "holdUntilDate":"TODO: release date"
    }
],
"attributes":[
    {
        "tag":"test",
        "value":"test_val"
    },
    {
        "tag":"test_1",
        "value":"test_val_1"
    }
  ]
 }

The HoldUntilDate specifies the public release date of any studies submitted within the submission. This can be at most two years in the future.

If the HOLD action is used without a date then the submitted studies, projects and samples will become public in two years time.

You can also update the release date of a study, project or sample by using the HOLD action with a new release date:

 "submission":{
 "alias":"submissionAliasName",
 "accession":"",
 "actions":[
    {
        "type":"HOLD",
        "target":"TODO: study accession number",
        "holdUntilDate":"TODO: YYYY-MM-DD"
    }
],
}

This applies only to non-public studies. It is not possible to suppress a public study by this method.

The new release date must be not more than two years in the future.

Identifying objects

Each submitted object is uniquely identified within a submission account using the alias attribute. Once an object has been submitted no other object of the same type can use the same alias within the submission account. Objects can refer to other objects within a submission account by either alias or the assigned accession number.

Identifying submitters

The center_name attribute defines the submitting institution. The center name is automatically assigned from submission account details except for broker accounts. Brokers should provide a center name which reflects the institute where the data was generated. For brokers, the broker_name field is automatically assigned at time of submission.

Synchronous Endpoint

The synchronous submission endpoint processes the submitted JSON/XML document right away and returns a receipt in either JSON/XML document. This route is appropriate for small-scale submissions. Submitters with large or high-volume submissions are advised to use the asynchronous endpoint explained below.

Below is an example of a submission to the Webin service using the synchronous endpoint:

curl -u username:password -X POST 'https://wwwdev.ebi.ac.uk/ena/dev/submit/webin-v2/submit' \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-T 'submission file name'

The JSON/XML submission document file name should be included in the field -T ‘submision file name’. The Accept and Content-Type headers must match the document format in the ‘submission file name’.

Asynchronous Endpoint

When a submission is made using the asynchronous endpoint, it enters a pending state in a queue of submissions. This submission is processed once it reaches the front of this queue. The asynchronous submission endpoint supports larger and a higher volume of submissions than the synchronous endpoint.

Below is an example of a sequence read data submission to the Webin service using the asynchronous endpoint:

curl -u username:password -X POST ''https://wwwdev.ebi.ac.uk/ena/dev/submit/webin-v2/submit/queue' \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-T 'submission file name'

When the asynchronous endpoint is used, a JSON document is returned with a submission ID and a poll link:

{
"submissionId": "ERA12944374",
"submissionAccountId": "Webin-xxxx",
"_links": {
    "poll": {
    "href": "https://wwwdev.ebi.ac.uk/ena/dev/submit/webin-v2/submit/poll/ERA12944374"
    }
 }
}

The poll endpoint and submission ID are used to retrieve the state of the submission and the JSON/XML receipt once the submission has been processed.

An example of the cURL command used for the poll endpoint to retrieve a receipt is shown below:

curl -u username:password "https://www.ebi.ac.uk/ena/submit/webin-v2/submit/poll/ERA12944374"

or

curl -u username:password "https://www.ebi.ac.uk/ena/submit/webin-v2/submit/poll/ERA12944374" \
-H 'Accept: application/json'

Submission Receipt

Once a submission has been processed a receipt is returned either immediately (synchronous endpoint) or after polling (asynchronous endpoint). The success attribute in the receipt is true if the submission was successful and false if the submission was not successful. The receipt also contains the accession numbers of the objects that you have submitted. An example of a successful sequence read data submission together with a project, sample and experiment object:

{
  "success" : true,
  "receiptDate" : "2023-11-16T10:44:41.618Z",
  "experiments" : [ {
    "alias" : "illumina-hiSeq",
    "accession" : "ERX9223136",
    "status" : "PRIVATE"
  } ],
  "runs" : [ {
    "alias" : "paired-data",
    "accession" : "ERR9669128",
    "status" : "PRIVATE"
  } ],
  "success" : true,
  "receiptDate" : "2023-11-16T10:20:02.982Z",
  "samples" : [ {
    "alias" : "stomach_microbiota",
    "accession" : "ERS27605861",
    "status" : "PRIVATE",
    "holdUntilDate" : "2023-01-01Z",
    "externalAccession" : {
      "id" : "SAMEA130793922",
      "db" : "biosample"
    }
  } ],
  "projects" : [ {
    "alias" : "comparative-analysis",
    "accession" : "PRJEB101337",
    "status" : "PRIVATE",
    "holdUntilDate" : "2023-01-01Z",
    "externalAccession" : {
      "id" : "ERP201886",
      "db" : "study"
    }
  } ],
  "submission" : {
    "alias" : "SUBMISSION-27-07-2022-09:54:36:278",
    "accession" : "ERA12956757"
  },
  "messages" : {
    "info" : [ "All objects in this submission are set to private status (HOLD)." ]
  },
  "actions" : [ "ADD", "HOLD" ]
}

If the submission is not successful the success attribute in the RECEIPT element is false and the error messages are available in the MESSAGES block:

{
"success" : false,
"receiptDate" : "2022-01-01T17:05:01.114+01:00",
"messages" : {
"error": [ "Error message displayed here" ]
 }
}