TAPRegExt: A VOResource Schema Extension for Describing TAP Services#

Official bibliographic entry for published version [TAPRegExt1.0].

Status:: TAPRegExt 1.1 WD 2015-11-17

1 Introduction#

The Table Access Protocol TAP TAP allows VO clients to send queries to remote database servers and receive the results in standard formats. In addition, it defines means to discover database schemata on the remote side, to upload data from the local disk or third-party hosts, and more. TAP builds upon a variety of other standards, premier among which is the Universal Worker Service [UWS1.1], which describes how client and server can negotiate the execution of a query and the retrieval of results without having to maintain a continuous connection.

To accommodate a wide variety of requirements, the TAP specification offers implementors many choices on optional features, resource limits, or locally defined functionality. One purpose of TAPRegExt is to allow the service to communicate such choices to remote clients using the mechanisms laid down in the VO Service Interfaces standard VOSI.

Clients also need to discover TAP services offering certain kinds of data. Central to this is the concept of a registry in which resources can be described and consequently discovered by users and applications in the VO. Registries receive resource descriptions as defined in the IVOA standard [VOResource1.1]. In this schema, support for a standard service protocol is described as a service’s capability; the associated metadata is contained within the service resource description’s <capability> element.

TAPRegExt defines this capability element for TAP services. In the context of registering TAP services, an important role filled by TAPRegExt is the communication of supported data models to the Registry.

The specification comes with a non-normative example document [1] showing this specification’s major features. It also declares a separate capability for the VOSI capabilities endpoint. The example is written as a response from a TAP service’s capabilities endpoint. When embedded in a VOResource record, the capability elements would be direct children of the element.

1.1 TAPRegExt within the VO Architecture#

role_diagram.pdf

This specification directly relates to other IVOA standards in the following ways:

VOResource, v1.1 [VOResource1.1]: Descriptions of services that support TAP are encoded using the VOResource XML schema. TAPRegExt is an extension of the VOResource core schema.
TAP, v1.1 TAP: The TAP standard defines some of the concepts that TAPRegExt deals with. The TAP standard document indirectly refers to this document in the specification of its capabilities endpoint.
UWS, v1.1 [UWS1.1]: The UWS standard describes additional parameters the choices of which are communicated using TAPRegExt.
StandardsRegExt StandardsRegExt: TAPRegExt uses the StandardKeyEnumeration mechanism introduced in StandardsRegExt to define controlled vocabularies.

This standard also relates to other IVOA standards:

IVOA Support Interfaces, v1.1 VOSI: describes the standard interfaces to discover metadata about services; this document defines the response TAP services should provide on the capabilities endpoint described by VOSI.
IVOA defined data models: Data models specified by the IVOA can define the structure of database tables holding instances of those data models. The first examples of such definitions are ObsCore ObsCore and RegTAP [RegTAP1.1]. Services providing access to such tables declare that fact within TAPRegExt instance documents.

2 The Extension#

2.1 The Schema Namespace and Location#

The namespace associated with the TAPRegExt VOResource extension is

\[\mbox{\texttt{http://www.ivoa.net/xml/TAPRegExt/v1.0}.}\]

The namespace is unchanged from version 1.0 of this standard as no changes that could break clients are introduced.

Just like the namespace URI for the VOResource schema, the TAPRegExt namespace URI can be interpreted as a URL. Resolving it returns the current XML schema document that defines the TAPRegExt schema (cf. Appendix A Obtaining the Schema).

Authors of VOResource instance documents may choose to provide a location for the VOResource XML schema document and its extensions using the attribute. While generators are free to provide any schema location (e.g., a local mirror), this specification recommends using the TAPRegExt namespace URI as its location URL, as in,

xsi:schemaLocation="http://www.ivoa.net/xml/TAPRegExt/v1.0
                    http://www.ivoa.net/xml/TAPRegExt/v1.0"

Note that you must give the xsi:schemaLocation of the TAPRegExt schema when the capability defined here is part of a published registry resource record as per the IVOA Registry Interface standard [RegistryInterface1.1]. This does not apply to the use in a TAP server’s capabilities endpoint.

2.2 Declaring Instantiated Data Models#

The IVOA defines certain data models that can be instantiated in database tables exposed by a TAP service. This allows a query built exclusively on a data model or a set of data models to work on all TAP services exposing tables instantiating the data model(s).

In TAPRegExt, a data model is identified by its IVOA identifier [IVOAIdentifiers2.0].

2.3 Languages Supported#

TAP services may offer a variety of query languages. In TAPRegExt, the element allows the communication of what languages are available on a service. TAP defines values of the LANG parameter to have either the form <name>-<version> or the form <name>, where the latter form leaves the choice of the version to the server. Therefore, a language is defined using a name and one or more versions.

The recommended way to associate larger amounts of documentation with a language entry in a capability element is via registration of the language using the mechanisms defined in StdRegExt StandardsRegExt and associating the registry record with the language element through the latter’s attribute. The IVOID for the only language mandatory for TAP services, ADQL 2.0, is ivo://ivoa.net/std/ADQL#v2.0. .

The type of the attribute on version is as opposed to since the latter does not allow fragment identifiers in VOResource 1.0. The description constrains the value to be an IVOID (i.e., a URI with a schema of ivo:), though. The same reasoning applies to the attributes of and .

Query languages may support optional features. For ADQL, the most prominent of those are user-defined functions, i.e., functions not defined in the language standard but added by the operators of the service, and geometry functions. Such optional features may be communicated to the service client in elements.

Each such list is labelled with a attribute indicating the type of language option being described. This string should be an IVOID whose semantics in this context, along with the semantics of the content of its descendant elements, can be documented in association with the language in question.

TAPRegExt itself defines the following feature types:

2.4 Output Formats#

A TAP service may offer a variety of output formats. What output formats are available is defined using elements. They declare an RFC 2046 media type [std:MIME] as well as aliases (the shorthand forms the server also accepts in the FORMAT parameter). If desired, the format can be further described with an IVOID in the ivo-id attribute; TAPRegExt provides keys for some variants of VOTables which are not interoperably distinguishable by their MIME types so far:

output-votable-td: A VOTable in which all DATA elements contain a TABLEDATA element
output-votable-binary: A VOTable in which all DATA elements contain a STREAM element with a BINARY child
output-votable-binary2: A VOTable in which all DATA elements contain a STREAM element with a BINARY2 child

2.5 Upload Methods#

TAP services should allow the upload of VOTables. They can support various methods to do this: HTTP upload, retrieval by URL, but also VOSpace or possibly retrieval using Grid protocols. Since an actual specification of the details of such protocols is far beyond the scope of a registry document and probably would not benefit clients anyway, the upload methods are given as IVOIDs.

IVOIDs for the standard upload methods are provided within the resource record ivo://ivoa.net/std/TAPRegExt. The IVOIDs are built by using the keys as fragments after the TAPRegExt IVOID.

It is permitted to register upload methods under authorities other than ivoa.net. The registry records can then provide more in-depth information. For the upload methods defined in the TAP specification, however, the IVOIDs of the keys in the TAPRegExt resource record must be used to enable clients to identify supported methods using string comparisons.

This document defines the following protocol identifiers:

upload-inline – HTTP upload as per section 2.5.2 of the TAP standard TAP.
upload-http – retrieval from an http URL.
upload-https – retrieval from an https URL.
upload-ftp – retrieval from an ftp URL.

Thus, a service offering upload by retrieving from ftp and http URLs would say:

<uploadMethod ivo-id="ivo://ivoa.net/std/TAPRegExt#upload-http"/>
  <uploadMethod ivo-id="ivo://ivoa.net/std/TAPRegExt#upload-ftp"/>

2.6 Resource Limits#

TAP services usually impose certain limits on resource usage by clients, e.g., a maximum run time per query, or a maximum number of rows in the result set. Services assign such limits to newly created jobs and may allow raising the limits by means of queries or query parameters (e.g., the size of the result set is limited by the MAXREC parameter, whereas the date of job destruction may be changed by posting to the destruction parameter). Services may put some limit to how far the resource limitations may be raised.

TAPRegExt’s element allows the declaration of such limits. These declarations are primarily intended for human consumption and should give conservative guidelines. Thus, the operators of a service implementing a complex, possibly dynamic limits policy should give lower estimates here.

If a service supports authentication and has different limits depending on what user is authenticated, it should make an effort to guess the limits applying to a given client (e.g., when authentication tokens are present in the request). Limits reported to the Registry should reflect limits for unauthenticated use unless the service does not admit unauthenticated requests.

The resource limits applying to newly created jobs are given in elements, the limits beyond which users cannot raise the limits are given in elements.

Note that the absence of a specification of limits does not imply that no limits are enforced.

2.6.1 Limits on Time#

This document defines two time-like resource limits:

– the time from job creation until destruction.
– the maximal run time given to a query.

All values in time-like limits are given in seconds. Both and are of type .

2.6.2 Limits on Data#

Limits on data are expressed much like time limits in that they give and a value as well. Both those values have a unit attribute that can either be byte or row for data limits.

This document defines two resource limits on data:

– if is row here, the gives the value of TAP’s MAXREC parameter the service will use when none is specified.
– the maximum size of uploads. This is not a TAP adjustable parameter. The value advises clients about the server’s wishes as to a limit above which some sort of acknowledgement should be requested from the user. The limit gives the maximum size of an upload to the server.

Data limits are defined using the and types:

2.7 Interface Declaration#

Version 1.0 of this specification used a VODataService interface to declare the TAP service’s base URL. This was mainly done to maintain common service discovery patterns – other DAL protocols use -typed interfaces as well –, but it ignored the semantics of the type, which in particular is that the access URL of such an interface is what clients use in queries. Against that, the TAP specification actually leaves open what directly dereferencing the base URL should result in. Instead, all functionality is offered through children of the base URL. The abuse of later had a severe fallout as discussed in Demleitner [note:caproles].

However, existing TAP clients rely on the interface declaration for discovery, and hence all capability elements with a TAPRegExt version 1 must include a -typed interface with

a child element with its content the base URL of the TAP service (the parent of sync, async, etc.) and its attribute set to base.
its attribute set to std
its attribute set to the version of the TAP specification implemented.

In order to open the path to advanced use cases (e.g., involving authenticated access), services implementing TAPRegExt 1.1 must include another interface element of type .

The -typed interface must satisfy the same requirements as the -typed one as regards its child and and attributes. Furthermore, this interface now enumerates the various endpoints exposed below the base URI in elements:raw-latex:todo{We may want to support vocab on endpoint; or do we want an implicit prefix as in datalink?}.

For TAP, at least the sync, async, and tables endpoints must be declared, as well as examples if present. The VOSI capabilities URL, while in TAP also a child of the base URL, continues to be declared in separate capability in this version rather than a DALI interface endpoint. TAPRegExt specifically discourages returning different capability contents from different interfaces (e.g., with and without authentication) of a service.

While in general, not overly much configurability should be enabled on the level of endpoints, operators can attach extra per-endpoint metadata using their children. These support a subset of the attributes defined by RDFa lite [std:RDFaLite11], enough to make statements consisting of a subject (), a predicate (), and an object (content or ):

This is intended to give operators the possibility to communicate extra information to custom clients primarily by defining properties, which are simply URIs controlled by the operator and should dereference to documents explaining the semantics of the properties, ideally authored in HTML. The object of the statement is typically given by the element contents, but where the object cannot be inlined, the ’s attribute can be used instead.

For instance, to convey a custom method to upload tables on an (authenticated) tables endpoint, an operator could say:

<endpoint>
  <name>tables</name>
  <meta property="http://operator.example.edu/supports-our-extension"
    >table-upload</meta>
</endpoint>

To prevent a fragmentation of the TAP standard, such extensions should be restricted to single service providers and custom clients and be considered for addition to the TAP standard as other parties take them up.

Because it models the most essential aspects of RDF, the element, in principle, allows operators to build almost arbitrarily complex metadata structures when other elements are used as subjects. Continuing the example above, the operator could declare media types and labels for upload formats like this:

<endpoint>
  <name>tables</name>
  <meta property="http://operator.example.edu/supports-our-extension"
    id="custom-upload"
    >table-upload</meta>

  <meta about="#custom-upload"
    property="http://operator.example.edu/supports-format"
    id="fits-binary"/>
  <meta about="#fits-binary"
    property="http://operator.example.edu/upload-label"
    >FITS Binary</meta>
  <meta about="#fits-binary"
    property="http://operator.example.edu/upload-media-type"
    >application/fits</meta>

  <meta about="#custom-upload"
    property="http://operator.example.edu/supports-format"
    id="votable"/>
  <meta about="#votable"
    property="http://operator.example.edu/upload-label"
    >VOTable</meta>
  <meta about="#votable"
    property="http://operator.example.edu/upload-media-type"
    >application/x-votable+xml</meta>
</endpoint>

Again, this facility is intended to support prototyping features as well as facilitate provider-specific extensions aiding custom tooling on top of standard interfaces. Interoperable TAP functionality should probably not need to rely on per-endpoint metadata of this complexity.

2.8 The Capability Record#

Using the types defined above, the type can be defined. Note that it is a type, not a (global) element. In instance documents, you will typically place it in a capability element with an explicit type specification, like this:

<capability
    xmlns:tr="http://www.ivoa.net/xml/TAP/v1.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    standardID="ivo://ivoa.net/std/TAP"
    xsi:type="tr:TableAccess">
    ...

-typed capabilites can be used with arbitrary standardIDs; this is different from previous TAPRegExt versions, which fixed it to ivo://ivoa.net/std/TAP. This URI should still be given for TAP 1.0 services. Services having other capabilities (e.g., TAP 1.1 or even UWS) will use other standardIDs.

Appendices#

A Obtaining the Schema#

The recommended way to obtain the TAPRegExt schema is to retrieve the namespace URI, http://www.ivoa.net/xml/TAPRegExt/v1.0, which will usually involve one or two redirects. Despite to suggestive name, this will always return the latest version of the TAPRegExt schema in major version 1; see Harrison et al. [XMLVers1.0] for the background of this somewhat unfortunate circumstance.

The schema accompanying this specific version is published alongside this document [2]. The document at this URI is only intended for use in review or debugging.

B Changes from Previous Versions#

B.1 Changes from REC-1.0#

Adding tr:DALIInterface to the schema.
Dropping the appendix with an example document. Using a standalone, validated file and auxiliaryurl instead.

B.2 Changes from WD-20110127#

userDefinedFunction was generalized to feature within languageFeatures.
The uploadmethods StandardKeyEnumeration was replaced by a resource record for TAPRegExt as a whole. This now includes keys of output formats and features as well; therefore, upload method names in their new IVOIDs are prefixed with upload-
Schema version was bumped to 1.0 (yes, we indulge in unversioned schema changes before this becomes REC).
uploadLimit interpretation was changed: The default limit is now “advisory” and to be interpreted as such by clients, the hard limit is what is actually required by the server.
There’s now an optional ivo-id attribute on the version element within language.
There’s now an optional ivo-id attribute on output formats.

B.3 Changes from WD-20110727#

The namespace in the schema is now http://www.ivoa.net/xml/TAPRegExt/v1.0 consistent with what has already been stated in the text.
The IVOID for ADQL is now ivo://ivoa.net/std/ADQL#v2.0; it is defined here to be in ADQL’s record since we do not want to wait for the ADQL standard to be fixed, but ADQL versioning should really not be done here, so a TAPRegExt IVOID is out of the question.
The IVOID of the TAPRegExt standard is now ivo://ivoa.net/std/TAPRegExt to conform with other standard IVOIDs. Unfortunately, this changes all other IVOIDs dependent on this.
We now allow AnyURI on the ivo-id of language to allow fragment identifiers as, e.g., in ADQL.

B.4 Changes from PR-20120812#

Fixed units in limits to "row" and "byte".

B.5 Changes from REC-1.0#

No longer restricting TableAccess’ standardID to the (old) TAP standardID. TAPRegExt capabilities are now allowed with arbitrary standardIDs
Changed former use of the term “IVORN” to “IVOID” (or Registry part, as appropriate) to comply with a proposed clarification of terminology in Identifiers 2.0
The example now shows an entire capabilities response
Repaired obscore data model URI in the example
dataModel/@ivo-id is now typed xs:anyURI to allow fragment identifiers on it
Migrated to ivoatex