1 Introduction
Changes in 4.0 ⬇
-
Use the arrows to browse significant changes since the 1.0 version of this
specification.
-
Sections with significant changes are marked Δ in the table of contents. New
functions introduced in this version are marked ➕ in the table of
contents.
The purpose of this document is to define functions to manipulate binary data for
inclusion in XPath 4.0, XQuery 4.0, and XSLT 4.0. The binary data is represented by
the type xs:base64Binary as defined by Section
3.2.16 base64BinaryXS2.
The exact syntax used to call these functions and operators is specified in [XML Path Language (XPath) 4.0], [XQuery 4.1: An XML Query Language] and [XSL Transformations (XSLT) Version 4.0].
This document defines several classes of functions:
-
Functions to create binary 'constants' and convert between the binary forms
and sequences of octets.
-
Functions to perform basic operations on binary values, such as joining,
selecting and searching.
-
Functions to perform bitwise operations.
-
Functions to pack or unpack numeric values from within or into binary
data.
-
Functions to decode or encode strings from within or into binary data>.
References to specific sections of some other specifications are indicated by
cross-document links in this document. Each such link consists of a pointer to a
specific section followed a superscript specifying the linked document. The
superscripts have the following meanings: FILE40 [EXPath File 4.0],
FO40 [XQuery and XPath Functions and Operators 4.0] and XS2 [XML Schema Part 2: Datatypes Second Edition]
1.1 Error management
Error conditions are identified by a code (a QName.) When such an
error condition is reached in the evaluation of an expression, a dynamic error
is thrown, with the corresponding error code (as if the standard XPath function
error() had been called.)
In this specification these codes use the
http://expath.org/ns/binary namespace and a 'descriptive string'
local part, e.g. bin:index-out-of-range, rather than the
http://www.w3.org/2005/xqt-errors namespace and alpha-numeric local
part, e.g. err:FOCH0004, used in [XQuery and XPath Functions and Operators 4.0]. These error codes were chosen originally in the 1.0 version of 2013.
1.2 Binary type
Changes in 4.0 ⬇ ⬆
-
Binary 'data' arguments to the functions are now declared to be either
xs:hexBinary or xs:base64Binary, but binary
function results remain of type xs:base64Binary. This should
not cause any backward incompatibilities as casting back and forth between
the two representations has been possible since at least version
2.0
The principal binary type within this module is xs:base64Binary as
defined by Section
3.2.16 base64BinaryXS2.
In general, for the functions defined in this specification, if the result return
is binary data, it will always be of type
xs:base64Binary. Any arguments to a function which are considered
to be binary data can be either of type xs:base64Binary or
xs:hexBinary.
Conversion to and from xs:hexBinary can be performed by casting with
xs:hexBinary() and xs:base64Binary().
Note:
As these types are normally implemented as wrappers around byte array
structures containing the data, and differ only when being serialized to or
parsed from text, such casting in-process should not involve data
copying.
An item of type xs:base64Binary can be empty, i.e.
contain no data, (in the same way that items of type xs:string can
contain no characters.) Where 'data' arguments to functions that return binary
data are optional (i.e. $arg as type?) and any of
those optional arguments is set to the empty sequence, in general an empty
sequence is returned, rather than an empty item of type
xs:base64Binary.
1.3 Test suite
A suite of test-cases for all the functions defined in this module, in [QT3] format, is defined at [Test-suite].
1.4 Conformance
This specification follows the general remarks on and terminology for conformance
given in Section 1.2 ConformanceFO40
In this document, text labeled as an example or as a note is provided for
explanatory purposes and is not normative.
1.5 Namespaces and prefixes
The functions defined in this document are contained exclusively in the namespace
http://expath.org/ns/binary and referenced using a
xs:QName binding to that namespace.
This document uses the prefix bin to refer to this namespace.
User-written applications can choose a different prefix to refer to the
namespace, so long as it is bound to the correct URI. In accordance with current
practice, it is recommended that the prefix bin be reserved for use
with this namespace.
1.6 Function signatures and descriptions
Each function (or group of functions having the same name) is defined in this
specification using a standard proforma, full details of which can be found in
Section 1.5 Function signatures and descriptionsFO40. In particular in this update
(trailing) optional arguments for functions (introduced in XPath 4.0) are used
where appropriate in the signatures, rather than multiple-arity signatures as
previously.
2 Use cases
Development of this specification was driven by requirements which some XML
developers regularly encounter in examining or generating data which is presented in
binary, or other non-textual forms. Some typical use cases include:
-
Getting the dimensions of an image file.
-
Extracting image metadata.
-
Processing images embedded as base64 encodings within a SOAP message.
-
Processing legacy text files which use different encodings in separate
sections.
-
Generating PDF files from SVG graphical data.
2.1 Example – finding JPEG size
As an example, the following code reads the binary form of a JPEG image file,
searches for the 'Start of Frame/DCT' segment, and unpacks the relevant binary
sections to integers of height and width:
<xsl:variable name="binary" select="file:read-binary(@href)" as="xs:base64Binary"/>
<xsl:variable name="location" select="bin:find($binary,0,bin:hex('FFC0'))"/>
<size width="{bin:unpack-unsigned-integer($binary,$location+5,2,'most-significant-first')}"
height="{bin:unpack-unsigned-integer($binary,$location+7,2,'most-significant-first')}"/>
→ <size width="377" height="327"/>
(The 'most-significant-first' argument ensures the numeric
conversion is 'big-endian', which is the format in JPEG.)
2.2 Example – reading and writing variable length ASN.1 integers
Changes in 4.0 ⬇ ⬆
-
The functions in this example have been moved into a differing namespace
prefix (asn:) to avoid suggesting that they are part of the
supported function set.
[ASN.1] defines several formats for identifying and encoding
arbitrary-sized telecommunications data as streams of octets. Many of these
forms specify the length of data as part of their encoding. For example, in the
Basic Encoding Rules, an integer is represented as the following series of
octets:
-
Type – 1 octet – in this case the value 0x02
-
Length – ≥ 1 octet – the number of octets in the integer value. The
length field itself can be variable in length – to accommodate VERY
large integers (requiring more than 127 octets to represent, e.g.
2048-bit crypto keys.)
-
Payload – ≥ 0 octets – the octets of the integer value in
most-significant-first order.
To generate such a representation for an integer from XSLT/XPath, the
following code might be used:
<xsl:function name="asn:int-octets" as="xs:integer*">
<xsl:param name="value" as="xs:integer"/>
<xsl:sequence
select="if($value ne 0) then (bin:int-octets($value idiv 256),$value mod 256) else ()"/>
</xsl:function>
<xsl:function name="asn:encode-ASN-integer" as="xs:base64Binary">
<xsl:param name="int" as="xs:integer"/>
<xsl:variable name="octets" select="bin:int-octets($int)"/>
<xsl:variable name="length-octets"
select="let $l := count($octets) return
(if($l le 127) then $l
else (let $lo := bin:int-octets($l) return (128+count($lo),$lo)))"/>
<xsl:sequence select="bin:from-octets((2,$length-octets,$octets))"/>
</xsl:function>
The function asn:int-octets returns a sequence of all the
'significant' octets of the integer (i.e. eliminating leading 'zeroes') in
most-significant order. Examples of the encoding are:
asn:encode-ASN-integer(0) → "AgA="
asn:encode-ASN-integer(1234) → "AgIE0g=="
asn:encode-ASN-integer(123456789123456789123456789123456789) → "Ag8XxuPAMviQRa10ZoQEXxU="
asn:encode-ASN-integer(123456789.. 900 digits... 123456789) → "AoIBdgaTo....EBF8V"
The first example requires no octets to encode zero, hence its octets are
2,0. Both the second and third examples can be represented in
less than 128 octets (2 and 15 respectively), so length is encoded as a
single octet. The first three octets of the result for the last example,
which encodes a 900-digit integer, are: 2,130,1 indicating that
the data is represented by (130-128) * 256 + 1 = 513 octets and the length
required two octets to encode.
Decoding is a matter of compound use of the integer decoding function:
<xsl:function name="asn:decode-ASN-integer" as="xs:integer">
<xsl:param name="in" as="xs:base64Binary"/>
<xsl:sequence
select="let $lo := bin:unpack-unsigned-integer($in,1,1,'BE') return (
if($lo le 127) then bin:unpack-unsigned-integer($in,2,$lo,'BE')
else (let $lo2 := $lo - 128, $lo3 := bin:unpack-unsigned-integer($in,2,$lo2,'BE') return
bin:unpack-unsigned-integer($in,2+$lo2,$lo3,'BE')))"
/>
</xsl:function>
(all numbers in ASN are 'big-endian') and the examples from above
reverse:
asn:decode-ASN-integer(xs:base64Binary("AgA=")) → 0
asn:decode-ASN-integer(xs:base64Binary("AgIE0g==")) → 1234
asn:encode-ASN-integer(xs:base64Binary("Ag8XxuPAMviQRa10ZoQEXxU="))
→ 123456789123456789123456789123456789
asn:encode-ASN-integer(xs:base64Binary("AoIBdgaTo....EBF8V"))
→ 123456789.. 900 digits... 123456789
3 Loading and saving binary data
This module defines no specific functions for reading and writing binary data from
resources, but other specifications provide some suitable mechanisms.
[XQuery and XPath Functions and Operators 4.0] provides a function to retrieve binary
resources:
fn:fn:binary-resource( |
$filesource |
as xs:string? |
|
) as xs:base64Binary |
The EXPath File Module [EXPath File 4.0] provides three functions
suitable for use in file-based situations:
fn:file:read-binary( |
$file |
as xs:string |
|
$offset |
as xs:integer? |
:= 0, |
$size |
as xs:integer? |
:= () |
) as xs:base64Binary |
which reads binary data from an existing file, with an optional offset and size.
fn:file:write-binary( |
$file |
as xs:string |
|
$value |
as xs:base64Binary |
|
) as empty-sequence() |
which writes binary data into a new or existing file.
fn:file:append-binary( |
$file |
as xs:string |
|
$value |
as xs:base64Binary |
|
) as empty-sequence() |
which appends binary data onto the end of an extant file.