Changes in 4.0 ⬇
-
Use the arrows to browse significant changes since the 1.0 version of this
specification.
-
Sections with significant changes are marked Δ in the table of contents. New
functions introduced in this version are marked ➕ in the table of
contents.
The purpose of this document is to define functions to manipulate binary data for
inclusion in XPath 4.0, XQuery 4.0, and XSLT 4.0. The binary data is represented by
the type xs:base64Binary as defined by
3.2.16 base64BinaryXS2.
The syntax used to call these functions and operators is specified in [XML Path Language (XPath) 4.0] and [XQuery 4.0: An XML Query Language].
This document defines several classes of functions:
-
Functions to create constant binary values and convert between the binary forms
and sequences of octets.
-
Functions to perform basic operations on binary values, such as joining,
selecting and searching.
-
Functions to perform bitwise operations.
-
Functions to convert between binary data and numeric values.
-
Functions to decode or encode strings.
References to specific sections of some other specifications are indicated by
cross-document links in this document. Each such link consists of a pointer to a
specific section followed a superscript specifying the linked document. The
superscripts have the following meanings: FILE40 [EXPath File 4.0],
FO40 [XQuery and XPath Functions and Operators 4.0] and XS2 [XML Schema Part 2: Datatypes Second Edition]
Error conditions are identified by a code (a QName). When such an
error condition is reached in the evaluation of an expression, a dynamic error
is thrown, with the corresponding error code (as if the standard XPath function
error() had been called).
In this specification these codes use the
http://expath.org/ns/binary namespace with a local part
in the form of a descriptive string, for example
bin:index-out-of-range. This convention is used in place of
the http://www.w3.org/2005/xqt-errors namespace and alpha-numeric local
part, e.g. err:FOCH0004 used in [XQuery and XPath Functions and Operators 4.0]. These error codes have been largely retained from the 1.0 version of the
EXPath specification.
Error codes are summarized in B Error summary.
Changes in 4.0 ⬇ ⬆
-
Binary arguments to the functions are now declared to be either
xs:hexBinary or xs:base64Binary, but binary
function results remain of type xs:base64Binary. This should
not cause any backward incompatibilities as casting back and forth between
the two representations has been possible since at least version
2.0
XML Schema, and therefore the XDM data model, defines two primitive atomic
types holding binary data:
3.2.16 base64BinaryXS2 and
3.2.15 hexBinaryXS2.
In both cases the value space is a sequence of octets; the two types differ only
in how the binary value is converted to or from a string. Although most functions
in this specification are not concerned with the string representation of the value,
they need to distinguish these two types, and xs:base64Binary
has been chosen arbitrarily as the result type for functions that return
binary values. Where binary data is supplied as input to a function, however,
both xs:base64binary and xs:hexBinary are accepted.
[Definition] The term binary value
is used to mean a sequence of octets represented as an instance of
xs:base64Binary or xs:hexBinary.
Conversion to and from xs:hexBinary can be performed by casting with
xs:hexBinary() and xs:base64Binary().
Note:
Internally it is likely that implementations will use the same byte
array representation for both data types, and that casting
will merely involve a change in the type annotation; it should not
require data to be copied.
[Definition] A binary value
that contains no octets is referred to as being zero-length.
A zero-length binary value is an item, and as such is a sequence of length
one, which is not the same thing as an empty sequence.
A suite of test-cases for all the functions defined in this module, in [QT3] format, is defined at [Test-suite].
The functions defined in this document are contained in the namespace
http://expath.org/ns/binary. The conventional
prefix for this namespace is bin.
This document uses the prefix bin to refer to this namespace.
User-written applications can choose a different prefix to refer to the
namespace, so long as it is bound to the correct URI.
Each function (or group of functions having the same name) is defined in this
specification using a standard proforma, full details of which can be found in
[XQuery and XPath Functions and Operators 4.0] section 1.5 Function signatures and descriptions. In particular in this version
(trailing) optional arguments for functions (introduced in XPath 4.0) are used
where appropriate in the signatures, rather than multiple-arity signatures as
previously.
All the functions defined in this specification are deterministic and independent
of the static and dynamic context.
All the functions described in this specification can in principle be defined
in terms of two primitives: bin:to-octets and
bin:from-octets, which convert between a binary value
and a sequence of xs:unsignedByte values (integers in the range 0 to 255).
In many cases functions are specified with reference to a formal equivalent
that defines the semantics of the function directly or indirectly in terms of these
two primitives.
Development of this specification was driven by requirements which some XML
developers regularly encounter in examining or generating data which is presented in
binary, or other non-textual forms. Some typical use cases include:
-
Getting the dimensions of an image file.
-
Extracting image metadata.
-
Processing images embedded as base64 encodings within a SOAP message.
-
Processing legacy text files which use different encodings in separate
sections.
-
Generating PDF files from SVG graphical data.
As an example, the following code reads the binary form of a JPEG image file,
searches for the 'Start of Frame/DCT' segment, and unpacks the relevant binary
sections to integer-valued attributes height and width:
<xsl:variable name="binary"
select="file:read-binary(@href)"
as="xs:base64Binary"/>
<xsl:variable name="location"
select="bin:find($binary, 0, bin:hex('FFC0'))"/>
<size width="{bin:unpack-unsigned-integer($binary,
$location+5,
2,
'most-significant-first')}"
height="{bin:unpack-unsigned-integer($binary,
$location+7,
2,
'most-significant-first')}"/>
The result is an element such as:
<size width="377" height="327"/>
Note:
The 'most-significant-first' argument could
be omitted, because it is the default. This is the numeric format used in JPEG:
for further information see 7.1.1 Octet order.
Changes in 4.0 ⬇ ⬆
-
The functions in this example have been moved into a differing namespace
prefix (asn:) to avoid suggesting that they are part of the
supported function set.
[ASN.1], used in a number of telecommunications industry standards,
defines a binary data syntax (the basic encoding rules)
for identifying and encoding
arbitrary data as streams of octets. Many of these
forms specify the length of data as part of their encoding. For example,
an integer has a variable-length representation
as a sequence of octets:
-
The first octet has the value 0x02 to indicate
that this is an integer.
-
Next, a variable-length length field to indicate
the length of the payload. (This is designed to accommodate very
large integers, such as those found in cryptography.)
-
Finally, the payload itself: the octets of the integer value in
most-significant-first order.
To generate such a representation for an integer from XSLT/XPath, the
following code might be used:
<xsl:function name="asn:int-octets" as="xs:integer*">
<xsl:param name="value" as="xs:integer"/>
<xsl:sequence select="if ($value ne 0)
then (bin:int-octets($value idiv 256), $value mod 256)
else ()"/>
</xsl:function>
<xsl:function name="asn:encode-ASN-integer" as="xs:base64Binary">
<xsl:param name="int" as="xs:integer"/>
<xsl:variable name="octets" select="bin:int-octets($int)"/>
<xsl:variable name="length-octets"
select="let $l := count($octets)
return ( if ($l le 127)
then $l
else ( let $lo := bin:int-octets($l)
return (128+count($lo), $lo)
)
)"/>
<xsl:sequence select="bin:from-octets((2,$length-octets,$octets))"/>
</xsl:function>
The function asn:int-octets returns a sequence of all the
significant octets of the integer (i.e. eliminating leading zeroes) in
most-significant order. Examples of the encoding are:
asn:encode-ASN-integer(0) → "AgA="
asn:encode-ASN-integer(1234) → "AgIE0g=="
asn:encode-ASN-integer(123456789123456789123456789123456789) → "Ag8XxuPAMviQRa10ZoQEXxU="
asn:encode-ASN-integer(123456789.. 900 digits... 123456789) → "AoIBdgaTo....EBF8V"
The first example requires no octets to encode zero, hence its octets are
2, 0. Both the second and third examples can be represented in
less than 128 octets (2 and 15 respectively), so length is encoded as a
single octet. The first three octets of the result for the last example,
which encodes a 900-digit integer, are: 2, 130, 1 indicating that
the data is represented by (130-128) * 256 + 1 = 513 octets and the length
required two octets to encode.
Decoding is a matter of compound use of the integer decoding function:
<xsl:function name="asn:decode-ASN-integer" as="xs:integer">
<xsl:param name="in" as="xs:base64Binary"/>
<xsl:sequence
select="let $lo := bin:unpack-unsigned-integer($in, 1, 1, 'BE')
return (
if ($lo le 127)
then bin:unpack-unsigned-integer($in, 2, $lo, 'BE')
else ( let $lo2 := $lo - 128,
$lo3 := bin:unpack-unsigned-integer($in, 2 ,$lo2, 'BE')
return bin:unpack-unsigned-integer($in, 2+$lo2, $lo3, 'BE')
)
)"
/>
</xsl:function>
Note:
All numbers in ASN are 'big-endian'.
This function is the inverse of the encoding function above:
asn:decode-ASN-integer(xs:base64Binary("AgA=")) → 0
asn:decode-ASN-integer(xs:base64Binary("AgIE0g==")) → 1234
asn:decode-ASN-integer(xs:base64Binary("Ag8XxuPAMviQRa10ZoQEXxU="))
→ 123456789123456789123456789123456789
asn:decode-ASN-integer(xs:base64Binary("AoIBdgaTo....EBF8V"))
→ 123456789.. 900 digits... 123456789
This section provides a summary of operations on binary values beyond those
defined in this document.
Casting is defined:
-
Between xs:hexBinary and xs:base64Binary,
in either direction. Since the two types have the same value space,
conversion is lossless, and can be expected in most implementations
to have very low cost.
-
From xs:string to xs:hexBinary
or xs:base64Binary. The semantics correspond to the
XSD rules for validation of simple types. The two binary
types have different lexical representation, so conversion from
strings requires different input.
-
From xs:hexBinary
or xs:base64Binary to xs:string.
The rules are defined in [TITLE OF XP40 SPEC, TITLE OF casting-to-string SECTION]XP40.
The two binary types have different lexical representation, so the output
in the two cases is different.
Comparison of binary values is defined using any of the value comparison
operators eq, ne, le, lt,
ge, or gt. It is therefore defined also for the
general comparison operators =, !=, <=, <,
>=, and >.
The semantics are defined in op:binary-equal
and op:binary-less-than.
The two types xs:hexBinary and xs:base64Binary
can be compared interchangeably.
Because binary values are ordered, sequences of binary values can be sorted
using the function fn:sort, or by using the xsl:sort
instruction in XSLT or the order by clause in XQuery.
Integers can be converted to a string representation using any radix
in the range 2 to 36 using the function fn:format-integer.
Conversely, strings representing an integer using any radix in the range
2 to 36 can be converted to an integer using the function fn:parse-integer.
These functions do not directly involve binary values, but they can be used to
convert between numbers and binary values by using a string representation as an intermediate
format.
See also the functions bin:pack-integer,
bin:unpack-integer, and bin:unpack-unsigned-integer
defined in this specification.
This module defines no specific functions for reading and writing binary data from
external resources, but other specifications provide some suitable mechanisms.
[XQuery and XPath Functions and Operators 4.0] provides a function to retrieve binary
resources:
fn:binary-resource( |
$filesource |
as xs:string? |
|
) as xs:base64Binary |
The EXPath File Module [EXPath File 4.0] provides three functions
suitable for use in file-based situations:
file:read-binary( |
$file |
as xs:string |
|
$offset |
as xs:integer? |
:= 0, |
$size |
as xs:integer? |
:= () |
) as xs:base64Binary |
which reads binary data from an existing file, with an optional offset and size.
file:write-binary( |
$file |
as xs:string |
|
$value |
as xs:base64Binary |
|
) as empty-sequence() |
which writes binary data into a new or existing file.
file:append-binary( |
$file |
as xs:string |
|
$value |
as xs:base64Binary |
|
) as empty-sequence() |
which appends binary data onto the end of an existing file.