Fix #1667

Various small copy edits.

Also adds summary descriptions to record definitions in the function catalog, which requires a schema and stylesheet change.

We have changed the data model (§2.8.4) (see PR #546) so that implementations may allow characters that are not valid XML characters.

We have not explored the impact of this change on parse-json(), which is one of the obvious places where non-XML characters may arise. For example, JSON allows unescaped C1 control characters.

(Note however that the data model explicitly bans unpaired surrogates, and I think that rule should apply to parse-json() even though the JSON grammar allows it).

Hopefully it only requires clarification notes to be added to the spec, and not any substantive change.

Brings the spec of fn:function-annotations into line with the test cases and examples

Fix #1649

Drop fn:node-kind from the function catalog so it disappears from the function finder

Correct one example of fn:type-of and add some more examples.

Fix #1650

We decided to close #374 without further action. This PR just cleans up the relevant schema file by removing the DTD and stylesheet related comment(s).

The fn:sort function supports composite sort keys (where the sort key is a sequence, it's treated as a sequence of sort keys in major to minor order).

We could add the same feature for xsl:sort, perhaps driven by the attribute composite="yes" for compatibility with grouping keys.

In #747, a syntax for QName literals was proposed (Q"prefix:local"). Concerns were raised that the new syntax could be mixed up with the existing Q{URI}local syntax, and that too many syntax variants are confusing.

In attribute and element constructors, it is already possible to supply prefix:local-name and Q{uri}local strings for names:

element { 'xml:name' } {},
element { 'Q{uri}name' } {}

With option declarations, it is possible to write:

declare option output:cdata-section-elements 'xml';
<xml>text</xml>

…and it is counterintuitive that serialize($xml, { 'cdata-section-elements': 'xml' }) is not legal.

I wonder whether we cannot simply allow both QNames and strings for arguments (and arguments in options) in the existing standard functions. There are fewer cases than I had suspected:

Function | Argument --- | --- fn:error | $code fn:format-number | $options: format-name (can also be xs:NCName) fn:deep-equal | $options: unordered-elements fn:serialize | $options: cdata-section-elements, suppress-indentation fn:function-lookup | $name fn:load-xquery-module | $options: variables, vendor-options fn:transform | $options: initial-function, initial-mode, initial-template, ... fn:elements-to-maps | $options: layouts fn:schema-type | $name

For some options, like the method option of fn:serialize, we already allow both strings and QNames.

Several good suggestions for fn:path made at today's review:

(a) For namespaces, an option to identify elements by the result of the name() function - that is, using the actual prefix of each element, rather than a prefix obtained from an externally supplied map (b) The ability to get a path to a node from a supplied ancestor rather than from the root. (Defaulting perhaps to a path from the context node, though that interacts awkwardly with the default for the target node itself.) (c) Some more advice and guidance, especially relating to the different use cases depending on whether the path is for use by software or for (diagnostic?) use by human developers.

The specification of the of the option parameter conventions contains this:

In cases where an option is list-valued, by convention the function should accept either a sequence or an array: but this rule applies only if the specification of the option explicitly accepts either. Accepting a sequence is convenient if the value is generated programmatically using an XPath expression; while accepting an array allows the options to be held in an external file in JSON format, to be read using a call on the fn:json-doc function.

In particular, it says

...this rule applies only if the specification of the option explicitly accepts either...

However I could not find any option that explicitly makes use of "accepts either" in the specification. In the tests, I found two cases where arrays are passed as option values: numberformat-510 (here the option isn't even list-valued), and serialize-xml-106a.

I am wondering whether the above paragraph might be superfluous. The text preceding it says that option values are coerced to the required type, and that implies converting an array to its member sequence, doesn't it?

The current rules say:

If empty($EE/(* | text()) (that is, if there are no child elements or text nodes) then: […] empty If empty($EE/text()[normalize-space()]) (that is, there are no text node children other than whitespace), then: […] list

Maybe it would be consistent to add [normalize-space()] to the condition of the empty layout?

Fix #1624

by adding a note explaining the problem.

If maps are updated, insertion/deletion order may be an issue, even more if maps will be ordered by default (#1651).

This topic needs to be discussed in more depth before we take any actions.

Personally, I think we should focus on XML updates first (related: #1225).

Now that we are discussing different types of maps, there could also be JSON maps. Then each map would have a property json, that can be false or true. If it is false, it is an ordinary map like now. If it is true, it is a "JSON map".

A JSON map can only have string keys, and all map functions would enforce that constraint by casting the key to string.

Parse-json and json-doc would return a JSON map. As would the bare brace {} constructor for compatibility with Javascript.

For example

let $json := parse-json('{"1": 234}')
return map:put($json, 1, 456)

would return a JSON map {"1": 456}.

let $json := parse-json('{"1": 234}')
return map:contains($json, 1)

would return true.

Currently maps and arrays have very little type safety. You can say that your function expects array(xs:string), but that involves testing what the array actually contains, and there's nothing to stop you then appending an integer to the array.

I would like to explore the possibility of having arrays and maps annotated with a type (either always, or optionally), and for this type to constrain operations such as array:append() and map:put().

@dnovatchev has suggested that ordered maps and unordered maps should be different types, and I think it would be difficult to do that unless we move to structural typing. It's also more consistent with typing of atomic values and nodes - though it raises a question about sequences, where the type is purely descriptive.

This would also have implications for records: presumably a map could be annotated with a record type, and this too could constrain the operations available on the value.

This is a rather big change and I put it forward fairly tentatively, but I'm interested to hear people's views.

Replace <code> with <function> tags where appropriate.

Fix #1652

PR #1616 instroduced improved support for the function and xfunction tags.

We should now change the documents to take advantage of this.

Currently, XDM maps are “unordered”: An implementation is allowed to organize entries in a way that optimizes lookup, not order. The entries do not have a predictable order unless they are explicitly sorted.

There are cases in which it is helpful if the “insertion order” is preserved – i.e., the order in which new map entries are added to a map. While the insertion order is not relevant if a map is exclusively used for lookups, it may be beneficial if the input includes deliberately sorted key/value pairs, such as (often) in JSON data, configurations or key/value sequences.

I created this issue because there was some confusion in #564, and on Slack, about this map flavor and “sorted maps”, which are discussed in issue #564: Sorted maps hold all map entries sorted by the key, using a comparator or (in its basic variant) fn:data#1.

PR #1609 attempts to solve both requirements at once.

• fn:node-kind is still listed in the function index
• fn:type-of: type-of($e//doc/child::node()) → type-of($e/child::node())

The function signature and the prose rules say

The result is a sequence of maps, each being an instance of record(key as xs:QName, value as xs:anyAtomicType*)

But one of the examples returns a singleton map in which the QName is the key and the value is the associated value. The test cases also follow that pattern.

Copied from https://github.com/qt4cg/qtspecs/issues/1592#issuecomment-2493270899:

With regard to types, I would propose to introduce a separate option:

elements-to-maps(
  <value>42</value>,
  { 'types': { 'value': 'number' } }
)

→ { "value": 42 }

I have a preference for strings, as we can prefix them with @. Next, the representation could be identical to the result, which I believe is more intuitive:

elements-to-maps(
  <value count='3'/>
  { 'types': { '@count': 'number' } }
)

→ { "value": { "@count": 3 } }

Of course, we could also have two options (element-types, attribute-types).

If a user chooses a custom layout, it should always be applied, or (if inappropriate, by all means) an error message should be raised.

The rationale: I think that the current fallback behavior is flawed. Explicit settings should never be overridden by implicit choices.

Copied from https://github.com/qt4cg/qtspecs/issues/1592#issuecomment-2493187896 and https://github.com/qt4cg/qtspecs/issues/1592#issuecomment-2495502890:

[USER2] More user feeback:

It’s confusing that the following function calls lead to completely different outputs:

elements-to-maps(
  <person>
    <name>Akila</name>
    <age>34</age>
  </person>
)

{"person":{"name":"Akila","age":"34"}}

elements-to-maps(
  <person>
    <name>Akila</name>
    <name>Jaha</name>
    <age>34</age>
  </person>
)

{"person":[{"name":"Akila"},{"name":"Jaha"},{"age":"34"}]}

The initial feedback I gathered so far is that the function works fine if the input is regular and uniform, but as soon as there are slight deviations, it can get wild. Here are some plain examples how a small change to the input results in fairly different output:

<xml>
  <info>X</info>
  <address>A</address><address>B</address>
</xml>
→ { "xml": ["A", "B"] }

<xml>
  <info>X</info>
  <address>A</address>
  <address>B</address>
</xml>
→ { "xml": [{ "info": "X" }, { "address": "A" }, { "address": "B" }] }

<xml id='id0'>
  <address>A</address>
  <address>B</address>
</xml>
→ { "xml": { "@id": "id0", "address": ["A", "B"] } }

Possible solutions:

• Enable uniform by default (performance considerations should not outweigh usability concerns)
• Change the rules for record from all-different(*!node-name()) to not(all-equal(*!node-name()))
• Editorial changes: Stress in the introduction that robustness is a secondary requirement.

Copied from https://github.com/qt4cg/qtspecs/issues/1592#issuecomment-2493180757, slightly revised:

For regular data, it is convenient to have heuristics that choose layouts automatically. For slightly irregular data that needs manual revisions, it can get messy:

[USER1] User feedback:

I have no idea which layout is used for my XML data. A function would be helpful that does not return the transformed data, but the layouts used for the transformation.

We could…

• offer an extra function,
• add a debug option to trace layout information, or
• add an option to include layouts in the output:

<p><a>A</a><b>B</b><c/></p> => elements-to-maps({ 'debug': true() })

{
  "p(record)": {
    "a(simple)": "A",
    "b(simple)": "B",
    "c(empty)": ""
  }
}

Even though the function may not be used primarily for mixed content, we should make it easier to convert such XML input to maps/JSON.

If I understand correctly, the safest solution to retrieve a consistent result currently is:

fn:elements-to-maps(
  $mixed-content,
  { "disable-layouts": ("empty", "empty-plus",
      "simple", "simple-plus", "list",
      "list-plus", "record", "sequence")
  }
)

Maybe we can simplify this? If we had an inclusive option, it could possibly be:

fn:elements-to-maps(
  $mixed-content,
  { "enable-layouts": "mixed" }
)

We could also consider xml:space=''preserve attributes and apply mixed to all descendant nodes (but it shouldn't be the only solution).

Related: #1592

Fix #1103

Simplifies the spec by doing line-ending normalization unconditionally prior to CSV parsing. CRLF sequences are no longer retained within quoted fields.

Fix #1637

Fix #1554

Changes the heading "formal specification" to "formal equivalent", and expands on the explanatory text.

Fix #1639

The rules for schema-aware layout selection produce no answer in the case where the element has a simple type but empty layout and simple layout are both disabled.

I propose to fall back to "mixed" layout in this case.

Fix #1634

Supersedes #1630

It says:

Equally, these specifications do not define any mechanism for creating context-dependent functions other than the built-in context-dependent functions, but neither do they rule out the existence of such functions.

which is no longer true (user-defined functions can take context-dependent default arguments)

Stylesheet change for cross-spec links (xspecref, xnt, xtermref) to drop the redundant "40" version suffix - for example the suffix becomes DM rather than DM40, since the vast majority of links will point to the 4.0 version of the document.

See also #1630.

The description of the decimal format properties in the XPath and XQuery static context needs to be updated to align with changes defining the options map of the format-number function.

The XSLT xsl:decimal-format element also needs to be checked for consistency, though most of the required changes have been made.

Fix #1627

Minor adjustments to the rules for fn:schema-type, fn:atomic-type-annotation, and fn:node-type-annotation based on implementation and testing experience.

I was surprised to discover that the xsl:map instruction does not allow a select attribute.

For many use cases it might make the instruction equivalent to xsl:sequence:

<xsl:map select="map:build(.....)"/>

<xsl:map select="{'a': 1, 'b': 2}"/>

but it still has documentary value; and there are other cases where it's not merely cosmetic:

<xsl:map select="$map1, $map2 => map:remove('extra'), {'extra': 17}"/>

There's no change to the semantics, the value of the select attribute is handled just like the value of the sequence constructor.

The specification of XSLT 3.0 on the matter is pretty loose and seems to allow the fact that xsl:apply-templates without select attribute should contains the elements to process inline.

It seems like a rather nice feature and may allow to do trick that are quite difficult to do right now

(copy of https://github.com/w3c/qtspecs/issues/31 )

Per #1250, minus-sign is now a string rather than a single character.

This change:

• corrects that in one place where it was still said to be a character
• changes the formulation from "represent" to "mark" a negative number.

Sorry for opening a branch in this repo. I did this accidentally, omitting the fork that I originally wanted to create.

Fix #1628

Hopefully the new rules are clearer. They were motivated by a couple of test cases using weird version numbers such as "4.00".

In XQuery 1.0 and 3.0 the version number was simply a string.

XQuery version 3.1 specified that

An XQuery version number consists of two integers separated by a dot.

In 4.0 we have taken this rather literally, and have spelled out the consequences in a note:

The version numbers 4.01 and 4.1 are equivalent: both have a major number of 4 and a minor number of 1. Version 4.10 by the same reasoning has a higher minor number than version 4.2.

This is completely counter-intuitive.

I propose that we eliminate the confusion by requiring the version number to consist of two single-digit integers separated by a dot.

The functions schema-type(), atomic-type-annotation(), and node-type-annotation() return a schema-type-record in which two of the fields are function items validate() and valid().

I've come to the conclusion that these are difficult to specify, difficult to implement, and difficult to test, and that the benefit of providing them is not great. I propose to drop them. They can always be added back in later.

Fix #1625

Purely editorial.

The note regarding leading-lone-slash ambiguity in 4.6.2 (relative path expressions) properly belongs in 4.6.1 (absolute path expressions)

document-node(a|b) is the same type as document-node(a) | document-node(b) but the current subtyping rules don't say this.

Revealed by test case misc/subtyping-076

Note, this problem existed before we introduced document-node(X), the same is true of the expansion:

document-node(element(a|b)) is the same type as document-node(element(a)) | document-node(element(b))

This PR is purely editorial.

It drops some rarely-used and imprecise terminology like "module context", and clarifies the description of the role of the static and dynamic context.

In XQuery, it pulls together the material from §2.3.5 (the "Serialization" section of the processing model) and Appendix C.1 (the "static context" appendix) into a new section 5.22 Output Declarations.

Note that much of this material differs between XPath and XQuery, so please review both.

Fix #1615

Fix #1619

Specifies an XSLT function map-for-key that converts a key to a map.

Refines the semantics of fn:key() to align with maps in edge cases.

Many functions that rely on equality-comparison of strings, for example deep-equal() and the eq operator, invoke compare(A, B, Collation). But we say that some collations only support equality comparison, not ordering. Presumably (we don't actually say), compare(A, B, Collation) will fail if the collation does not support ordering; but if it fails, then equality comparisons will fail as well.

It's not obvious what we should do about this. The simplest fix is probably to say that all collations must support ordering as well as equality comparison. Or we could have a fourth result value from compare() to say "values not equal, but their ordering is not defined"

Fix #332

I propose an XSLT function map-for-key('keyname', $root) which returns a map $M having the property that map:get($M, $key) returns the value of key('keyname', $key, $root).

This enables XSLT keys to be exploited in new ways: for example it becomes easy to merge the indexes for multiple documents, or to iterate over all the keys in a document.

These benefits can already be obtained by scrapping keys entirely and building maps instead; but keys do have some benefits (like remaining implicitly associated with particular documents, and being "more declarative") and if you've got a legacy application that makes extensive use of keys, this function gives you a bridging capability.

There are a few edge cases that will need ironing out, for example keys allow matching using a collation, which maps don't. (And the spec of xsl:key, now I come to think of it, says nothing about comparing date/time values in different timezones; I don't expect anyone has ever tried.)

We should make the serialization spec more liberal when it comes to the representation of double values. The prescribed output format is format-number(?, '0.0##########################e0'), which is very strict and often confusing when maps and arrays are serialized. Maps resulting from JSON conversions often contain doubles without users noticing it (related: #1583).

We should make the behavior implementation-dependent or align it with the serialization of JSON data (without losing its additional features to e.g. serialize function items or sequences). Backward compliance shouldn’t be an important issue, as the method was mainly introduced for debugging purposes.

Fix #1606 Fix #1506 Fix #1485

This PR drops the general concept of declaring named item types in XQuery and XSLT, and focuses on declaring named record types. The rules for named record types are tidied up editorially in XQuery (for example there is a clearer distinction between the syntax production RecordType and the concept of a record type, which can be declared either using that syntax, or otherwise). In XSLT the <xsl:item-type> declaration is dropped and an <xsl:record-type> declaration is introduced.

This is another PR related to #1610

@michaelhkay suggested that it would be nice to be able to use <function> consistently. This PR attempts to implement that. (It also implements <xfunction> which appears to have been an attempt to do this in the XSLT spec.)

Markup of the form <function>prefix:name#arity</function> will attempt to find the definition of prefix:name in the F&O and XSLT specifications. It will make an appropriate link. If no prefix is provided fn: is assumed and the #arity part is optional.

If someone can pull this PR locally (instructions below) and kick the tires (excuse me, "tyres") I'd appreciate it. I've done a little spot checking, but I can't say I've been comprehensive.

If we agree to merge this, it will then be possible to cleanup markup in some places. For example, it appears that the F&O spec relies on special processing of <code> rather than <function>. We should never have done that!

(There's no point looking at the formatted version of this PR, it's all build changes that won't be reflected there.)

It's not at all clear what these terms are supposed to mean; they are rarely used, and when they are used, they only cause confusion.

For example, the sentence "The names of public variables and public functions must be unique within the [module contexts] of a query" doesn't bear scrutiny. (Can variables have the same names as functions? Yes they can. Can they have the same names as private variables and functions in the same module? No they can't.)

The term "expression context" can probably be usefully replaced in most places by "the static context of an expression".

I think that the idea behind "module context" is that a large part of the static context for expressions is the same for all expressions within a module. But I think that when we use the term, there is usually a better way of saying what we mean.

I changed xfunction refs so that they point to the right URI for 40 functions.

(Partial fix for #1610 )

Fix #1605

Debugging output was accidentally left in place.

Fix #1608

If you look in, for example XSLT, at references to the "FO40" spec, they actually attempt to link the w3.org location where it would have been if it was a REC. Not sure what the fix is, but...

Fix #564

Introduces ordered maps: specifically, sorted maps which return entries in order sorted by key, and fifo maps which return entries in the order of insertion.

Although this has been on the TODO-list for a long time and has many useful applications, raising a PR at this stage is particularly motivated by comments on the elements-to-maps() function pointing out that having a predictable order of properties in serialized JSON can be very useful, and that many existing XML-to-JSON converters achieve this. This gives the opportunity, for example, to parse JSON into a representation that retains input order, delete and/or add some properties, and then serializate the JSON with the order retained.

The properties of fn:compare state that it is context-dependent on collations, but fail to say that it also depends on implicit timezone.

Fix #1590.

A draft spec for this function is in the function-catalog, but it has never been referenced in the published spec and the draft is incomplete.

This PR has no impact on the published specs, only on processes that access the function catalog.

We started with named item types, they were mainly intended for defining records, but allowed any type. Then we realised that records required extra capability, especially recursive definitions and constructors, so we introduced declare record for that case. This begs the question as to whether the general declare type is still useful enough to merit inclusion. I suspect that if we had declare record and didn't have declare type, no one would be clamouring for it.

It's not as if all our work on this feature is done. There's still a fair bit to do on the XSLT side, as well as issues like #1520, and I don't think we've really sorted all the issues relating visibility of types to visibility of variables and functions using those types.

Generally, functions that construct a new node tree return a document node rather than an element node. This is friendlier, because it means for example that path expressions starting with "/" can be used. (An exception is analyze-string, which it's too late to change).

I propose to bring csv-to-xml into line.

Fix #1593

Updates and aligns the "Other Operations" sections for maps and arrays.

F&O Sections 18.3.1 (Singleton Arrays) and 18.3.2 (Value Maps) are almost identical to each other, and neither seems to bear much relationship to the section heading. The material is non-normative so this is a purely editorial issue.

Fix #1516

1. Fixes some examples in the spec where the expected results were apparently incorrect
2. Introduces a mechanism for giving a test assertion for an example that is separate from the published result, for example where alternative results are possible
3. Marks some tests as schema-aware so a non-schema-aware processor won't attempt to run them

All the tests for features that are implemented in Saxon now run successfully.

Issue: #1594

Issue: #1598

One unfortunate thing about $err:stack-trace is that it is difficult to serialize, for example as json: serialize($err:map, { 'method': 'json' }) does not work anymore.

I think we should not focus on optimization concerns, but rather return a plain string. If an implementation wants to optimize it further, it shouldn’t be that hard to internally represent it as a lazy string that is generated only when requested.

Issue: #1595

I've reworked the rules for selecting a layout. There's probably more to be done, but this is a start - feeback welcome. I'm marking this "revise" for the moment because I haven't finished it yet. There's no deliberate changing of the spec apart from fixing errors.

I think there is a slight wording/grammar issue in https://qt4cg.org/specifications/xpath-functions-40/Overview.html#func-concat saying

The function can now take any number of arguments (previously it had to be two or more), and the arguments can sequences of strings rather than single strings.

It it probably meant to say

The function can now take any number of arguments (previously it had to be two or more), and the arguments be can sequences of strings rather than single strings.

I spotted two typos in the XSLT 3 spec, and I'm reporting them here because they are in the XSLT 4 spec as well, in https://raw.githubusercontent.com/qt4cg/qtspecs/refs/heads/master/specifications/xslt-40/src/xslt.xml

• dependant, in "versions of the packages on which this package is dependant."
• "the the ", in "appears in the the initializing expression"

I propose that the syntax document-node(NameTest) be allowed as a synonym for document-node(element(NameTest)) to match a document node that has exactly one element node child matching NameTest (possibly with comment, and processing-instruction siblings).

The rationale is that document-node() is often used in situations where a well-formed document is required, and people are more likely to use the more precise type document-node(*) if it can be expressed with less verbosity.

For example, the return type of parse-xml() can then be document-node(*), and the return type of parse-html() can be document-node(*:xhtml). I propose also that csv-to-xml() be brought into line by returning document-node(fn:csv).

There are a number of places in F&O where we currently accept or return document-node() and could be more specific by changing this to document-node(*).

The verbosity is especially apparent when we want to use the type (document-node(element(X)) | element (X)) which can now be abbreviated to (document-node(X) | element(X)) which reads much more clearly,

There are also a number of places where we currently require element(*) -- for example the first argument of elements-to-maps -- where it would be user-friendly to change this to (document-node(*) | element(*)) (with the semantics that supplying a document node has the same effect as supplying the outermost element of the document).

We should also clarify that document-node(element()) does not match a document node having text node children - something that is not currently stated very explicitly.

This is a placeholder for feedback on the recently added fn:elements-to-maps function.

Adopted from https://github.com/qt4cg/qtspecs/pull/529#issuecomment-1765060154 (and as also suggested by @dnovatchev), some rules still refer to JSON. I think we should refer to the XDM, XML or maps instead. Examples:

• mapping XML to ~~JSON~~ a map
• ~~JSON~~ Map equivalent (13x) → adjust syntax
• their ~~JSON~~ map equivalents
• …etc

Issues that have not fully been discussed: https://github.com/qt4cg/qtspecs/pull/529#issuecomment-1765761565

• https://github.com/qt4cg/qt4tests/issues/181: empty-plus shouldn't require that attributes exist.
• https://github.com/qt4cg/qt4tests/issues/180: "list" incorrectly states that it doesn't apply where the INNER element has attributes.

…more to come.

I propose to classify E[P] as an implausible expression if the only possible value of P that has an effective boolean value is the empty sequence.

An example might be $uris[parse-uri(.)]. The result of parse-uri is either a map or an empty sequence, so computing the EBV will give either false or an error.

Classifying an expression as implausible licenses the processor to reject it as a static error.

It appears in the function catalog but not in the specification.

See also #1588

This is a minor thing, but it annoys me every single time. Open up the XSLT spec and search for any instruction and the first hit in the ToC is always 19.8.4.x "Streamability of [instruction]" which has never been the reason I was looking for the instruction.

Seems we could move 19 to 27, putting it before Serialization or we could tinker with the markup so that 19.8.4.x didn't appear in the ToC.

Adds the function fn:binary-resource

Also fixes some inconstencies in the handling of static/executable base URI in other resource access functions.

Fix #557

Changes the rendition of record type definitions so each is now defined in a section of its own, extracted from the function catalog into the narrative spec by means of a processing instruction, following the precedent of function definitions. Record type definitions can therefore be cross-referenced using a specref, but they are automatically cross-referenced if named in a function signature.

Fix #1527

This PR updates the RELAX NG grammar to be (more) consistent with the XSD grammar (and consequently more correct for XSLT 4.0)

1. Relaxes the definition of QNames so that prefixes declared with fixed-namespaces can be supported
2. Adds the fixed-namespaces attribute to xsl:stylesheet, xsl:transform, and xsl:package
3. Adds the xsl:switch element
4. Adds the xsl:array and xsl:array-member elements
5. Adds shallow-copy-all to the possible values for on-no-match
6. Adds separator to xsl:apply-templates
7. Removes select attribute from xsl:copy
8. Adds as attribute to xsl:sequence
9. Updated the declarations for xsl:accumulator-rule, xsl:array, xsl:array-member, xsl:attribute, xsl:catch, xsl:comment, xsl:map-entry, xsl:matching-substring, xsl:message, xsl:namespace, xsl:non-matching-substring, xsl:on-empty, xsl:on-non-empty, xsl:otherwise, xsl:param, xsl:processing-instruction, xsl:sequence, xsl:sort, xsl:value-of, xsl:variable, xsl:when, and xsl:with-param so that they accept either a select attribute or a sequence constructor

More changes may also be required. A comprehensive comparision of the RNC and XSD schemas is needed, #1584

It's likely that we've allowed them to drift.

If JSON numbers are converted to XML and serialized as JSON, it is confusing to end up with an E notation for large numbers. An example:

'100000000000000000000'
=> parse-json()
=> serialize(map { 'method': 'json' })

Obviously, lossless roundtripping is not possible (1e20 is a valid JSON number, so we cannot distinguish it from 100000000000000000000), but as the E notation is much less common than integers, maybe we could try to return more numbers in their integer representation if the result would be equivalent?

Related: #1445

The reference to §13.2.9 of the WhatWG DOM spec should be a reference to §13.2.9 of the WhatWG HTML spec.

Fix #767

I found that the two functions mentioned in the issue (document and function-available) had been updated as suggested. However I also checked all the XSLT-specific functions, and found that for current-merge-group(), the prose has been updated to say what happens if the argument is omitted, but the signature does not actually define a default. I have corrected this.

Fix #69

I wonder if we could relax the 3rd line of the current signature...

map:contains(
   $map	as map(*),	
   $key	as xs:anyAtomicType	
) as xs:boolean

...to:

   $key	as xs:anyAtomicType?	

It hit me tonight, as I was running a lengthy operation, and some nulls were passed to key. I expected the function to return false, not raise an error.

I have (a) added more explanation of why the conversion is needed, and (b) described the conversions more prescriptively.

Fix #1493

As well as allowing record() for an empty record type as proposed in #1491, this PR also allows named record declarations in XQuery to have no fields. In the course of implementing this I discovered there was old text in the F&O "Constructors" section which duplicated but had become out of sync with the XQuery spec, so much of this has been deleted and replaced with a cross-reference. Named record definitions for XSLT have not yet been defined, this is unfinished business.

Fix #1491

Marks the relevant production rules as XQuery-only

Fix #1574

Supersedes #529 Fix #528

Coming back to this after a long gap, I have redrafted the proposal. I've tried to take as many of the comments into account as possible, but I'm aware that I haven't responded to them all.

The XPath grammar currently projects the following productions, which appear to be only relevant to XQuery, are not reachable from the XPath root, and either lack an if="xquery40" condition, or wrongly have xpath40 in the condition:

ExtendedFieldDeclaration, ParamListWithDefaults,ParamWithDefault

In fact, the grammar doesn't allow default values for arguments of inline functions. Is this the case?

Fix #1552

Merging allowed a changes block to become split into two blocks, which isn't allowed.

Functions and Operators

There are 4 classes of function here:

1. functions that have to be implemented natively -- e.g. fn:parse-html;
2. functions that are implemented in terms of native operations -- i.e. the dm:* and op:* functions;
3. functions that can be implemented in XSLT or XQuery but can be done more efficiently natively;
4. functions that can be implemented in XSLT or XQuery as efficiently as they can natively.

It could be useful to generate a function library of the form namespace/function.xqy and namespace/function.xsl that has the implementation of the functions that can be implemented in XSLT and XQuery. This would allow implementors to import/include those implementations into their processors/engines. -- This is more flexible than providing them all in a single file as implementors can include the functions they don't have implementations for without having to edit the files every time the spec changes.

Note: JavaScript supports polyfill files for new classes/functions so that engines that don't support those features can get a functioning implementation of that function/class.

Note: Many JavaScript engines implement various functions in JavaScript itself.

XPath and XQuery

We could make the EBNF available as a separate file in addition to the iXML grammar that has been discused/worked on. This would help implementors on the lexer and parser at least. There's not much else we can do here as the language is custom.

XSLT

We have the XMLSchema and RelaxNG grammars to help with validation. Implementors could use these in their build systems to provide API bindings to the data model.

XDM

We could provide the XDM/XPath specific XMLSchema extensions as a separate XMLSchema definition to allow implementors to get access to the type infomation for these such as for xs:numeric.

Drops the newly-introduced fn:node-kind() function in favour of a more general function fn:type-of().

Fix #1567

Unfinished business from issue #668.

Analogously to the current HTML case-insensitive collation (which case-normalises ASCII characters only), define a Unicode case-insensitive collation that case-normalizes all Unicode characters. It is basically equivalent to converting both strings to lower-case and then comparing using code-point collation. Although UCA collations allow for case-insensitivity, they combine this with lots of other baggage such as ignoring punctuation characters.

Some 4.0 changes have no entries in the change log.

Would it be realistic to move the most important EXPath specifications (Binary, File, maybe other modules) to the W3 realm?

Nowadays, it has become close to impossible to get into contact with Florent Georges reliably, and we have no guarantee that https://expath.org/ remains online.

I would be ready to transform the File Module to a new format and to maintain it in the future.

Fix #1555

Fix #1563

primitive-type and base-type are functions.

The base type of positiveInteger is nonNegativeInteger.

Fix #1561

Test case catalog-005 now passes, showing that all the non-error stylesheets in the test suite are valid against the schema.

The schema for XSLT 4.0 (included as a free-standing file and incorporated as an appendix) is not a valid schema.

(a) The union type for the fixed-namespaces attributes contains children of the form <xs:simpleType ref=".."/> which though intuitive, is not allowed.

(b) The first xs:assert in the definition of xsl:for-each-group has misplaced parentheses in the call on count().

In addition, the schema needs some updates to catch up with the latest XSLT 4.0 syntax changes. The problems are revealed by XSLT 4.0 test case catalog-005.

Fix #1548

XSLT 3.0 specified no default for xsl:output/@indent in the case of the JSON and Adaptive output methods. This PR sets the default to "no".

I believe this is sufficient to close #1548.

Purely editorial; adds cross-references to map:build (for example from XSLT and XQuery grouping) to make the function more visible.

@Fix #1515

Fix #1522

Purely editorial.

Fix #1553

Issue: #1486

See https://github.com/w3c/qt3tests/issues/65

where it is pointed out that we have test cases that assume the default for the escape option of parse-json() is false, whereas the spec says it should be true.

The Saxon implementation (and presumably any other implementation that passes the tests) sets the default to false, and if we were arguing from first principles then I think I would argue this is a better choice.

We need either to change the tests or to change the spec. Since we can't change the 3.1 spec retrospectively, neither choice is particularly attractive.

In the XQFO, the “Formal Specification” sections present XPath/XQuery expressions that are equivalent to the introduced function.

In a previous meeting, it has been noted that “Equivalent Expression” may be a better term for these sections.

Would it be sufficient to simply rename the section, and adapt the wording in 1.5.5 Formal Specification?

The effect of positional predicates on axis steps (for example preceding-sibling::*[1]) is an area that causes users a lot of trouble. We could provide a more formal definition, and we could also provide more notes and examples.

In particular, we haven't expanded the notes and examples to explain what happens when you have a range of integers such as preceding-sibling::*[1 to 3], which is now allowed. (Spoiler alert: if the siblings are A B C D E, you get C D E in that order).

It feels rather odd that the result of fn:siblings() should include the argument node, except when it is parentless.

If fn:siblings were defined as preceding-sibling::node() | self::node() | following-sibling::node(), then the start node would be included even if it is parentless -- and even if it is an attribute or namespace node.

As noted during the review at yesterday's meeting. (I was supposed to correct it before applying the PR, but pressed the wrong key...)

See original issue #148.

There have been requests for further type information beyond that supplied by the four new functions

node-kind()
atomic-type-annotation()
node-type-annotation()
schema-type()

One of the requests was to be able to test if an item is a map, and array, some other function, a node, or an atomic value.

This could perhaps be done by broadening node-kind() to a function item-kind() that returns "map" for a map, "array" for an array, etc. We could also return the result in item-type syntax, say map(*) or array(*) or function(*).

What else is needed?

Fix #1534

In the discussion of PR #1497 at meeting 096, some concern was expressed that the default value for indentation might be problematic in testing.

It was observed that all of the serialization parameter settings are defined by the host language, not by the serialization specification, but that did not resolve the concerns.

Can we/should we/would we mandate that indentation is disabled by default? (Is that not already the case in XQuery and XSLT?)

Fix #1542

Fix #1538

I also did some editorial cleanup of the serialization spec, in particular parameters like indent now have the value true or false, while recognizing that some host languages may allow alternative representations such as yes/no or 1/0.

Fix #1539

Fix #1540

Fix #1463

It would be good if the definitions of the various axes were less informal.

The four axes children, parent, attribute, and namespace are defined directly in terms of data model accessors.

The remaining axes can be defined as follows, where axis names are used as functions:

self($node): $node
ancestor($node):  transitive-closure($node, parent#1)
ancestor-or-self($node): ancestor($node) | $node
descendant($node): transitive-closure($node, child#1)
descendant-or-self($node): descendant($node) ! $node
following($node): $node / ancestor-or-self() / following-sibling() / descendant-or-self()
following-or-self($node): following($node) | $node
following-sibling($node): parent() / child() [. >> $node]
following-sibling-or-self($node): following-sibling($node) | $node
preceding($node): $node => ancestor-or-self() / preceding-sibling() / descendant-or-self()
preceding-or-self($node): preceding($node) | $node
preceding-sibling($node): parent() / child() [. << $node]
preceding-sibling-or-self($node): preceding-sibling($node) | $node

We should change the rules for XSLT global variables to align with the new rules for XQuery.

Specifically, drop the rule that a global variable is out of scope within its own definition, falling back on the existing circularity rules to disallow cases where the reference is genuinely circular.

The effect would be to allow a global variable to be bound to a recursive inline function, for example (nonsense example)

<xsl:variable name="tot" select="fn($x){if ($x=0) then 0 else $tot($x - 1) + 2"/>

It is not easy to find out the correct timezone for a given UTC xs:dateTime in the current region. We have fn:implicit-timezone, but it only refers to the current date and time.

The following code works for at least BaseX and Saxon; it applies the system’s default time zone to a given xs:dateTime item:

let $dtm := xs:dateTime('2024-07-01T01:01:01Z')
let $ms := xs:integer(($dtm - xs:dateTime('1970-01-01T00:00:00Z')) div xs:dayTimeDuration('PT0.001S'))
let $tz := xs:dayTimeDuration('PT' ||
  Q{java:java.time.ZonedDateTime}ofInstant(
    Q{java:java.time.Instant}ofEpochMilli($ms),
    Q{java:java.time.ZoneId}systemDefault()
  )
  => Q{java:java.time.ZonedDateTime}getOffset()
  => Q{java:java.time.ZoneOffset}getTotalSeconds()|| 'S')
return adjust-dateTime-to-timezone($dtm, $tz)

It returns 2024-07-01T03:01:01+02:00 (MESZ) on systems located in Leipzig and nearby cities.

We could either introduce a function that

1. returns an xs:dayTimeDuration timezone for a given xs:dateTime item (with the system’s default time zone applied), or
2. converts an xs:dateTime item to the system’s default time zone at the given date/time.

Suggestions for good names are welcome.

See also https://xmlcom.slack.com/archives/C01GVC3JLHE/p1730216267200449:

Add XSLT support for the new json-lines serialization option (PR #1497)

I recently wrote a multi-phase transformation and tried out the new "enclosing modes" feature where the template rules for a mode are enclosed within the xsl:mode element. Worked very well, and really helps to give the stylesheet a more modular structure. But I found myself wanting to write "helper" functions within the xsl:mode definition.

I don't think it would too difficult to add this feature. I imagine that such functions would be scoped to the enclosing xsl:mode, and would automatically have higher import precedence than anything outside the mode. There are probably a few complications e.g. if the arity range overlaps a function with the same name declared outside the enclosed mode, but I would think it's manageable.

When using XSLT transformations, it would be helpful in many cases to know the transformation target, that is the base-uri of the primary result document. This would allow, for example,

• to make the uri of secondary result documents relative to the primary base-uri to ensure that all output is in the same target directory
• to copy media files from transformation source directory into transformation target directory
• to generate a transformation report as secondary output, which informs and links to the primary result document

I am pretty sure that there are situations where the URI of the primary result document is undefined or unknown. The XPath function fn:document-uri is decribed as "Returns the URI of a resource where a document can be found, if available.". So maybe we could have a new function fn:primary-result-document-uri as "Returns the URI of the primary result document, if available.".

Fix #1478 Fix #1325

This PR drops variadic functions, reverting to the situation in previous versions where concat was in a special category of its own.

We decided (see the referenced issues) not to introduce further variadic system functions, mainly in the interests of extensibility, and the same argument apply to user-defined functions.

It is not a great hardship to write f((x, y, z)) rather than f(x, y, z) and dropping the feature therefore removes a fair bit of complexity that has proved to be of rather limited value.

Nearly all XSLT instructions that accept a sequence constructor also allow the input to be supplied using a select expression.

xsl:result-document is an exception.

A particular use case for this is with a multi-phase transformation where you want to capture the result of the first phase of processing in a variable, and then output it before further processing, perhaps for diagnostics or perhaps because the processing pipeline branches.

1. Adds advice and guidance on avoiding reserved words
2. Drops XQuery-specific material from the XPath spec
3. Adds a paragraph to the incompatibilities appendix

Fix #1519

Fix #1499

Fix #1500

Fix #1525

Observations/conclusions from the exchange on Slack regarding computed node constructors:

1. We should highlight the breaking change in the appendix: J.1 Incompatibilities relative to XQuery 3.1.
2. We should present a list of keywords which is ambiguous and exclude other ones (like e.g. count or value)
3. Maybe we should generally discourage users from using the legacy NCName syntax, and remove corresponding examples in the spec, as further versions of the language may lead to new incompatibilities.
4. Syntax errors in quoted element names were should already be detected at parse time, for example by using pseudo quotes:

# currently
CompNodeName    ::=  StringLiteral | UnreservedName | ("{" Expr "}")
CompNodeNCName  ::=  StringLiteral | UnreservedNCName | ("{" Expr "}")
# proposed
CompNodeName    ::=  UnreservedName | ('"' UnreservedName '"') | ("'" UnreservedName "'") | ("{" Expr "}")
CompNodeNCName  ::=  UnreservedNCName | ('"' UnreservedNCName '"') | ("'" UnreservedNCName "'") | ("{" Expr "}")

A quick evaluation over appr. 8,000 XQuery files resulted in the following list of occurrences of possible incompatibilities (spread across appr. 400 files):

• 300x attribute type {
• 107x element option {
• 98x element record {
• 85x attribute count {
• 81x attribute value {
• 31x element value {
• 24x element type {
• 18x attribute text {
• 15x attribute values {
• 12x attribute namespace {
• 11x attribute default {
• 10x element text {
• 10x attribute key {
• 10x element key {
• 9x element group {
• 9x element count {
• 8x element to {
• 8x element item {
• 5x element div {
• 4x attribute collation {
• 4x attribute encoding {
• 4x attribute comment {
• 3x attribute context {
• 3x attribute case {
• 3x element map {
• 3x element empty {
• 2x element items {
• 2x attribute to {
• 1x attribute empty {
• 1x attribute item {
• 1x element element {
• 1x attribute where {
• 1x element values {
• 1x attribute if {
• 1x element in {
• 1x element comment {
• 1x attribute start {
• 1x attribute end {
• 1x element document {

We're making progress here but there are still things that need fixing.

Currently a record definition like uri-structure-record is expanded:

(a) into a full definition (with explanations of all the fields) at the point where a PI of the form <?record-description uri-structure-record?>appears. This may be either within a specific function entry in the function catalog, or in the narrative prose of xpath-functions.xml.

(b) into a concise definition (without explanations of fields) at the point where the type is referenced in a function signature.

A hyperlink to the record definition is created (i) where the type name appears in a function signature, and (ii) manually, using the generic link format <loc href="#uri-structure-record">uri-structure-record</loc>. However, the target of the link is the first concise definition (see (b) above) rather than the full definition.

There are also some limitations in the rendering of the definitions. The full definition does not indicate which fields are optional/required, though this information is available in the XML. Neither the full nor the concise definition appears to indicate whether the record is extensible.

There's a parse-xml() call in the test generator stylesheet that's failing to process some of the test examples, for reasons that aren't entirely clear. This is causing the entire build to fail. This PR adds a try/catch around the parse-xml() call so that a failure only affects the individual tests, not the entire build.

In 3.2.6 Enumeration Types we already have some explanation of how enumeration types work, and I think it's sound, but I think a few more words about the consequences might be useful.

We note correctly that 'red' instance of enum('red', 'green', 'blue') is false.

We should also note that let $red as enum('red', 'green', 'blue', 'yellow') := "red" return $red instance of enum('red', 'green', 'blue') is true; and indeed that a string S that is cast or coerced to any enumeration type that permits S is an instance of every enumeration type that permits S. This is a conscious design decision that has both advantages and disadvantages, so we should explain the consequences carefully.

By analysing the coercion rules for complex data structures (see the discussion in #1501), I wondered which error codes, apart from XPTY0004, may arise from the coercion of records.

If we attempt to coerce a function item to a string, we get FOTY0013:

let $r as xs:string := true#0
return $r

I would expect to also get FOTY0013 if a map value, which is a function item, is coerced to a string. Is this correct? If not, how does this case differ from the first one?

let $r as record(a as xs:string) := { 'a': true#0 }
return $r

Provides four new functions:

• node-kind
• schema-type
• atomic-type-annotation
• node-type-annotation

to return type information using a new record structure schema-record-type.

Fix #148 Partial Fix for #1271

I've tested the grammar against the ~500 distinct patterns in the stylesheets of the attr/match test sets (which is the largest collection I can find). As far as I can tell, there is one ambiguity inherent in the current grammar, which is not covered by notes, in that the pattern

id()

(and similar for element-with-id(), key() and root()) can parse (in reduced form) in two ways:

<Pattern40 xmlns:ixml="http://invisiblexml.org/NS" ixml:state="ambiguous">
   <RootedPath>
      <FunctionCallP>
         <OuterFunctionName>id</OuterFunctionName>
         <ArgumentListP/>
      </FunctionCallP>
   </RootedPath>
</Pattern40>
<Pattern40 xmlns:ixml="http://invisiblexml.org/NS" ixml:state="ambiguous">
   <PostfixExprP>
      <FunctionCallP>
         <OuterFunctionName>id</OuterFunctionName>
         <ArgumentListP/>
      </FunctionCallP>
   </PostfixExprP>
</Pattern40>

that is, there are two paths, via RootedPath and RelativePathExprP/StepExprP/PostfixExprP to get from PathExprP to FunctionCallP

Fix #1357

This is a purely cosmetic PR. I think @ChristianGruen is right that it is unfortunate that we've lost the distinction between "new" and "only updated" functions. I'm trying to make that work again, in a way that's more visually distinct.

I have:

1. Use Δ for changed sections.
2. Use Δ➕ for new functions. (I'm making no effort to determine if non-function sections are new or changed; I doubt that it's either worth the effort or likely to be correct. One could argue that ➕ alone is sufficient, but I liked the consistency this way. Even new functions do change between drafts.)
3. Per @michaelhkay, a function is "new" if it doesn't appear on F&O 3.1.
4. I found it distracting that the delta symbol preceded the ToC entry. That's purely aesthetic, I guess, but I've put them at the end.

You can see the results here: https://qt4cgtest.nwalsh.com/branch/iss-1357/xpath-functions-40/Overview.html

The specification of Item Type Declarations has this restriction:

The declaration of an item type (whether locally declared in a module or imported from a public declaration in an imported module) must precede any use of the item type name: that is, the name only becomes available in the static context of constructs that lexically follow the relevant item type declaration or module import. A consequence of this rule is that cyclic and self-referential definitions are not allowed.

But modules explicitly are allowed to have cyclic dependencies:

Implementations must resolve cycles in the import graph, either at the level of target namespace URIs or at the level of location URIs, and ensure that each module is imported only once.

Does that mean that the modules below are valid? They depend on each other, but the types that they define do not have a cyclic dependency. In particular, the use of each type lexically follows the relevant module import, as asked for.

If this case must be supported, the usefulness of the above restriction for XQuery processors is somewhat limited, IMO it compromises this suggestion for handling declared types:

Named item types have been designed so that a reference to an item type name can be expanded (that is, replaced by its definition) as soon as the reference is encountered during query parsing.

(: a.xqm :)
module namespace a = 'a';
import module namespace b = 'b' at 'b.xqm';
declare type a:t1 as b:t2;
declare type a:t2 as xs:integer;

(: b.xqm :)
module namespace b = 'b';
import module namespace a = 'a' at 'a.xqm';
declare type b:t1 as a:t2;
declare type b:t2 as xs:integer;

Add the axes preceding-or-self, following-or-self, preceding-sibling-or-self, following-sibling-or-self, with the obvious meanings.

A minor convenience avoiding the frequent need to write things like (. | preceding-sibling::*)[@x][last()] ; especially useful because preceding-sibling-or-self returns results starting at the context node whereas (. | preceding-sibling) reverses the order.

No issue raised.

Adds links from change metadata in the spec to issue and PR numbers in GitHub in many cases where these were previously missing.

Corrects many of the errors in tests identified in issue #1516

I'm getting the following test failures (exluding ones where the Saxon implementation is known to be incomplete).

fo-test-fn-csv-to-xml-004 to -010

These are failing because the results are formatted with whitespace which the example expression does not actually generate.

fo-test-fn-format-number-005

Error on line 3 at column 16 (in expression on line 1116) of ... qt4tests/app/fo-spec-examples.xml FODF1310 format-number picture: Digit sign must not appear after a zero-digit sign in the integer part of a sub-picture

The test is format-number(12345, '0,###^0', { 'percent': '%:pc' }) Result: "14pc"

which seems to be a complete aberration.

fo-test-fn-format-number-008, -009, -010

fail saying decimal-format 'de' is not defined. The dependency is documented in narrative prose, but is not codified so that the test generator knows about it.

fo-test-fn-function-annotations-* **

failing because the examples use a non-existant function xs:QName#2. Probably fn:QName#2 is intended.

** fo-test-fn-highest-005 ** Error XPST0142 Keyword key does not match the name of any declared parameter of function fn:highest

Possible Saxon bug ??

fo-test-fn-hours-from-dateTime-007

Example is incorrect - it calls year-from-dateTime not hours-from-dateTime

fo-test-map-build-007

Results are not deep-equal to the stated result -haven't established why. Possible whitespace issue.

fo-test-map-pairs-001

Getting "assert-permutation failed" from the Saxon test driver. Possible problem with the test driver?

xpath 4 offers a new function fn:partition: "Partitions a sequence of items into a sequence of non-empty arrays containing the same items, starting a new partition when a supplied condition is true". Looks like an equivalent to XSLT xsl:for-each-groupwith @group-starting-with.

I would appreciate another new HOF as an equivalent to XSLT xsl:for-each-groupwith @group-by, that is Partitions a sequence of items into a sequence of non-empty arrays containing the same items, where all items in a partition give the same value when a function f is applied. This is called gather-by in Mathematica.

The change in 4.0 for the higher order function for-each is that "the $action callback function accepts an optional position argument". But the function signature is

fn:for-each(
  $input as item()*, 	
  $action as fn(item(), xs:integer) as item()* 	
) as item()*

I read $action as fn(item(), xs:integer) as item()* as a function with two mandatory parameters item() and xs:integer. Since the second (position) argument should be optional, i would expect:

fn:for-each(
  $input as item()*, 	
  $action as fn(item(), xs:integer?) as item()* 	
) as item()*

There are several HOFs with this new optional position argument which seems to be mandatory in the function signature.

Fix #1512

The new rules banning unquoted reserved names in computed element and attribute constructors should apply equally to processing instruction and namespace node constructors.

Fix #1345

Having changed computed element/attribute constructors to require reserved names to be quoted, we can now reintroduce bare-brace map constructors without ambiguity.

Fix #1509

In XQuery §5.11, the paragraph starting "The first [URILiteral]) in a schema import..." contains obsolete information about the handling of location hints. It should refer instead to the new and more complete treatment given four paragraphs later at "The [URILiterals] that follow the at keyword are optional location hints..."

Fix #1507

Make the "formal spec" of fn:parse-integer legible (and portable between XPath and XQuery) by avoiding use of XML character references in the code.

In the formal spec of fn:parse-integer, it is not visually clear what the second argument of translate() is in

let $preprocessed-value := translate($value, "_ 	

", "")

We should spell out the characters by using codepoints-to-string() or char().

I’ve tagged this as a discussion issue:

Would it make sense to declare constructor functions for the new XQuery type declarations, similar to what we now have for records, or do we believe that’s over the top?

It might improve typing in complex code, and it would allow us to write things like:

declare type world:continent as enum('Africa', 'America', 'Asia', 'Australia', 'Europe');
world:continent('Africa')

For the type above, an implicit constructor function would be created that would simply coerce its argument to the declared type:

declare function world:continent($value) as world:continent { $value };

One current drawback is that this only works for prefixed types (see #657), whereas it’s possible to define types without prefix, and reference them in local type declarations:

declare type continent as enum('Africa', 'America', 'Asia', 'Australia', 'Europe');
let $c as continent := 'Africa'
return $c

Fix #1503

Issue: #868

XSLT needs to be brought into line with XQuery in terms of the variables available in a catch clause, in particular $err:map (see PR #493)

Issue: #1458

Issue: #1318

Coercion rules added for maps and arrays.

In an application I am writing now, the xsl:output-characters I am writing in my xsl:character-map are of interest elsewhere in the XSLT complex that is slowly emerging.

The exercise makes me realize that character maps can be interesting in their own right. We give xsl:character-maps names, and include them within each other, because they group meaningfully related character-string pairs. Such sets are the sort of thing one might want to have more closely coupled to the XSLT apparatus. For example, someone might create a xsl:character-map to deal with Unicode characters in a particular script. And those characters are of interest in their own right, and the character selection might engage with other processes that need to interact with those characters.

What if we were to extend @use-character-maps to allow character maps to draw from other preexisting maps? The list of eqNames in @use-character-maps would be resolved first against names of character maps. For any eqName that is not the name of a character map, the processor would search for a global variable or global parameter by that name. Any referenced global variable/parameter must be empty or a map. Every key must be castable as a character, and every value must be a string. Failure on any of these points would raise an error.

Here is an example of hypothetical XSLT code, to illustrate how the innovation might be productively useful, producing two different character maps, each of which might be appropriate for one type of serialization or another:

    <xsl:item-type name="letters:grc" as="record(transliteration as xs:string, name as xs:string)"/>
    <xsl:variable name="master-map" as="map(*)">
        <xsl:map>
            <xsl:map-entry key="'α'" select="letters:grc('a', 'alpha')"/>
            <xsl:map-entry key="'β'" select="letters:grc('b', 'beta')"/>
        </xsl:map>
    </xsl:variable>
    <xsl:variable name="serialization-transliteration-map" as="map(xs:string, xs:string)">
        <xsl:map>
            <xsl:for-each select="map:keys($master-map)">
                <xsl:map-entry key="." select="$master-map(current())?transliteration"/>
            </xsl:for-each>
        </xsl:map>
    </xsl:variable>
    <xsl:variable name="serialization-name-map" as="map(xs:string, xs:string)">
        <xsl:map>
            <xsl:for-each select="map:keys($master-map)">
                <xsl:map-entry key="." select="$master-map(current())?name"/>
            </xsl:for-each>
        </xsl:map>
    </xsl:variable>
    <xsl:character-map name="transliteration" use-character-maps="serialization-transliteration-map"/>
    <xsl:character-map name="names" use-character-maps="serialization-name-map"/>

In other words, if an xsl:character-map is just a map, why not give it access to other XSLT structures that are maps?

I suggest that instead of the repetitive use of paragraphs like "The json-lines serialization parameter is not applicable to the XML output method.", we should have a general statement that serialization parameters are not applicable unless otherwise specified; perhaps accompanied by a chart showing which parameters apply to which methods.

Fix #1366

Closes #1471 and #1472.

Fix #1495

Also corrects one or two broken cross-spec links.

Drop the "context value static type" from the definition of the static context.

In the absence of a specification for static typing, the feature is unused, and we have dropped similar features such as statically-known document and collection types.

It has been reported to me that the XQuery specification provides a nice and compact introduction on maps and arrays, but there currently is no comparable introduction for records yet. The “Changes” section on 3.2.8.3 Record Type gives a hint: It contains the sentence “Record types are added as a new kind of ItemType, constraining the value space of maps.”.

In addition, maybe we could rename the section Named Record Types to “Record Declaration”, analogous to “Variable Declaration”, “Context Value Declaration” and “Function Declaration(s)”. “Item Type Declarations” could be renamed to “Type Declaration”.

Maybe I was too quick in waving through #1476 as I believe that the current version is a bit sketchy. It says:

An element $E named number is processed by copying the string value of $E to the output, making any changes that are necessary to ensure that the result is a valid JSON number. Such changes include:

• Removing leading and trailing whitespace.
• Removing a leading plus sign.
• Removing redundant leading zero digits.
• Adding a zero digit before or after a decimal point that is not preceded and followed by a digit.

1. For input like X, we cannot ensure that it will be a valid JSON number, so I assume that the changed string needs to be validated before being output?
2. “Removing redundant leading zero digits” may not consider negative number like -01.
3. Does “Such changes include” imply that the list may not be comprehensive?

In many cases, the numbers to be output will be the result of an earlier json-to-xml conversion. If it is generated with XPath numbers, we shouldn’t encounter plus signs, redundant leading zeros etc. either, so I would suggest getting rid of all post-processing. We could simply say:

An element $E named number results in the output of the string value of the element if it is a valid JSON number. Otherwise, [FOJS0006] is raised.

If we do want to tweak the string value, we should provide a complete set of rules; maybe something like…

The string value of an element $E named number is modified by:

• removing leading and trailing whitespace,
• removing a single leading plus sign,
• removing redundant leading zero digits, which are optionally preceded by a leading minus sign,
• adding a zero digit before a decimal point that is not preceded by a digit, and
• adding a zero digit after a decimal point that is not followed by a digit.

If the result is a valid JSON number, it is output. Otherwise, [FOJS0006] is raised.

…but that’s still fuzzy (for example, it lacks the explanation of what redundant zero digits are).

Also updates the list of tokens using angle-brackets.

Fix #1312

Even if the use cases may be limited: Shouldn’t we allow empty record types and declarations for the sake of completeness?

(: check for empty map :)
if($map instance of record()) then ...

declare record empty();

let $empty as empty := {}
return $empty

Fix #1308

Fix #1483

We reviewed and accepted a revised specification for xsl:array, but there was some unease about one of the options: specifically the ability to supply the content in the form of a sequence of arrays, which is then converted to an array of sequences.

Further work on implementation and on writing test cases inclines me to drop this option. There are several reasons:

1. It is error-prone. When constructing nested arrays, there is a tendency to use xsl:array/xsl:array rather than xsl:array/xsl:array-member/xsl:array, and rather than leading to an error, this leads to incorrect results which can be hard to diagnose.
2. The specification relies on converting an array to a sequence using $array?*, but this is lossy, for example the array [(1,2),3] is converted to the sequence (1,2,3). This provides a further source of potential confusion when users get it wrong.
3. Use of xsl:array-member seems to handle all the requirements and result in more readable code, and it's best to focus on providing a smaller number of different ways of achieving the same effect.

The proposal is to drop option 2(e) of section 22.1.

XQFO:

• [x] fn:hours-from-dateTime: Wrong example: year-from-dateTime( xs:time("12:30:00") ) → https://github.com/qt4cg/qtspecs/pull/1517/files#diff-7625c07ae8131ff65c3caa677b188ed2b9b66237312d11c05a2fa2838c6f5c67R9794
• [x] fn:void: “Formal Specification” should be dropped. As it is ·implementation-dependent· whether the supplied argument is evaluated or ignored, the empty sequence may not be equivalent.
• [x] fn:parse-uri: Ampersands in 4th example (...&sort=relevance) must be escaped.
• [x] fn:format-number: format-number(0.14, '01%', {'percent': '%:pc')
• [x] Unify equivalent expressions
• [x] fn:civil-timezone: $dateTime → $value
• [x] fo-test-fn-siblings-002

XPath:

• [x] Add changes section for =?> (https://github.com/qt4cg/qtspecs/pull/985)

XQuery:

• [x] declare record p:person {$first as xs:string, $last as xs:string, *);
• [x] declare type app:invoice as map("xs:string", element(inv:paid-invoice))
• [x] declare type app:overdue-invoices as map("xs:date", app:invoice*)

General:

• [x] Format code (see #1124)

In PR #1355 we have added "declare record" syntax to XQuery.

We should now do the same for XSLT.

In general a function (or other operation, e.g. an XSLT instruction) that expects a record type as input should make that record type extensible. For example, array:of-members should accept record(value as item()*, *) rather than record(value as item()*) as currently defined; similarly map:of-pairs should accept record(key, value, *).

Two reasons: (a) it avoids the user having to remove extraneous fields from records if they happen to be present, and (b) it avoids the system having to check whether extraneous fields are present.

For example, it now becomes legal (and perhaps sometimes useful) to write array:of-members(map:pairs($map)); currently this fails because the result of map:pairs includes key fields which array:of-members does not permit.

The function fn:error uses none as return type. This type is only defined in the outdated XQuery 1.0 and XPath 2.0 Formal Semantics specification:

1. Do we want to live with the current state?
2. Should we include a better definition in the current specifications?
3. Should we simply use item()* as return type, even if the function never returns anything?

If we keep none, the test case generator should be revised for Keywords-fn-error-1.

Attempts an improved (more intuitive) specification for the xsl:array instruction.

Fix #1468

Fix #1448

Fix #1450

In §5.14 we document two changes to "declare default element namespace":

1. The [default namespace for elements and types] can now be declared to be fixed for a query module, meaning it is unaffected by a namespace declaration appearing on a direct element constructor. [citation missing - should be issue #65, PR #753]

2. The [default namespace for elements and types] can be set to the value ##any, allowing unprefixed names in axis steps to match elements with a given local name in any namespace. [Issue #296 PR #1181]

We don't discuss how these two options interact. Are they completely orthogonal?

The description of ##any says its effect is that "an unprefixed name appearing in any other context where an element or type name is expected is treated as being in no namespace." This deserves a mention in the rules for direct and computed element constructors.

In §2.2.1 we have added the statement: The statically known namespaces may include a binding for the zero-length prefix; however, this is used only in limited circumstances because the rules for resolving unprefixed QNames depend on how such a name is used.

It's not clear how such a binding can come into existence. It can't be done using any of the prolog declarations that bind a prefix. And under the current rules, I don't think it can be done using a namespace declaration attribute on a fixed element constructor. Perhaps it was the intent that xmlns="abc" would do this if "fixed" is set, but that's not the case currently.

Section §4.12.1.2 Namespace Declaration Attributes could do with a "changes" section explaining what has changed.

In §4.12.1, Direct Element Constructors, we say "If the element name has no namespace prefix, the namespace binding for the zero-length prefix in the [statically known namespaces] is used; if there is no such binding, the element name will be in no namespace." That can't be right, there's no mention of the default element/type namespace at all.

In §4.12.3.1 Computed Element Constructors, the same problem arises for a bare unprefixed NCName, but for a dynamically-computed name, we use the default element/type namespace.

We've made a decision -- for good reasons of maintaining extensibility -- not to make any of the functions in the system function library variadic.

I think this raises the question of whether the feature is worth retaining. And we already had the situation that many functions that were obvious candidates for variadicity (min, max, distinct-values, sum) couldn't use the feature because they already had additional optional arguments.

If it's not a good design principle for system-defined functions, then the same applies equally to user-defined functions.

We could drop the feature and just revert to defining concat() as a special case. It would remove a significant amount of complexity.

Fix #1475

Fix #1474

Describes the changes that might be needed to supplied numbers to ensure the output conforms with JSON syntax.

Independently, adds a note to appendix to G.4 to affirm that changes to argument keywords have no backwards compatibility implication, satisfying action QT4CG-091-01

For example in uri-structure-record as shown beneath the signatures of fn:build-uri and fn:parse-uri, the field names are not marked "?" to indicate the field is optional.

The underlying XML contains this information correctly.

xml-to-json now retains the format of supplied numbers, except that it strips off redundant leading zeros to ensure they are valid JSON.

It also needs to strip off redundant leading plus signs for the same reason.

…see https://github.com/qt4cg/qtspecs/pull/1437#issuecomment-2346599503.

Issue: #1471

Extracted from #576:

All serialization methods except for json allow sequences to be output on top level. We should also allow this for JSON data. This way, constructs like…

declare option output:item-separator '
';
declare option output:method 'text';

for $json in ({ 1: 2 }, { 3: 4 })
return serialize($json, { 'method': 'json' })

…can be simplified to:

declare option output:method 'json';
{ 1: 2 },
{ 3: 4 }

We should still not allow sequences within JSON structures. It would be inconsistent to have different output rules depending on the number of items in a sequence ((1) and [1] would not be serialized identically).

Issue: #689

With the number of functions in the fn:namespace now standing at 213, the use of a drop-down in the F+O "function finder" has become a little unwieldy. Are there ways it could be improved, for example by reducing the size of the list when you type the first character of the required function name?

In reading the XSLT specs on the xsl:array constructor (#406 adopted at meeting 28) I find myself somewhat confused. Either I need to be illumined, the specs need to be clarified, or some other adjustments are needed.

First, I'm still struck by the disparity between maps and arrays; there's a xsl:map-member but no xsl:array-member. That horse has been beaten many times, I know. But I need to record that the disparity is very noticeable. The disorientation is aggravated by the second example in the constructor overview:

<xsl:array use="?value">
   <xsl:for-each-group select="0 to 19" group-adjacent=". idiv 4">
     <xsl:map-entry key="'value'" select="current-group()"/>
   </xsl:for-each-group>
</xsl:array>

A map entry within an array? This just strikes me as a hack, and is suggestive of a design flaw in the language. (I know what you're about to say in response; hold that thought...) That's exacerbated when I see example 2c invoking an array constructor within an array, only to get rid of the nested array to get at what you wanted in the first place, the items grouped into members.

Furthermore, the value of attribute @use, ?value, is cryptic. Yes, I get what's going on, but the first several times reading it I thought it was very abracadabrish. And that same feeling hit me for the other examples' values of @use: .() and ?*. The .() bit of syntax appears only twice in the specs (here), without explanation.

If @use is evaluated once for every item, then in example 2b...

<xsl:array use=".()">
   <xsl:for-each-group select="0 to 19" group-adjacent=". idiv 4">
     <xsl:sequence select="current-group#0"/>
   </xsl:for-each-group>
</xsl:array>

...I would expect the expression .() to be applied to twenty items, because a sequence of five sequences of four items each is simply one sequence of twenty items, innit? Or are we allowed here and only here for a sequence constructor to create a sequence of sequences?

It seems that @use is striving to do something similar to @group-by in xsl:for-each-group. But it isn't. Understanding the goal is obfuscated by the vague name. Use what, to what end?

Overall, I feel that the array constructor is a 2nd-class citizen in the specs. I love using the map constructor in XSLT: it tells users very clearly what's going on. I don't look forward to the using the array constructor in XSLT, because I think it does the opposite. But maybe I'm wrong.

I don't have a specific proposal to fix, because I understand some of the conceptual hurdles to a putative xsl:array-member. Still. I can't help but wish that that's what we had.

• Two largish sections had duplicate prose nearby;
• Some punctuation rendered consistently;
• Some substantive insertions/edits for clarification, when it seemed clear to me.

Let me know if any of these changes are misfires.

Fix #1464

This PR addresses issue #1461 in a fairly narrow way, without attempting to tackle the deeper problem identified in issue #1336.

This involves expanding the fos:type entries for types such as uri-structure-record that are referenced in function signatures, resulting in some duplication with the fos:record-description entries that describe the same types in a different place.: hopefully the resolution to #1336 will resolve that duplication.

We use both spellings.

oxfordreference.com says: Although the variant spelling implementor predominated for much of the late 20th century, today implementer is considered standard.

Personally, being a late 20th century kind of guy, I prefer "implementor", but we should be consistent.

1. All other XQFO functions that have . as the default value for a node parameter (fn:name, etc.) return an empty sequence if the passed argument is an empty sequence. Could we use the same rule for fn:element-number?
2. The last example is incomplete; it must be $e//section...
3. The last example should return 1.2 instead of 1.1. If the given result is correct, the equivalent XPath expression needs to be revised.

While it was a no-brainer to implement the function, its specification seems overwhelming to me. I eventually read all the notes, but at the end I was more confused than in the beginning ;·) It could be one of those functions that are more accessible to users than to implementors, though.

Still, maybe at least 1, 2 more examples could be provided? The last example goes into that direction, but it could be a bit cryptic for non-experts (e.g. it expects users to know that the result of $s/ancestor-or-self::section is returned in document order).

As this function seems to be mostly targeted to XSLT users, maybe we should offer XSLT examples. I assume that XQuery developers may rather be tempted to enumerate all children/descedants of a node, and write recursive code like…

declare function enumerate($element, $numbers) {
  for $child at $pos in $element/*
  let $n := $numbers || '.' || $pos
  return ($n, enumerate($child, $numbers || $n))
};

In general, if we were able to generalize the function in one way or another, that would be great.

The default value for the option parameter of fn:deep-equal is:

{ 'collation': fn:default-collation() }

Can we simply use {}, or is there a particular reason for mentioning the collation and no other option?

          ### Errors in `misc/BuiltinKeywords`

Test cases Keywords-fn-parse-html-1 and Keywords-fn-build-uri-1 contain invalid function signatures, caused I think by a bug in generate-keyword-test-set.xsl. Both seem to omit the SequenceType of one of the arguments:

 fn:parse-html(html := ?, options := ?) instance of 
    function((xs:string | xs:hexBinary | xs:base64Binary)?, ) as document-node(element(*:html))?

fn:build-uri(parts := ?, options := ?) instance of function(, map(*)?) as xs:string

Examining generate-keyword-test-set.xsl shows:

<xsl:for-each select="arg">
      <xsl:if test="position() != 1">, </xsl:if>
      <xsl:text>{@type}</xsl:text>
</xsl:for-each>

but for these two function definitions, type information is indirected via a @type-ref attribute:

<fos:proto name="parse-html" return-type="document-node(element(*:html))?">
        <fos:arg name="html" type="(xs:string | xs:hexBinary | xs:base64Binary)?"/>
        <fos:arg name="options" type-ref="parse-html-options"
                     default="{
                                 &quot;method&quot;: &quot;html&quot;,
                                 &quot;html-version&quot;: &quot;5&quot;
                              }"/>
 </fos:proto>

and

<fos:proto name="build-uri" return-type="xs:string">
            <fos:arg name="parts" type-ref="uri-structure-record"
               example='{
               "scheme": "https",
               "host": "qt4cg.org",
               "port": (),
               "path": "/specifications/index.html"
               }'/>
            <fos:arg name="options" type="map(*)?" default="{}"/>
</fos:proto>

Originally posted by @johnlumley in https://github.com/qt4cg/qtspecs/issues/1451#issuecomment-2358082401

Second attempt. Closes #1323 and #1436.

In addition, includes a change to fn:json-doc($uri), and fixed an editorial bug for the fn:escape-html-uri function.

I try to understand the semantics behind the Properties section in the XQFO spec. Here are some examples of existing functions:

1. fn:format-integer

The two-argument form of this function is ·deterministic·, ·context-dependent·, and ·focus-independent·. It depends on default language. The three-argument form of this function is ·deterministic·, ·context-independent·, and ·focus-independent·.

2. fn:format-time

This function is ·deterministic·, ·context-dependent·, and ·focus-independent·. It depends on default calendar, and default language, and default place, and implicit timezone.

3. fn:index-of

The two-argument form of this function is ·deterministic·, ·context-dependent·, and ·focus-independent·. It depends on collations, and implicit timezone. The three-argument form of this function is ·deterministic·, ·context-dependent·, and ·focus-independent·. It depends on collations, and static base URI, and implicit timezone.

4. fn:element-number

The zero-argument form of this function is ·deterministic·, ·context-dependent·, and ·focus-dependent·. The one-argument form of this function is ·deterministic·, ·context-independent·, and ·focus-independent·. The two-argument form of this function is ·deterministic·, ·context-independent·, and ·focus-independent·. The three-argument form of this function is ·deterministic·, ·context-independent·, and ·focus-independent·.

I am afraid I cannot give hints for improvement yet; I just noticed it is increasingly difficult to interpret. For the moment, I wonder…

1. Are all of these current definitions correct?
2. In many cases, if the user supplies an empty sequence as argument, the default value is used instead. This contrasts with the current property description that relies a lot on the function arities. For example, fn:index($a, $b) and fn:index-of($a, $b, ()) should have identical properties, but the current wording implies that the three-argument of this function has different properties, e.g., it is said to rely in this case on the static base URI.
3. Some functions with default values have a single property description (see fn:format-integer), other have multiple descriptions (see fn:format-time). Is there a particular rule behind that?
4. Could we merge the descriptions for arities with identical properties?
5. Maybe it will be easier not to consider the arities at all anymore?

The following functions define an argument that has a default value, but don't allow () to be supplied explicitly:

lang
id
element-with-id
idref
parse-html

As determined by running the following query against the function catalog:

//*:arg[@default and not(ends-with(@type,'?') or ends-with(@type, '*'))]/(ancestor::*:function/(@prefix||':'||@name))

There are others which this query doesn't pick up, mainly where the defaultable argument is a function, including:

array:build map:build array:get map:get fn:highest fn:lowest

Past thoughts:

• https://github.com/qt4cg/qtspecs/issues/826#issuecomment-1821359131
• https://github.com/qt4cg/qtspecs/issues/1456#issuecomment-2358257374 ff.

Copied from Michael:

It would be great if we could agree on a collective term for "maps and arrays". "Structured item" feels too generic to me. I've toyed with terms like "tabulation", "tabula", "composition", "dataset", "compendium", "aggregate". Perhaps "combo"? It's best to have a word that stands out from the crowd if we can't find one whose meaning is self-explanatory.

We have dropped the syntax ??type(T) for filtering the results of lookup expressions, because of problems with syntax ambiguity. This issue seeks an alternative.

Although selection by type also makes sense with shallow lookup, it is most relevant with deep lookup. The main need arises with intermediate steps of a path such as ?? X ?? Y which gives a dynamic error if X selects something that is not a map or array. This is consistent at one level with // X // Y, except that // X can never select something that isn't a node.

The main problems with filtering using an [. instance of record(p, q)] predicate is that it's very long-winded. For example, if we want to select only those members of a selected array that are sequences of a particular record type, without flattening everything else, we have to write something like ?? values::* ?[. instance of record(p, q)+] ? *, which is a bit of a nightmare.

Starting from the end goal, I would like to be able to write something close to ??record(first, last) to select all the items of this record type at any depth. We know that syntax doesn't work, because ??NCName is already taken. That's also true for ??items::record(first, last), unless we change the rules for what can appear after ::.

Also, there's another syntax hazard: what we want here is a SequenceType, not an ItemType, and that means that it can contain a trailing ? occurrence indicator, which is easily confused with the next lookup operator in a path.

Looking at it from all angles, I do feel the best solution is to prefix the record(first, last) with a marker character so that we know we've got a type filter here. Characters that might do the job include @, #, $, %, ^, ~. Of these, my preference remains ~, for three reasons:

(a) it's currently unused: overloading a different symbol is more likely to cause visual confusion (b) one of the traditional uses of ~ is to indicate a "matches" or "is kind of like" relationship. (c) there's a mnemonic association between "tilde" and "type" (compare "at" and "attribute")

Fix #1445

Fix #1449

Fix #1451

Could they?

S.M.O.P. I suppose.

The example for fn-load-xquery-module contains I think an errant semicolon:

let $expr := "2 + 2"
let $module := `xquery version "4.0"; 
                module namespace dyn="http://example.com/dyn";
                declare %public variable $dyn:value := {$expr};`
let $exec := load-xquery-module("http://example.com/dyn", 
                                {'content':$module}); 
let $variables := $exec?variables
return $variables(QName("http://example.com/dyn", "value"))

The semicolon after the third let is not permitted by the grammar - it only appears as a separator in Version, Module and Prolog. Removing that semicolon permits a parse under the current grammar.

The syntax of computed element and attribute constructors causes parsing problems, and problems in extending the grammar (for example it restricts the contexts in which bare-brace map constructors can appear).

In the syntax element|attribute _name_ { _content-expression_ } I propose that:

(a) it should be possible to use a StringTemplate in place of the name. This will normally just be the name enclosed in backticks but interpolated expressions are allowed.

(b) reserved names should be disallowed. This is a backwards-incompatible change. The reserved names are the non-delimiting terminal symbols listed in A.3.2. We should add a warning that additional names might be reserved in future versions, and advise use of backticks.

I don't especially do this.

a) because the environment that we run XSLT in (95% of the time) doesnt support it b) because its too rigid to use (maybe I'm doing it wrong).

Motivation:

I have 'module' List.xsl lets say that models lists, and has a function (in psuedo XPath so you can see the types) (a module contains constructors and ideally all functions related to, here, a list)

function tryHead($xs as list()) as maybe()

I have 'module' Maybe.xsl lets say that models Maybe

function toList($xs as maybe()) as list()

So Maybe.xsl needs to know about List.xsl and List.xsl needs to know about Maybe.xsl

if I use xsl:include or xsl:import (in saxon) I get:

The stylesheet module includes/imports itself directly or indirectly

(which I'm happy reflects the correct behaviour given the spec - i.e. I don't think this is a bug in the implementation)

Given that I CAN write this cycle in a single file, without restrictions on the order of the constructs (i.e. this isnt a restriction of the language itself - this isnt alway true in other languages), it seems less than ideal that I can't freely compose files in order to replicate the situation an allow decomposition into logical files.

(I expect this is a consequence of the rules around priority of templates, which I am broadly ignorant of, and to be honest, largely not directly concerned with - I don't use this to 'compose' templates, but to write 'function' libraries)

(MK has answered a question in stack overflow related to this, which solved the issue by reorganising the files, which is what I currently do, but its not ideal that I cannot decompose and test code in isolation)

Is this worth resolving?

Many years ago I was a C programmer, and I remember a similar issue that was resolved with #define, whereby header files were imported something like this (excuse my made up C syntax)

#ifndef _Maybe_
#include "Maybe.h"
#endifndef

and then Maybe.h would define _Maybe_, this would mean, if you followed the idiom, that the file was included by the preprocessor, at most once (I'm not suggesting this mechanism is directly applicable, just that this is a common issue elsewhere).

Now that we support choice types in function signatures, we can easily generalize functions such as year-from-dateTime to work on all types in the dateTime family (specifically dateTime, date, time, gYear, gYearMonth, gMonth, gMonthDay, gDay). A request for a component that is not present in the value returns ().

We can apply this to all seven X-from-dateTime() functions.

Of course this makes the X-from-date() and X-from-time() versions redundant, but that's not a problem.

Fix #1446

Section 9.1.1 says

All minimally conforming processors must support positive year values with a minimum of 4 digits (i.e., YYYY)

The word "minimum" here is ambiguous.

I suggest

All minimally conforming processors must support year values in the range 1 to 9999.

For bonus points, drop the use of the undefined term "minimally conforming", and link instead to something defined in section 1.2 Conformance.

I try to, but I don’t get happy with the number-formatter option of fn:xml-to-json. It results in a lot of special-casing at serialization time for a very special case. More generally, …

• The option would be much more helpful for outputting maps and arrays as JSON with at serialization time, but it is not available as standard serialization parameter.
• fn:xml-to-json represents numbers as text nodes, so numbers already have a string representation. If needed, they could easily be preprocessed before being serialized as JSON.

I imagine that the major use cases for the option will be:

• to avoid scientific representation for doubles, and
• to be able to serialize INF and NaN.

Couldn’t we simply allow numeric string values (i.e., the children of fn:number elements) to be output without changes?

'1000000' => json-to-xml() => xml-to-json()  (: 1000000 instead of 1.0E6 :)
xml-to-json(<fn:number>INF</fn:number>)      (: INF instead of error :)

We already have the number-parser option, which allows us to generate a string representation for numbers that will be suitable for being converted back with fn:xml-to-json. For other cases, the XML source can easily be updated before being passed to fn:xml-to-json.

If we really want to keep the option, …

1. We should add examples for its usage in the spec.
2. We need more tests.
3. I would suggest using the function type (fn(xs:untypedAtomic) as xs:untypedAtomic)?, similar as for fn:replace and fn:json-to-xml (this would allow things like xml-to-json($input, { 'number-formatter': fn { . + 1 } })).
4. We should also add it as a general serialization parameter to be able to write:

serialize(xs:double('NaN'), {
  'method': 'json',
  'number-formatter': fn($n) { if(is-NaN($n)) then '"NaN"' else $n }
})

If it’s too much hassle to introduce it as a general serialization parameter, we should at least provide it for fn:serialize.

Thank you, Wendell.

1. Make the productions a little easier to read by removing the border and background on code elements in production tables.
2. Make the background color for production notes a little less ... intense.

Adds rules for the default priority of new match pattern options such as element(p:*) and element(p|q).

Fix #1394

Improves the presentation of grammar rules.

• Production numbers are dropped
• The width of the RHS column is increased so there is less line wrapping
• The summary EBNF is alphabetically sorted

In #1387 @ChristianGruen observes that the + seems to be a special case in the tel: scheme. In fact, I think we concluded that the special case is that you shouldn't encode the path segment of a non-hierarchical URI.

This PR attempts to implement that.

Closes #1235

Closes #1322

Closes #1325

Closes #1323

Closes #1421

Closes #1373

Closes #1422

With function declarations, recursive functions can be declared:

declare function local:factorial($x) {
  if($x > 1) then $x + local:factorial($x - 1) else $x
};
local:factorial(5)

We should allow the same for variable declarations:

declare variable $factorial := fn($x) {
  if($x > 1) then $x + $factorial($x - 1) else $x
};
$factorial(5)

I believe it is sufficient to simplify the definition of initializing expressions and drop the exception “other than the variable being declared”.

Related: https://www.w3.org/Bugs/Public/show_bug.cgi?id=15791 (the static dependency check was given up before due to fn:function-lookup).

Closes #1379

Closes #1372

Fix #1427

fixes #1403

Allow Array and Map tests to omit the asterisk to match any array or map.

• array() <> array(*)
• map() <> map(*)

Fix #1426 (partially; it would also be good to have a better citation to IEEE 802.3)

I propose adding a function to perform node numbering in a manner analogous to xsl:number (but without the formatting aspects, which can be handled using format-integer, and without multi-level numbering).

I envisage a function along the following lines:

node-number($node as element(), 
                         $from as (document-node()|element()), 
                         $count as fn($node as element()) as xs:boolean?)
   as xs:integer

The function returns the number of element nodes that satisfy all the following conditions:

• they are descendants of $from
• they are preceding-or-self nodes of $node
• they satisfy the $count predicate

$node defaults to the context node. $from defaults to the parent of $node. $count defaults to a function that returns true for an element that has the same name as $node, false otherwise.

With no arguments, node-number() applied to (say) a <p> element returns the number of preceding-sibling <p> elements plus one. [Not quite. Under this definition, it would also count <p> elements that are descendants of a preceding-sibling element]

The rationale for this proposal is (a) to make the core functionality of xsl:number available in environments other than XSLT, and (b) within XSLT, to make it available in contexts such as a predicate of a match pattern where it is currently difficult or impossible to invoke xsl:number except by wrapping it in a user-defined variable or function.

Test case hash-str-034 takes a 150,000-character string consisting of 50,000 repetitions of the string "ABC", and computes its CRC-32 hash value.

On Java the result comes back as decimal 105475755, which we translate to 0x06496EAB

On C# the result comes back as a 4-byte array AB 6E 49 06. My guess is we need to reverse the byte order, but I'd like to understand why...

I guess we need to say something about byte ordering, but what?

I tried to find the definitive specification. We simply point to IEEE 802.3 without a specific URI or section or version number. 802.3 actually appears to be a large family of standards. Can we do better than that?

I'm interested to know whether the specification actually defines the result to be a 32-bit (unsigned?) integer rather than an array of four bytes. If that's the case, then our spec needs to say how the 32-bit integer is converted to a sequence of 4 octets, which is what fn:hash actually returns. The C# example shows that there's more than one way of doing it.

Close #1424

Thank you, Amanda.

The XQuery 3.1 spec and XQuery 4.0 draft spec both have an extra "s" in the word expressions, showing it as expresssions. The typo occurs in the C.1 Static Context Components section.

I know this repo is only for 4.0 work. Should I make a separate issue in the https://github.com/w3c/qtspecs repo?

This is a slightly more radical alternative to #1388

In addition to clarifying the encoding rules (maintaining slightly different rules for path segments, query parameters, and fragment identifiers), it removes the path-separator and query-separator options.

We should make the algorithm argument in the the function signature of fn:hash explicit and change the signature from…

fn:hash(
  $value    as (xs:string | xs:hexBinary | xs:base64Binary)?,	
  $options  as map(*)?  := {}
) as xs:hexBinary?

…to:

fn:hash(
  $value      as (xs:string | xs:hexBinary | xs:base64Binary)?,	
  $algorithm  as xs:string? := 'MD5',
  $options    as map(*)?  := {}
) as xs:hexBinary?

It will happen often enough that the function is used for algorithms other than MD5:

(: OLD :) hash("ABC", { "algorithm": "CRC-32" })
(: NEW :) hash("ABC", "CRC-32")

In yesterday’s meeting, fn:unix-time was added to the spec. It was proposed to…

1. drop support for negative values and
2. change the function name to fn:unix-dateTime.

I sympathize with both suggestions. What do others think?

There's a markup error in the hash function that breaks the automatic test build. This PR fixes that.

(I'm just going to merge this straight away.)

Replace a few remaining occurrences of "atomic value" with "atomic item".

Fix #1415

Adds xsl:item-type to the list of declarations. Adds xsl:array. xsl:map, and xsl:map-entry to the list of instructions.

Fix #1408

A built-in record type should be defined for key-value pair maps, which currently defined as follows…

A key-value pair map is a map containing two entries, one (with the key "key") containing the key part of a key value pair, the other (with the key "value") containing the value part of a key value pair.

…as we can expect them to be used more often in the future. Would it be possible/make sense to assign the record type to the xs namespace (xs:pair)?

In addition, we should…

• use it consistently in the text (XQFO, deep lookups, etc.),
• replace existing record(key as xs:anyAtomicType, value as item()*) references with xs:pair, and
• rename $input or $map arguments to $pair.

Examples for function signatures to be updated:

map:of-pairs(
  $pairs    as xs:pair*,
  $combine  as fn(item()*, item()*) as item()* := fn:op(',')
) as map(*)

map:pair(
  $key    as xs:anyAtomicType,
  $value  as item()*	
) as xs:pair

map:pairs(
  $map  as map(*)	
) as xs:pair*

We could add numerous other record types, but maybe we can handle those in separate issue and keep this one focused on key/value pairs.

https://qt4cg.org/specifications/xslt-40/Overview.html#dt-declaration needs xsl:item-type adding

As currently written, preliminary parts of the the XSLT specs are out of sync with other specs, or have inconsistencies.

• The section called Abstract mostly describes what's different in 4.0. That despite a section shortly later specially devoted to the topic.
• The section called Abstract does not lead with its best foot forward: the language does a lot more than XML-to-XML transforms.
• The section called "What is XSLT?" never answers the question posed in the header.
• The section called "What is XSLT?" begins with a sentence that both repeats earlier material and is not germane to the header.

In the attached PR, I have proposed moving several paragraphs. I have not intentionally edited them. I have also taken a stab at defining XSLT. I have tried to calibrate the Abstract so that it is both sparing, in the spirit of the XQFO abstract.

I don't think that what I've put in here is any way finalized. But I hope it is a step in the right direction, and a good basis for further refinement.

This PR disposes of my action to add absolute to the output of fn:parse-uri.

The rules that define an absolute URI: "has a scheme", "does not have a fragment identifier", "is a hierarchical scheme" don't fit neatly into a single place, so there are two different places where the setting is considered.

Because of this, I think it's simpler to set absolute to true when the URI is absolute, but not to false when it isn't. That could be rectified with a bit more prose.

Fix #1411

The current spec defines uri-structure-record as having a

path-segments? 	as xs:string?

property, which doesn't make sense and also contradicts the example given later:

The unescaped form is easily accessible from path-segments:

("", "path", "to", "a/b")

It should be xs:string*.

Just testing

Fix #1401

The main changes are:

• The three derived types xs:integer, xs:dayTimeDuration, and xs:yearMonthDuration are no longer treated as primitive for the purpose of this section. They are now treated as derived types, but given special status where necessary as "quasi-primitive".
• In places where the F+O rules give the same result as the canonical representation in XSD 1.1, we now defer to XSD 1.1 rather than replicating the rules. Many of the rules originate with XPath 2.0, which was published before XSD 1.1, but which anticipated some of the changes in XSD 1.1, for example the use of a seven-component model for dates/times, and a two-component model for durations. XPath 3.0/3.1 failed to take advantage of the resulting opportunity for rationalisation.
• Generally the language is a bit less terse, with more notes and examples
• The rules have more to say about the type annotation of the result. In some places the spec appeared to imply that the type annotation on the result must be the target type; in others it appeared to imply that the type annotation must be unchanged from the source (for example 19.1.1 "If ST is xs:string or a type derived from xs:string, TV is SV. [presumably with unchanged type annotation]). The spec is now hopefully clearer that the result TV MUST be an instance of TT and MAY be an instance of some other type derived from TT, especially in the case where the value is unchanged.

The description of XPTY0117 still refers to the "function conversion rules" rather than the "coercion rules".

This has been discussed on Slack. The key points are:

1. The terminology in the table of content is mixed (Union Types vs Function Tests), focused on the grammar not the organisation.
2. The test sections (Function Test, etc.) lack definitions and overviews of the item types like is done with the other types, such as choice item types.

I also think the other sections like the Subtype Relationships are largely fine as they are as they are dealing with specific rules around subtypes, coercion, etc.

So I propose:

1. Adding an overview of what the function types/tests, etc. are like is done with the union types, etc.
2. Adding a note to the maps overview similar to "A map is a function of the form function (xs:string) as item()* that can be passed a key name and returns the value of that key entry."
3. Adding a note to the arrays overview similar to "An array is a function of the form function (xs:integer) as item()* that returns the items of the array at the specified index, or raises a FO.... error if the index is out of bounds."

I also propose renaming and reorganizing parts of the table of contents as follows (omitting unaffected sections):

• 3 Types
  • 3.1 Sequence Types
    • 3.1.1 Examples of Sequence Types
    • 3.1.2 Sequence Type Matching
  • 3.2 Item Types
    • 3.2.1 General item types
    • 3.2.2 Atomic Types
    • 3.2.3 Union Types
    • 3.2.4 Namespace-sensitive Types
    • 3.2.5 Δ Choice Item Types
    • 3.2.6 Δ Enumeration Types
    • 3.2.7 Δ Node Types
      • 3.2.7.1 Simple Nodes
      • 3.2.7.2 Δ Elements
        • 3.2.7.2.1 Schema Elements
      • 3.2.7.3 Δ Attributes
        • 3.2.7.3.1 Schema Attributes
    • 3.2.8 Δ Function Types ~ NOTE: Covers the current 3.2.8.1 Δ Function Test section.
    • 3.2.9 Maps and Records
      • 3.2.9.1 Map Types
      • 3.2.9.2 Δ Record Types
        • 3.2.9.2.1 Recursive Record Tests
    • 3.2.10 Array Types

And to make the heading names in the subtype section consistent:

• 3.3 Δ Subtype Relationships
  • 3.3.1 Subtypes of Sequence Types
  • 3.3.2 Subtypes of Item Types
    • 3.3.2.1 General Rules
    • 3.3.2.2 Choice Item Types
    • 3.3.2.3 Atomic and Union Types
    • 3.3.2.4 Nodes
      • 3.3.2.4.1 General Rules
      • 3.3.2.4.1 Elements
      • 3.3.2.4.1 Attributes
    • 3.3.2.5 Functions
    • 3.3.2.6 Maps
    • 3.3.2.7 Arrays
    • 3.3.2.8 Records
    • 3.3.2.9 Named Items

Fix #1399.

The only change is to remove a sentence from <note>...</note> markup.

In the invisible-xml function, the $grammar argument has type item()?. There's no evident reason why it isn't xs:string?; there's no clue in the specification or example what it might mean to supply something that isn't a string (or an empty sequence).

Note that declaring it as item() rather than xs:string? suppresses atomization of the supplied value.

I always wondered why ElementTests for any element can be written as element() which is equivalent to element(*) but as map() and as array() are not permitted and always have to include the asterisk (map(*)) .

This is still the case even in XPath4 and I think we should have a look, if this can be omitted. It does not add any value IMHO, but maybe I am overlooking something important.

https://qt4cg.org/specifications/xquery-40/xquery-40.html#doc-xquery40-ElementTest https://qt4cg.org/specifications/xquery-40/xquery-40.html#doc-xquery40-ArrayTest https://qt4cg.org/specifications/xquery-40/xquery-40.html#doc-xquery40-MapTest

The "fix" would be as simple as

[255]   AnyMapTest   ::=  "map" "(" "*"? ")"

[265]   AnyArrayTest ::=  "array" "(" "*"? ")"

-- update after comment from @michaelhkay --

The same does not apply to FunctionTest as as function() specifies a zero-arity function wheras as function(*) matches functions of any arity.

Updates the schema for XSLT 4.0 stylesheets to accommodate changes recently agreed:

• Support for xsl:function/@variadic
• Support for extensions to decimal-format properties, for example percent="%:pc".

We've just been examining the rules for casting from duration / dayTimeDuration / yearMonthDuration to string (F+O section 20.1.1) and they are fairly impenetrable: I think we could do better.

• We use the phrase "canonical representation as defined in XSD 1.1 part 2"; a more specific reference would be helpful. Mentioning the function ·durationCanonicalMap· would take readers straight to the right place.
• The actual XSD spec has one or two typos, and uses some notations that might not be immediately obvious to every reader, for example |m| to mean abs(m).

I think the best answer to this is to add non-normative examples, which are badly missing from this whole section.

This would probably require splitting of 20.1.1 into subsections. Alternatively, we could add examples to the string() function, which would have the benefit that we have validation machinery to check that the examples under a specific function are correct; we could point to these examples from 20.1.1.

Corrects/clarifies the rules for determining whether A is a subtype of B when either or both is a choice item type.

Fix #1395

We say:

Any one of the strings xsl, xml, xs, xsi, fn, math, map, array, err. This has the effect of binding that particular namespace prefix to the [reserved namespace] with which it is conventionally associated, whether or not the [native namespace bindings] contain a binding for this prefix.

Note: ...

If the namespace prefix is explicitly bound to a different namespace, for example xmlns:math="java:java.util.Math", then that binding takes precedence.

Given

<xsl:stylsheet fixed-namespaces="math" xmlns:math="java:java.util.Math">

The normative statement implies that "java" is bound to http://www.w3.org/2005/xpath-functions/math, while the note implies it is bound to "java:java.util.Math" -- which surely makes more sense. I suspect the intended reading of the normative sentence is "whether or not the [native namespace bindings] contain this binding; but it needs clarifying/correcting.

Fix #1397

We've changed constructor functions to allow arity zero, but there is no change log entry for this.

Issue #658, PR #662

In the EBNF syntax tabulation, the column width available for the RHS of productions is rather limited, leading to excessive line wrapping, especially where the entire syntax is displayed in Appendix A.

I haven't been entirely successful in sorting this out, but record my experiments here.

(a) In the CSS stylesheet qtspecs.css line 169, change the padding from 30px to 10px

(b) remove all the non-breaking spaces that are inserted after the production number and around the ::= operator. (Search for ::= in xmlspec-2016.xsl

(c) get rid of the tbody element that surrounds each tr by putting the productions in a prodgroup, achieved by adding a prodgroup element around the line <prodrecap id="BNF-Grammar-prods" ref="BNF-Grammar-prods"/> (line 43 of ebnf.xml)

Unfortunately this doesn't seem to allocate extra width to the columns, it merely creates extra whitespace to the right of the table. I can't work out why that is happening.

Another step would be to remove the final right-hand column containing comments, instead moving the comments to be under the RHS of the production. But the logic here is complex...

Finally, a pragmatic move would be to reduce the length of the longest production names, such as FunctionSignatureWithDefaults, which bloat the width of the LH column.

We say: [with the example changed for clarity]

3.3.2.2 Choice Item Types

[1] If B is a [choice item type], then A ⊆ B is true if A ⊆ M is true for some item type M among the alternatives of B.

[2] If A is a [choice item type], then A ⊆ B is true if M ⊆ B is true for every item type M among the alternatives of A.

Note:

Because an [enumeration type] is defined as a choice type of singleton enumerations, these rules have the consequence, for example, that enum("P", "Q") is a subtype of enum("P", "Q", "R").

Now, the first condition [1] doesn't hold: enum("P", "Q") is not a subtype of enum("P"), nor of enum("Q"), nor of enum("R").

But the second condition [2] does hold: enum("P") and enum("Q") are both subtypes of enum("P", "Q", "R"), under rule [1].

So the rules are far from clear when both A and B are choice item types.

I think it probably needs a combined rule.

(i) if both A and B are choice item types, then A ⊆ B is true if every item type a among the alternatives of A satisfies a ⊆ b for some item type b among the alternatives of B.

The existing two rules are really just special cases of that for singleton choices.

There are new possibilities for node tests used as XSLT patterns, for example element(p:*), element(*:local), element(p:*, T), element(*:local, T). These need to have default priorities assigned.

Fix #1391

• Changes the data model to clarify exactly what the annotations of a function item are
• Clarifies the XQuery description of what effect annotations in the query prolog have
• Changes the fn:function-annotations function to return a sequence of key value pairs in which there can be duplicate keys.

We have introduced two syntax extensions which achieve the same effect:

element(a|b) vs (element(a)|element(b))

Do we actually want both?

If we do, I would suggest that we define element(a|b) as a shorthand for (element(a)|element(b)).

Are they actually equivalent? I think it's fairly clear that they match the same items. It's less clear what the subtyping rules have to say. I think each is a subtype of the other, which means they are substitutable for each other in a function signature, but it needs a lot of digging in the spec to demonstrate this. Are they equivalent from the point of view of coercion rules? I think they probably are, because no coercion actually takes place when the required type is a node type.

So I think there's no technical bug here just a lack of clarity. It would be easier (assuming we want to retain the syntax at all) if we defined element(a|b) as a shorthand for (element(a)|element(b)).

Minor editorial issues:

• In 3.2.7.2 example 5 is incorrect -- element(xhtml:*|svg:*|mathml|*) - the last | should be :.
• In the changes section of 3.2.7.2, the element(A|B) syntax is not mentioned.

The function fn:function-annotations returns a map from QNames to values, which rather assumes that a function cannot have two annotations with the same name. But there is nothing in either the data model or the XQuery language spec that imposes this restriction.

Rewrite of stylesheet for generating keyword tests

In many cases, a compiler may detect that a fn:while-do or fn:do-until function call will not terminate. Trivial examples:

while-do((), true#0, identity#1)
while-do((), exists#1, fn($c) { $c, $c }

We should add a fixed error code that an implementation sʜᴏᴜʟᴅ (or ᴍᴀʏ?) raise when it encounters such a case. The error can be raised statically (e.g., as the result of type checks) or dynamically (when the input is not known at compile time). We could reuse XQDY0054.

In principle, the error code could also be used to reject non-terminating recursive functions:

declare function local:rec() { local:rec() };
local:rec()

declare function local:duplicate($input) {
  if(exists($input)) then $input else local:duplicate(($input, $input))
};
local:duplicate(())

let $oh := fn($my) { $my($my) }
return $oh($oh)

(: if we legalize this, see #1379 :)
declare variable $rec := fn() { $rec };
$rec()

I decided to put this in a separte PR so that it can be reviewed in isolation. I don't expect it to be controversial, exactly, but it may require adjustment.

@ndw I’ll again start with a single test case, fn-build-uri-from-parse-020:

<test-case name="fn-build-uri-from-parse-020">
    <description>Builds an example from the specification</description>
    <created by="Norm Tovey-Walsh" on="2023-03-10"/>
    <test>fn:build-uri(map {
  "uri": "ldap://[2001:db8::7]/c=GB?objectClass?one",
  "scheme": "ldap",
  "authority": "[2001:db8::7]",
  "host": "[2001:db8::7]",
  "path": "/c=GB",
  "query": "objectClass?one",
  "query-parameters": map {
    "": "objectClass?one"
  },
  "path-segments": ("", "c=GB")
})</test>
    <result>
      <assert-eq>"ldap://[2001:db8::7]/c=GB?objectClass?one"</assert-eq>
    </result>
  </test-case>

I would expect ldap://[2001:db8::7]/c=GB?=objectClass%3Fone as result, as the spec says…

If the query-parameters key exists in the map, its value must be a map. A sequence of strings is constructed from the values in the map. For each key and each value associated with that key in turn:

• If the key is the empty string, the string constructed is the value encoded with encode-for-uri.

…and encode-for-uri('?') gives us %3F. How do you handle that in your code?

Fix #1382

This has been requested multiple times before (albeit not in this repository): What about adding a positional argument to quantifier expressions? It it simple to implement, and it would allow us to do things like…

every $item at $pos in $input satisfies $item = $pos * 2

…which would be equivalent to…

every($input, fn($item, $pos) { $item = $pos * 2 })

…and much better to read than e.g.…

not((for $item at $pos in $input return $item = $pos * 2) = false())

Fix #1316

1. Quantified expressions now allow a type declaration in both XQuery and XPath
2. A type declaration now induces coercion of the supplied value to the required type

Fix #1374

§2.7.4 says:

If R has an as attribute, the SequenceType S declared by R must be a subtype of T, according to the relationship subtype(S, T) defined in [].

There is no allocated error code, I propose to allocate XTSE4040.

The rules for fn:parse-uri say:

If the fragment is the empty string, it is discarded and the fragment is the empty sequence.

However, the result of fn-parse-uri-043 contains an empty fragment string:

<test-case name="fn-parse-uri-043">
    <description>Parses a URI ending in "#"</description>
    <created by="Michael Kay" on="2023-10-27"/>
    <test>fn:parse-uri("https://qt4cg.org/specifications/xpath-functions-40/Overview.html#")</test>
    <result>
      <assert-deep-eq>map {
        "authority": "qt4cg.org",
        "fragment": "",
        "host": "qt4cg.org",
        "path": "/specifications/xpath-functions-40/Overview.html",
        "path-segments": ("", "specifications", "xpath-functions-40", "Overview.html"),
        "scheme": "https",
        "hierarchical": true(),
        "uri": "https://qt4cg.org/specifications/xpath-functions-40/Overview.html#"
        }</assert-deep-eq>
    </result>
  </test-case>

I assume the tests should be updated?

With regard to the spec, I would suggest adding the normalization rule to query parameters:

If the query is the empty string, it is discarded and the query is the empty sequence.

Fix #1320

(A speculative PR in the hope that my proposed resolution is acceptable.)

It remains a challenge (for me) to understand how the specification expects implementations to handle circular dependencies. Sorry in advance for mixing up the specification, tests and the behavior of Saxon (it basically demonstrates my confusion):

Error Codes

The definition of err:XPST0008 is:

It is a static error if an expression refers to an element name, attribute name, schema type name, namespace prefix, or variable name that is not defined in the static context, except for an ElementName in an ElementTest or an AttributeName in an AttributeTest.

The definition of err:XQDY0054 is:

It is a dynamic error if a cycle is encountered in the definition of a module’s dynamic context components, for example because of a cycle in variable declarations.

Queries

The following test case K-InternalVariablesWith-15b expects XPST0008 as error:

declare variable $var1 := $var1;
true()

I would expect XQDY0054 as error, and true() to be a valid alternative, as $var1 is indeed defined in the static context. Think of a similar query (for which I would as well expect XQDY0054 or true() to be correct results):

declare variable $var1 := $var2;
declare variable $var2 := $var1;
true()

Saxon outputs the following error for the first query…

[XPST0008] Circular definition of global variable: $var1 uses $var1.

…which is confusing in itself, as the error code and the message do not really match. The output of the second query is consistent:

[XQDY0054] Circular definition of global variable: $var2 uses $var1, which uses $var2.

Another test case that expects XQDY0054 is K-InternalVariablesWith-17a:

declare variable $var := local:func1();
declare function local:func1() { local:func2($var) };
declare function local:func2($arg2) { 1 };
true()

Again, I would expect true() to be a valid alternative, in K-InternalVariablesWith-17 (the original XQuery 1.0 test), the rational for reporting an error was…

A prolog variable having a circular dependency, by having a variable reference in a call site argument. This is an error even though the variable isn't used, because implementations cannot skip reporting static errors.

…but we now have a dynamic error.

Suggestions

1. XPST0008 should only be raised if no declaration exists for that variable in the query prolog.
2. XQDY0054 should be optional for cases in which the affected code is never evaluated.

I would like to hear some feedback on this, and I will be happy to move the corresponding questions to qt4tests and/or the Saxon bug tracker if we believe that the spec is comprehensive enough.

1. Changes the pattern syntax to allow any EQName to be used for function calls; the constraints on which functions can be called are defined outside the actual grammar.
2. Reinstate root() as a permitted function
3. Allow doc()/id() without requiring parentheses around the second function call
4. Stylesheet changes to point external spec references at the 4.0 specs rather than 3.0.

Fix #1375

The DTD defines it as a IDREF type which it clearly isn't.

This came up on stack overflow and is probable worth a few comments on.

The motivation was actually to implement something called applicative style programming, but the essence of it, is it requires operators.

stack overflow question

The simplest (to me) mechanism is the one used in Haskell (to turn a function into an operator) whereby

haskell backtick mechanism

Prelude> let concatPrint x y = putStrLn $ (++) x y
Prelude> concatPrint "a" "b"
ab
Prelude> "a" `concatPrint` "b"
ab

the 1st line defines a function the 2nd line calls the function the 3rd line uses the function as a operator by wrapping it in backticks.

I don't believe anything 'clever' happens its simply a mechanical rewriting of the expressions before its parsed. As Haskell is curried it generalises naturally to n parameters, in XPath you'd probably restrict it to two (and maybe one) parameters.

The spec says

In a [FunctionCallP], the EQName used for the function name must have local part doc, id, element-with-id, key, or root, and must use the [standard function namespace] either explicitly or implicitly.

But the grammar does not allow an EQName, it only allows:

[10]  FunctionCallP  ::=  OuterFunctionName  ....
[11]  OuterFunctionName  ::=  "doc" | "id" | "element-with-id" | "key" |  URIQualifiedName

For example the grammar does not allow fn:doc('a:xml') when fn is bound to the conventional namespace.

Also: the XSLT 3.0 grammar allows the root() pattern, but the 4.0 grammar does not.

We say (concerning map constructors)

If two or more entries have the [same key value] then a dynamic error is raised [[err:XQDY0137].

I think we should say:

if every MapKeyExpr is a Literal then this error SHOULD be reported as a static error.

• Wrong example: hash("password123" || $salt, "sha-1234567")
• The following test lacks a use="v-deep-equal-at" attribute:

deep-equal(
  $at//name[@first="Bob"], 
  $at//name[@last="Barker"],
  options := { 'items-equal': op('is') } 
)

• graphemes($crlf): Result should be char('\r') || char('\n') instead of $crlf (otherwise, test doesn’t work)
• fix fn:unix-time example: 1969-12-31T23:59:59.999Z (thanks @johnlumley)
• TODO for fn:replicate can be dropped
• fn:parse-uri: Remove example for FOUR0002.

…to be continued

I would suggest dropping https://qt4cg.org/specifications/xpath-functions-40/Overview.html#ERRFORG0013 and replacing it with a type error, XPTY0004. This way, it can be perfomed during the type checks (which may also be the best solution if we define options as records).

I’ll be glad to create the PR for the spec and the tests.

As suggested by Pieter Lamers in https://app.slack.com/client/T011VK9115Z/C01GVC3JLHE, we could alternatively allow curly braces in case clauses:

switch($n) {
  case 1  { 'one' }
  case 2  { 'two' }
  default { 'dunno' }
}

This would be in aligment with the new syntax alternative for if expressions: if($n) { 'one' } else { 'two '}.

Closes #1369

…no need to update any tests.

We could rename the new $rounding-mode parameter of fn:round to $mode (analogous to $precision, which isn’t $rounding-precision).

If it sounds reasonable, it’s very probable I’d be able to take charge of this.

1. The test generated for fn:seconds() is incorrect (can't handle required type xs:decimal?)
2. An incorrect argument is supplied for fn:pin()

Fix #1321

Fixed as suggested in the issue. I have added a list of tokens that can appear at the start of a RelativePathExpr, produced by analyzing the grammar using a custom stylesheet leading-tokens.xsl which is available for reuse, but not integrated into the build.

We could pick up the ++ and ** operators from Invisible XML for defining "a sequence of [one|zero] or more Xs separated by Ys".

So for example

Annotation ::= "%"  EQName ("("  AnnotationValue ("," AnnotationValue)* ")")?

changes to

Annotation ::= "%"  EQName ("("  (AnnotationValue ++ ",") ")")?

and

TypedFunctionTest ::= ("function" | "fn") "(" (SequenceType ("," SequenceType)*)? ")" "as" SequenceType

becomes

TypedFunctionTest ::= ("function" | "fn") "(" (SequenceType ** ",") ")" "as" SequenceType

Note that there is an impact on anyone who uses the machine-readable XML version of our grammar. However, the current version could easily be generated by preprocessing.

There are several reasons it would be good to have an occurrence indicator meaning "exactly one".

1. It's generally a good idea if there is explicit syntax that's equivalent to the default: it enables you to write code that is more explicit about the fact that you have chosen the default option; it makes code generation easier.
2. Sometimes you need to write an ItemType in parentheses to avoid a following ? or * being interpreted as an occurrence indicator. Having an explicit occurrence indicator for "exactly one" would avoid this need.
3. It makes it clear that you are writing a SequenceType, not an ItemType.

I don't expect this to be used very often, but I think it would be useful to allow it. I propose using U+00B9, superscript 1.

For example, as="xs:integer¹"

We can now write:

function($a, $b) as xs:integer¹ *

to mean a sequence of functions each returning a single integer; while writing

array(xs:integer¹ )

just serves to remind the reader that the array entries must be single integers.

Fix #1314

I find the callback/fallback arguments to map:get and array:get rather unsatisfactory. They complicate the specification, and the use cases and examples are tenuous. I don't think they offer a great deal of convenience over alternative ways of achieving the same effect.

I would like to propose scrapping these arguments, reverting to the 3.1 specification, and adding a new pair of functions map:try-get() and array:try-get() with a return type of record(found as xs:boolean, value? as item()*).

The specification for map:try-get($map, $key) is

if (map:contains($map, $key))
then {"found":true(), "value":map:get($map, $key)}
else {"found":false()}

The specification for array:try-get($array, $index) is

if ($index = 1 to array:size($array))
then {"found":true(), "value":array:get($map, $index)}
else {"found":false()}

Though it would probably be better to define it the other way around, that is define *:get in terms of *:try-get.

The name try-get comes from C#. I'm not immensely enthusiastic about it, but it will get some name recognition. I would probably prefer test-get.

One of the aims, of course, is to enable you to find whether a value exists and get the value in a single call to the map (so the key only gets hashed once, for example). This benefit will only materialise if ?found and ?value are implemented without requiring another full map lookup, that is if access to fixed fields in simple record types is optimised. That's a challenge we leave to implementors. We're not quite there yet in Saxon - we do have a map implementation in which the fields occupy fixed slot positions, but the static inferencing to reference fields by slot number isn't quite there yet.

This (should!) fix the diffs

Fix #1337

I was half-expecting this change to have repercussions, but in practice it seems very clean.

There are probably more places that should change, e.g. use "xs:boolean item" rather than "xs:boolean value", but we can deal with them as we come to them.

Fix #1348

Makes some simplifications to the grammar rules, including some of those suggested.

I've avoided removing productions that are referenced by name in the prose of the spec; these names provide a useful handle to relate the syntax to the semantics.

I would love to introduce BNF for a comma-separated list. Invisible XML uses

XX ** ","

for zero or more occurrences of XX, separated by ",", and

XX ++ ","

for one or more occurrences of XX, separated by ",", and

Fix #1346

Closes #959

In previous versions of the XQFO drafts, it was possible to differentiate between NEW and UPDATED functions. It would be nice if this distinction could be preserved:

Is there any good reason for the rule that the name of a private function or variable declared in a library module must be in the target namespace of the module? I think it makes sense to allow:

• private functions to be in any non-reserved namespace
• private variables to be in any non-reserved namespace or in no namespace

The reason I ask is that I'm looking at what the rule should be for item type declarations.

This PR does the following:

• Renames MapTest, ArrayTest, RecordTest, FunctionTest to MapType, etc (suggested in #1351 and elsewhere)
• Changes the XQuery item type declaration syntax to "declare type NNNN" rather than "declare item type" or "declare item-type", and fixes examples accordingly (also in #1351)
• Provides a separate syntax for declaring record types (as proposed in issue #1277). This declaration creates both a named item type and a constructor function for items of that type, with provision for default values. Only this kind of record type can be recursive.

I haven't yet taken on board the suggestion of changing the data model so that records are maps with a type annotation. I'm not totally opposed to the idea, but it's a significant change that needs more exploration, in particular the impact on all the map-related functions.

Fix #1351 Fix #1277

Follow-up work is needed to make equivalent changes for XSLT.

Fix #1202

Since the changes are purely editorial but are likely to cause conflicts with technical changes, I would request an expedited review.

Fix #1347

Fix #1350

3.2.8.4 Recursive Record Tests presents the following declaration:

declare item type my:list as record(value as item()*, next? as my:list);

It should probably be declare item-type.

Personally, I would prefer to just use declare type (I guess we do not plan to introduce “sequence type” declarations?).

The $options parameter of unparsed-text-available() should have the same item type as in unparsed-text(), that is either a string or a map.

In the Data Model we are missing an important concept - the concept of Nothing.

It has been believed that the type item()* is sufficient to express all results that can be produced or expressed in the evaluation of an XPath expression.

In reality, there are XPath expressions whose result cannot be expressed unambiguously. Consider:

array:values([ (), 1, (2 to 4), [ 5 ] ])

According to the F&O 4.0 specification, the result must be:

(1, 2, 3, 4, [ 5 ])

And here the value of the first array member - () - is not returned.

Thus, in the general case, array:size($ar) is not equal to count(array:values($ar)),

and array:size($ar) - count(array:values($ar)) can be any non-negative integer.

At present there isn't a way in XPath to represent the lack of value. The empty sequence () is a value and thus using it to represent the lack of value is wrong. Also, if the lack of value is represented by () then this is completely lost/destroyed/vanishes when concatenated with other sequences.

The expression:

deep-equal([ ], [ () ])

evaluates to false(), because the first array-argument has no members (contains nothing), and the second array-argument has one member - the value ().

Thus, () is not nothing.

It is time for us to be able to represent nothing in an explicit way.

We need a type nothing, or maybe xs:nothing that tells us that an expression may not evaluate to any value, even not to ().

Then we will be able to express correctly the type of a lookup expression as item()* | nothing

I’m a humble user of our grammar rules, and I’m definitely not an expert when it comes to their definition (my main obstacle is that changes to the XQuery grammar rules need to be compatible with the XPath and possibly XSLT grammars).

What made it difficult for me to read them in the past were the numerous redundancies (with some of them attached). Is this just “history”, or are there particular reasons for preserving or even enforcing redundance? I noticed that, sometimes, symbol names are used in the prose, but I failed to detect any reasonable pattern.

Do we believe it would be helpful to clean up the grammar rules, or does it rather feel out of scope?

# currently
ParamWithDefault ::="$" EQName TypeDeclaration? (":=" StandaloneExpr)?
Param ::= "$" EQName TypeDeclaration?
# could be
ParamWithDefault ::= Param (":=" StandaloneExpr)?`
Param ::= "$" EQName TypeDeclaration?

# currently
SchemaAttributeTest ::= "schema-attribute" "(" AttributeDeclaration ")" 
AttributeDeclaration ::= AttributeName 
AttributeName ::= EQName 
# could be
SchemaAttributeTest ::= "schema-attribute" "(" EQName ")" 

# currently
WindowVars ::= ("$" CurrentItem)? PositionalVar? ("previous" "$" PreviousItem)? ("next" "$" NextItem)?
CurrentItem ::= EQName
PreviousItem ::= EQName
NextItem ::= EQName
LetBinding ::= "$" VarName TypeDeclaration? ":=" StandaloneExpr
PositionalVar ::= "at" "$" VarName
VarName ::= EQName
VarRef ::= "$" VarName
# could be
WindowVars ::= ("$" EQName)? PositionalVar? ("previous" Var)? ("next" Var)?
LetBinding ::= Var TypeDeclaration? ":=" StandaloneExpr
PositionalVar ::= "at" Var
VarRef ::= Var
Var := "$" EQName

# currently
MapConstructorEntry ::= MapKeyExpr ":" MapValueExpr
MapKeyExpr ::= ExprSingle
MapValueExpr ::= StandaloneExpr
# could be
MapConstructorEntry ::= ExprSingle ":" StandaloneExpr

# currently
ForwardAxis ::= ("child" "::") | ("descendant" "::") | ("attribute" "::") | ("self" "::") |
  ("descendant-or-self" "::") | ("following-sibling" "::") | ("following" "::")
# could be
ForwardAxis ::= ("attribute" | "child" | "descendant" | "descendant-or-self" |
  "following" | "following-sibling" | "self") "::"

# currently
CompNamespaceConstructor ::= "namespace" (Prefix | EnclosedPrefixExpr) EnclosedURIExpr
Prefix ::= NCName
EnclosedPrefixExpr ::= EnclosedExpr
# could be
CompNamespaceConstructor ::= "namespace" (NCName | EnclosedExpr) EnclosedURIExpr

# currently
Argument ::= StandaloneExpr | ArgumentPlaceholder
ArgumentPlaceholder ::= "?"
# could be
Argument ::= StandaloneExpr | "?"

…and so on.

PS: If we tweak the grammar, I would propose to rename ExprSingle to SingleExpr.

Recently, we added an escape-solidus option to fn:serialize(). That option should apply to fn:xml-to-json() as well.

It's not immediately obvious to me if there are other serialize options that should apply as well.

In the rules section of fn:format-number, in the para starting "In the table", the word "rendition" is in the wrong font (use term markup).

In the options table, the notation xs:string (: matching '.(:.*)?' :) is flawed because a comment cannot contain the characters (:. Simplest answer to this is to just write xs:string matching '.(:.*)?' - it's intended for the human reader after all, not for machine execution.

I have applied the new bare brace grammar rules to two existing projects, and (if I’m correct) all of the the following constructs cannot be parsed anymore (with the order reflecting my sense of urgency):

1. Simple maps: $data ! {}
2. Sequence arrow: {} => map:size()
3. Predicates: { 'x': 1 }[map:keys(.) = 'x']
4. Return clause: let $x := ... return {}
5. Paths: <a/>/{}
6. Instance of: {} instance of record(x)
7. If expression: if($x) then { $x: $y } else {}

I think we should try hard to tweak the grammar for some of these.

I didn’t manage to construct an ambiguous example with return and {}; could someone help me?

Related: #1309

Fix #1343

Since XQuery 3.0, the effect of the static typing feature has been almost entirely implementation defined. Since there is no interoperability when this feature is in effect, there seems very little point leaving it as an optional feature in the spec. In any case, I think user experience of processors that attempt to implement the static typing feature has been rather negative, and most of the processors that implemented it have been left to languish and are unlikely to be upgraded to 4.0.

The "declare ordering mode" declaration, and the ordered{} and unordered{} declarations are retained for compatibility, but are deprecated and no longer have any effect.

Fix #1339

The $position argument, passed to the $action-function-argument of the folds is unnecessary and artificial:

1. The addition of this argument resulted from the automatic adding this position-aware $action-function-argument to all functions processing sequences and producing result based on their values.
2. In doing so, no further analysis was made on the specifics of the fold functions.
3. Though this issue has been raised again and again for months, no single and meaningful use-case has been provided.
4. This addition departs away from the original meaning of folds as has been well-established by the developers community. Whether or not one needs to produce a variety of folds, such as sum, min, max, average, product, all / none / some / any - in all these cases the position of the individual items does not matter.
5. One bad consequence of this change is that it makes it more difficult for the reader to grasp the meaning of a particular fold-function, and even to wonder if the spec talks about the same fold functions that the reader thought he knew well.
6. This change results in unnecessarily complex documentation and testing.
7. Users have expressed their dismay over the resulting complexity. To quote @benibela: "Too many variables make the code hard too read. And the implementation becomes slow, when it has to handle too many arguments. Especially with function coercion adding further type checks" . And @michaelhkay himself: "I'm inclined to propose dropping the position argument for both fold and scan. It complicates the specification and the use cases are unconvincing. I believe it has been incorrectly specified (for fold-left, the first time $action is called, the value supplied for $pos is 2, whereas for fold-right it is count($input)-1; and the "Error conditions" section talks of $action being applied to 2 arguments). For the -right forms in particular, the semantics are mind-bending enough without introducing this complication."
8. It is very easy to make an accidental mistake and pass a 2-arg. $action function when a 3-arg. function was meant (or the other way around).
9. The giant think-tank of Microsoft gives us a good example of a better solution. They never added position-aware overloads to any folds or fold related methods of the Enumerable class. All the following methods do not have position-aware action-function arguments:

• Aggregate

• All

• Any

• Average

• First

• FirstOrDefault

• Last

• LastOrDefault

• Max

• MaxBy

• Min

• MinBy

• Zip

  This is not an accidental mistake, as Microsoft added to other Enumerable methods overloads that do require position-aware $action-function arguments:

• Select

• SelectMany

• SkipWhile

• TakeWhile

• Where

10. Last and probably most important: @michaelhkay gave us a general and an elegant and very readable way of expressing any fold that needs positional information as a 2-step operation where the 1st step creates a map with entries: {"position": $input[$pos]} and then a fold-operation that only has a non-position-aware $action-function argument, and has this map as input.

Proposed solution:

1. Leave the folds unchanged - in order to preserve their original, established meaning and avoid introducing inadequate complexity.
2. If deemed really necessary, define separate functions that can take as parameter a position-aware $action-function.

It would be nice to bring XSLT, XPath, and XQuery into line here.

The current state of play seems to be:

XQuery: the namespace axis is not supported. Namespace nodes can be constructed, but they exist only as detached orphans; they can never be attached to a parent element.

XPath: the namespace axis is deprecated and support is optional. There is no mechanism for constructing namespace nodes.

XSLT: the namespace axis is mandatory. Namespace nodes can be constructed and can be attached to elements.

I believe that the only reason for the differences is that XQuery implementors were concerned that it would be difficult to implement namespace nodes efficiently. I think XSLT has clearly demonstrated that this concern is unjustified.

However, there are implementation complexities, primarily around the fact that namespace nodes have identity and parentage, so if a namespace is declared on a root element, then every element in the document has a namespace node for this namespace, and these have distinct identity. To implement this efficiently, the implementation has to instantiate namespace nodes lazily on demand, and then has to ensure that if the "same" namespace node is instantiated again, it has the same "identity".

I suggest a solution along the following lines, applied to all three languages:

(a) the namespace axis is supported and delivers namespace nodes

(b) operations that depend on the ordering, identity, or parentage of namespace nodes are deprecated and implementation-defined.

(c) the data model says that the in-scope namespaces of an element are in the form of a (prefix, URI) map. The semantics of the namespace axis are described in terms of constructing transient namespace nodes from this map.

Is there any evidence that unordered mode is useful, or that any implementations actually take note of it (by delivering a different result if unordered mode is set)?

If not, could we drop it? I would suggest continuing to recognize the syntax, marking it deprecated, and saying it has no effect.

It's not doing a great deal of harm, but there are a lot of places where our examples in the spec assume ordered mode, but the examples don't explicitly call out this assumption. Similarly, a great many QT3 test cases would fail on an implementation that sets unordered mode by default.

With version 4.0, we are adding a lot of promising and powerful new map and array features. This is a big step forward, compared to the obvious limitations of 3.1.

Some aspects of the 3.1 design have made it difficult (or impossible) to fully adjust array and maps, but (in my opinion) the old overall concept was impressively consistent – and it is definitely a big challenge to achieve a 4.0 design that is not too fragmented.

To me, this becomes particularly evident in the case of arrays. The following example sums up the items of all members of an array. For the cumbersome 3.1 solution…

for $pos in 1 to array:size($array)
return sum($array($pos))

…we now have at least several (roughly?) equivalent options to do this; for example…

1. for member $m in $array return sum($m)
2. array:members($array) ! sum(?value)
3. $array?entry::* ! sum(?value)
4. $array?value::* ! sum(.)

…which is great – but the downside is that we have introduced a terminological jungle. The examples above could imply that:

• for 1., an array member is a sequence (which it indeed is);
• for 2., an array member is a map;
• for 3., an array has entries (but there is no array:entries);
• for 4., an array has values (which is true, but array:value returns a different structure).

Next, with the current proposals, $array:content::1 gives us the sequence-concatenated version of the first member of an array. Similar observations can be made with maps: map:entries($map) returns singleton maps, whereas $map?entry::* is actually equivalent to map:pairs.

The fundamental obstacle are clear have already been discussed a lot, but I think that with each new concept, we should try really hard not to blur terminology, and work with terms that users can assign to the underlying concepts without too much guessing or trial’n’error.

My general suggestions would be to…

1. align the new lookup terminology and the builtin functions, and
2. omit, rename or drop builtin functions that do not rely on the existing or arising terminology.

My concrete proposals:

1. As we already have map:pairs, $map-or-array?entry::* should become $map-or-array?pair::*, and we should add a array:pairs function, and probably array:of-pairs (see #832). We shouldn’t do it the other way round and rename map:pairs to map:entries, as the existing map:entry function returns a singleton map.
2. If we keep calling the sequence-concatenated result “content”, we should include it in the definition of sequence-concatenation. In addition, (array|map):values should be renamed to (array|map):contents (see #1179).
3. Due to the existence of array:value::*, we should make clear what an “array value” is, how it it positions itself in relation to an “array member”, and we should add map:values and array:values for equivalent results.
4. Due to the existence of array:key::*, we should add a array:keys function (which returns a dense integer range). 1 to array:size($array) could then be written as array:keys($array).
5. As we have map:entries and map:merge, we could add equivalent array:entries and array:merge functions.
6. I would suggest dropping array:members/array:of-members in favor of either array:split/array:join, array:pairs/array:of-pairs (see 1.) or array:entries/array:merge (see 5). I really believe that an “array member“ should not be a map; an “array pair” or ”array entry” certainly could.