The current "?>" method call operator is ugly, difficult to read, difficult to find and understand. We have much better alternatives.

One obvious big improvement is to have ==> .

This is:

1. Readable.
2. Distinctly distinguishable.
3. Understandable and intuitive for anyone who has used an OOP language (C++, C#, Java)
4. Expresses the similarity with the => operator. The same way => provide the LHS item as the first argument of the RHS function, the similar, and extended, in appearance ==> provides the LHS map/object as an implicit argument to the RHS method.
5. Because of 1, 2, 3, and 4 above, very little additional learning and understanding effort is required from the XPath user.

Proposed action: Replace the ugly, difficult to read, difficult to find and understand operator "?>" with ==> .

StringInterpolation currently defines a two-character token, curly right brace + backtick, to follow Expr as a terminator:

StringInterpolation ::= "`{" Expr? "}`"

On other occasions, Expr is followed by a single right curly brace:

EnclosedExpr ::= "{" Expr? "}"

The following applies to tokenization (the "longest-token" rule):

If the current position is not the end of the input, then return the longest literal terminal or variable terminal that can be matched starting at the current position, regardless whether this terminal is valid at this point in the grammar.”

My concern is that input like

<a>{42}`</a>

is going to be mis-tokenized under the longest-token rule: after 42, the next (longest) token is the two-character StringInterpolation terminator, which however is not a valid terminator of the EnclosedExpr serving as CommonContent of the direct element constructor.

Proposed fix

My proposal is to replace the two-character tokens that introduce and terminate StringInterpolation with single backticks around an EnclosedExpr with no intervening whitespace:

StringInterpolation ::= "`" EnclosedExpr "`"   /* ws: explicit */

This replaces both of the two-character delimiters of StringInterpolation, while still describing the intended language, but without causing the longest-token rule to produce a token that cannot be handled afterwards.

Fix #2139

hexBinary and base64Binary become mutually comparable under all comparison operators: which may affect backward compatibility.

Fix #2166

In reverting many of the features previously added to lookup expressions (for example deep lookup and lookup modifiers) we seem to have accidentally lost text that actually defines what the different key specifiers mean; we're left with lots of examples but no actual specification.

I was reading to see what the current spec says about array bound checking: it appears to say nothing.

Related: Michael’s observation in https://github.com/qt4cg/qtspecs/issues/2163#issuecomment-3185618064.

The current spec says for treat:

XPath 4.0 provides an expression called treat that can be used to modify the static type of its operand.

It further mentions the static analysis phase, which has been removed from the specs; maybe we should remove these references.

There are hardly any uses of the expression in the current spec. One is for Absolute Path Expressions:

An expression of the form /PP (that is, a path expression with a leading /) is treated as an abbreviation for the expression self::gnode()/(fn:root(.) treat as (document-node()|jnode())/PP.

The use seems confusing, as in many cases self::gnode() can only be evaluated at runtime. Maybe we could rewrite it to a variant that coerces the node? It would generally be easier for optimizers to rewrite paths when there is no need to differentiate between treat and coercion (and I have never seen code that catches XPDY0050).

Indeed I think it would be helpful to have a coerce to expression, even if people will rarely use it explicitly. It would allow us to remove all remaining uses of treat as (except, of course, for the expression itself), and we could simplify various examples that use variable declarations only for coercing values.

In f2e1f48, fn:parse-csv was changed to return an empty sequence, when its first argument is an empty sequence. This is however not reflected in the function's return type, which is here changed to parsed-csv-structure-record?.

I propose using =?> for method calls rather than ?>

(a) I lilke the association with => to call a function item with an implicit first argument; =?> combines selection of an item from a map (?) with function invocation (=>)

(b) ?>, while technically unambiguous, smells strongly of XML processing instructions

Expands the explanation of the example of method chaining

Adopted from https://github.com/qt4cg/qtspecs/issues/2136#issuecomment-3135426200:

The feedback for U+00D7 (MULTIPLICATION SIGN, ×) and U+00F7 (DIVISION SIGN, ÷) that we got so far was not very positive either, so I would suggest dropping also those operators; they offer no real added value.

This is a first draft of a PR, giving the data model changes only, for a change to the JNode model affecting maps and arrays with sequence-valued entries. A sequence of length 2 or more now has children representing the items in the sequence. Although there is still an asymmetry between sequences of length 1 and longer sequences, it is more manageable than i the previous model.

For those who have not stumbled upon JSONiq yet, I am adding some introductory links:

• https://www.jsoniq.org/docs/JSONiq-usecases/html-single/
• https://www.jsoniq.org/docs/Introduction_to_JSONiq/html/
• https://www.jsoniq.org/docs/JSONiqExtensionToXQuery/html-single/index.html

JSONiq has been designed as a query and update language for JSON data. Its first versions were based on XQuery. Due to its similarities, it may give us some good inspirations for traversing and modifying JNodes.

RumbleDB is a current implementation maintained by Ghislain Fourny (@ghislainfourny).

Fix #2157

In the F&O reference to UTS#10 we say incorrectly that: The current version is 9.0.0, dated 2016-05-18.

Similarly for UTS#35 we say incorrectly: The current version is 29, dated 2016-03-15.

In §5.5, functions based on substring matching, we say

"In the definitions below, we refer to the terms match and minimal match as defined in definitions DS2 and DS4 of [[UTS #10]]."

It's not made clear what "the definitions below" is referring to: the terms "match" and "minimal match" are actually used in the rules of the individual functions.

The parenthetical sentence (“collation unit” is equivalent to "collation element" as defined in [[UTS #10]]) is not very elegantly expressed.

Addresses part of issue #2092.

While the function family map:pair, map:of-pairs, and map:pairs can be handy, they are not necessary, especially now that we have JNodes. They are also very easily user-written:

map:pair => {'key': $key, 'value': $value}

map:of-pairs => map:build($pairs, fn{?key}, fn{?value})

map:pairs => map:for-each($map, fn($k, $v){ {'key': $k, 'value': $v })

On the grounds that we should avoid providing multiple ways of solving the same problem, I propose dropping these three functions.

Note: in some ways I would have preferred to drop the alternative trio map:entry, map:entries, and map:merge; but two of these are present in the 3.1 specification.

Fix #2150 Fix #2010

Fix #2152

Revises the rules for enumeration types: they are now structural subtypes of xs:string rather than nominative subtypes. The main effect is that "x" instance of enum("x") is now true. The change is motivated by use cases involving XSLT pattern matching, where strict "instance of" matching is required, with no coercion.

"Tolkein" isn't an actual instance of enum("Tolkein"), it's only coercible to that type, and when types are used in paths it has to be an actual instance. I think we need to fix that.

Originally posted by @michaelhkay in #2150

It seems strange that there is no way to create a value that is an instance of a singleton enumeration type. Only casting (and annotation, which is a kind of casting too) is available.

On the other hand:

let $x as enum("foo") := "foo"
return ( ()
  , $x instance of enum("foo")
  , $x instance of xs:string
  , atomic-equal($x, "foo")
)
(: true(), true(), true() :)

This means that "foo" should be an instance of enum("foo"), and then enum("foo") is a subtype of xs:string.

And the following is unclear (from 3.2.6 Enumeration Types):

• It follows from these rules that an atomic item will only satisfy an instance of test if it has the correct type annotation, and this typically requires an explicit cast. So the expression "red" instance of enum("red", "green", "blue") returns false, while "red" cast as enum("red") instance of enum("red", "green", "blue") returns true.

Probably, a more narrow reason is that a singleton enumeration type is an "anonymous atomic type derived from xs:string by restriction using an enumeration facet" that permits only one value. Yes, this makes type checking for an enum more complex, but seems not more complex than casting.

Anyway, is it possible to make any instance of xs:string also an instance of the corresponding singleton enumeration type? (that is, essentially make it so that this casting happens "hidden", if required).

The usefulness of enum() types is limited by the fact that the string "x" is not actually an instance of enum("x"), it is only coercible to that type. This means that in contexts where strict type matching is required (for example, in XSLT patterns), either (a) you can't use enum() the way you would like, or (b) you use it and fail to understand why it fails.

This PR simply moves the section on Patterns to a more logical place in the XSLT specification. Unless anyone objects, I will merge the PR without waiting for group approval, so that I can use the result as a baseline for further work on patterns and templates, hopefully giving a better diff baseline.

Supersedes #1776.

Part of the motivation for introducing JNodes was to make rule-based recursive-descent transformation of JSON structures much easier. This issue addresses part of that capability, namely defining patterns that match JNodes (and perhaps improving the patterns that match maps and arrays).

In general I think the patterns that match JNodes should be distinct from the patterns that match XNodes; although we have unified path expressions so that a/b can select either an XNode or a JNode, I think there would be too much scope for confusion if match="a/b" were able to match a JNode as well as an XNode.

My first idea would be to allow the syntax match="jnode(a)" for a template rule that matches JNodes having a selector property of "a", similarly jnode(a/b), jnode(a//b), jnode(a/*/b), jnode(a[x="c"]) with semantics defined in much the same way.

But there's a question how this relates to type patterns. With type patterns, we can already do match="type(jnode(record(Author, Title, *)))" which matches a JNode whose content is of type record(Author, Title, *). Where syntactically possible we allow type patterns to be abbreviated, so this would become match="jnode(record(Author, Title, *))" which conflicts with the above.

An analogy with element(N, T) might suggest match="jnode(K, V)" where K constrains the selector property of the JNode, and V constrains its content property. So we might have match="jnode(books, array(record(Author, Title, *)))" to match a JNode whose selector is "books" and whose content is of type array(record(Author, Title, *)).

At the same time, while matching maps by a type such as match="record(Author, Title, *))" works well, I find that this is often accompatied by a predicate so it becomes match="record(Author, Title, *))[Author='Tolkein']". It would be nice to express this more concisely and readably perhaps as match="record(Author[.='Tolkein'], Title, *))"

Fix #2100

The (rather old) test case K2-BaseURIFunc-29 indicates that invalid URIs may result in an error:

<test-case name="K2-BaseURIFunc-29">
  <description> Use an URI in an xml:base element that is a valid URI, but an invalid HTTP URL. 
    Since implementations aren't required to validate specific schemes but allowed to, 
    this may either raise an error or return the URI. 
  </description>
  <created by="Frans Englich" on="2007-11-26"/>
  <dependency type="spec" value="XQ10+"/>
  <test><![CDATA[let $i := fn:base-uri(<anElement xml:base="http:\\example.com\\examples">Element content</anElement>) 
    return $i eq "http:\\example.com\\examples" or empty($i)]]></test>
  <result>
    <assert-true/>
  </result>
</test-case>

I raise this issue in the qtspecs repository as I wondered whether we should clarify how invalid URIs are to be handled by fn:base-uri.

If it’s the test that is misleading, I will be glad to correct the comment, or add an error code.

Although issue #2143 envisaged redefining method calls in terms of JNodes, this PR takes a different approach.

The "magic" performed by the lookup operator when the entry in a map is annotated %method is dropped. Instead we have a new operator ?> which is essentially defined as a macro: in simple cases $map ?> method (X) is defined to be essentially an abbreviation for ($map ? method)($map, X).

I have used the operator ?> suggested by Christian, but in some ways I prefer the operator we had originally, =?>, because (a) there is a stronger analogy with =>, and (b) ?> brings up images of XML syntax for processing instructions.

The grammar rules for StringTemplate are as follows:

StringTemplate              ::=  "`" (StringTemplateFixedPart | StringTemplateVariablePart)* "`"
                                                                                      /* ws: explicit */
StringTemplateFixedPart     ::=  ((Char - ('{' | '}' | '`')) | "{{" | "}}" | "``")*
                                                                                      /* ws: explicit */
StringTemplateVariablePart  ::=  EnclosedExpr
                                                                                      /* ws: explicit */

But StringTemplateFixedPart should not be allowed as a zero-length token, because this is causing an ambiguity: the input `` currently can be parsed as any of

<StringTemplate>`<StringTemplateFixedPart/>`</StringTemplate>

<StringTemplate>`<StringTemplateFixedPart/><StringTemplateFixedPart/>`</StringTemplate>

<StringTemplate>`<StringTemplateFixedPart/><StringTemplateFixedPart/><StringTemplateFixedPart/>`</StringTemplate>

and so on.

In order to ensure an unambiguous result, StringTemplateFixedPart should be required to consist of at least one character. Also the /* ws: explicit */ on StringTemplateVariablePart is superfluous. The grammar rules thus should be changed to:

StringTemplate              ::=  "`" (StringTemplateFixedPart | StringTemplateVariablePart)* "`"
                                                                                      /* ws: explicit */
StringTemplateFixedPart     ::=  ((Char - ('{' | '}' | '`')) | "{{" | "}}" | "``")+
                                                                                      /* ws: explicit */
StringTemplateVariablePart  ::=  EnclosedExpr

Production StringInterpolation currently does not allow implicit whitespace:

StringInterpolation ::= "`{"  Expr?  "}`"
                                                                         /* ws: explicit */

But this is likely not intended - all examples in the spec do have whitespace adjacent to the braces.

This change thus removes /* ws: explicit */ in order to allow implicit whitespace.

I propose changing the mechanism for invoking methods to take advantage of JNodes.

Instead of the current magic rule for the "?" operator, we move the magic to the rules for dynamic function calls: in a dynamic function call F(X, Y), if the value of F is a JNode J whose content property is a function item annotated with %method, then the function body is executed with the parent of J (that is, the containing map or array) as the context value. It seems much cleaner semantics to make this a rule for dynamic function calls rather than for map lookup.

The downside is that the call syntax now would become ($rectangle/area)() rather than $rectangle?area(). Unfortunately $rectangle/area() parses as $rectangle/(area()). So we might want to invent some better syntax.

1. Moved all the processor comments to the end; this avoids having a comment before <!DOCTYPE HTML> which is frowned upon because ... reasons
2. The XSLT stylesheet was adding links for sections, but so was the main stylesheet, so they were coming out nested.
3. Don't attempt to link to functions or elements inside titles. (This also results in nested links)
4. Attempt to "unwrap" <p> elements around things that can't be inside a <p>, like various sorts of lists. It's a bit ugly, but it makes for much cleaner HTML.

I'm just going to merge this because there's no practical way to see the consequences in the PR.

Apologies in advance that this will introduce some spurious diffs. I think those will go away after the build finishes and after you've rebased your PRs on the new stylesheets.

I have no idea why the DTD allows <p> inside <p> but I assume this is a markup error and not intentional.

With great appreciation to the fine folks at DeltaXignia!

It seems confusing to me that deep-equal & atomic-equal return different results than eq for binary types:

let $hex := xs:hexBinary(''), $base64 := xs:base64Binary('')
return (
  (: false :) deep-equal($hex, $base64),
  (: false :) atomic-equal($hex, $base64),
  (: true  :) $hex eq $base64
)

The rules say:

fn:deep-equal

If both $i1 and $i2 are instances of xs:hexBinary or xs:base64Binary, $i1 eq $i2 returns true.

This can be interpreted in two ways, but it seems to mean that $i1 and $i2 need to have the same type?

fn:atomic-equal

One of the following conditions is true: $value1 and $value2 are both instances of xs:hexBinary. $value1 and $value2 are both instances of xs:base64Binary.

op:binary-equal

op:binary-equal(
  $value1	as (xs:hexBinary | xs:base64Binary),	
  $value2	as (xs:hexBinary | xs:base64Binary)	
) as xs:boolean

The function returns true if $value1 and $value2 are of the same length, measured in binary octets, and contain the same octets in the same order. Otherwise, it returns false.

As atomic-equal(xs:double(3), xs:float(3)) returns true, I would also expect true for binary items with the same contents.

Related (for numbers): #986

It is not currently possible to write a NodeTest as (for example) child::type(xs:string | xs:integer). As a consequence of the way the grammar is defined, two pairs of parentheses are needed: child::type((xs:string | xs:integer))

It would be easy enough to fix this usability glitch.

Fix #2136

The option of using full-width angle brackets doesn't seem to have attracted great enthusiasm, and now that we have the precedes and follows operators, I suggest we drop them. Nearly all cases of plain < and <= can be replaced with lt and le.

One of the problems with using non-ASCII characters is not just that it's hard to type them, it's also quite hard to recognise them by their appearance. There are so many characters that look a bit like less-than and greater-than symbols.

Expands on the let binding example

Revised; closes #1996

Note: depends on #2115 because of terminology changes.

Fix #2132

In §2.4.5 we introduced the concept of guarded expressions, and included the rules

• In an and expression, the second operand is guarded by the value of the first operand being true.
• In an or expression, the second operand is guarded by the value of the first operand being false.

This change is not mentioned in 4.11 Logical Expressions. For example, this still says

The order in which the operands of a logical expression are evaluated is [implementation-dependent]

and the truth tables in 4.11 are unchanged from 3.1.

(We have also introduced defined terms "and expression" and "or expression", and should use them here).

My understanding of the new rule for guarded expressions is that with (A and B), if A is false then the result is false even if B raises an error; this is not what the truth table says.

Currently the spec says:

The expression is supplied with two variables: $group is set to the contents of the current group being constructed, and $next is the next item in the population.

1. Do these supplied variables shadow the user-defined variable of the same name? (probably they should, but it is not mentioned) For example:

  <xsl:variable name="group" select="(1,2,3)"/>  
  <xsl:for-each-group select="$input" split-when=" $next = $group "> ... </xsl:for-each-group>

Maybe these variables should be in the fn: namespace?

2. To access the current grouping key and current group the functions fn:current-grouping-key and fn:current-group are used. What is the rationale of introducing variables instead of functions to access the group and item in split-when?

The operators << and >>, in my opinion, are poorly known, and challenging for developers working in XSLT. Many punctuation-based operators have aliases in ordinary-language quasi-equivalents, but << and >> lack any ordinary verbal equivalents, and break this principle.

The attached proposal offers to make precedes a keyword equivalent to << and follows a keyword equivalent to >>. This means that //title[. << following-sibling::isbn[1]] can now be expressed as //title[. precedes following-sibling::isbn[1]]

Renames the function fn:jnode as fn:jtree, and the item type jnode-type() as jnode()

Fix #2099

In principle, there's no reason why JTrees shouldn't be streamable. However, it's an immense amount of work both for the specification and for an implementation, so I intend to rule it out.

That then leaves the question about deciding streamability in the case of constructs that could be processing JNodes.

I think we might be able to define a rule something like: "if a template rule (etc.) is declared with streamable="yes", this amounts to a declaration that it will only be used to process XNodes, even if without this declaration it would also be capable of processing JNodes."

The details depend on other aspects of how we define template rule processing and pattern matching for JNodes. At present I'm inclined to say that the pattern syntax for matching JNodes should be distinct from that for matching XNodes, so that a template can only match one or the other, never both.

With the introduction of JNodes, it feels like a natural step to enhance the processing of documents, collections and databases to JSON data. Currently, the roots of JNodes are restricted to maps and arrays. We should generalize them and include support for atomic items. The Background:

JSON data types are not restricted to maps (objects) and arrays, they can also be strings, numbers and booleans. As a consequence, a json-doc('input.json') call may also return atomic types. When iterating over JSON input, we should ensure that there is no need for additional type checks to ensure that code does not fail:

for $i in 1 to 10
for $doc in json-doc($i || '.json')
(: where $doc instance of (map(*)|array(*)) :)
return $doc/a/b/c

I assume it would be no substantial change to open $json/step for types other than nodes, maps and arrays. The only tricky special case is null, but it is converted to an empty sequence, why json-doc('document-with-single-null-value.json')/a/b/c already succeeds.

To address any concerns that JNodes do not make sense for standalone atomic items: There is a certain analogy to XML text nodes, which can also be created without serving any immediate purpose, but are helpful and necessary for mapping the entire XML data model.

I'm thinking there may be a case for disallowing or restricting the use of absolute path expressions over JTrees.

Firstly, there's a strong likelihood that users are only dimly aware of where the root of the tree actually is. They are likely to imagine, if they parse a json text, that the root of the tree will be the root of that text. But in fact the root is wherever they started path navigation from, which may be different. It's likely to be particularly confusing if you move out of JNode space into map/array territory and then back again. Requiring an explicit call on root(), or on ancestor::*[last()], might mean that users think more carefully about it. This makes it clearer that the root is not some kind of absolute fixed point, it is simply "the place where you started your current journey".

Certainly, I don't think we should allow /x to implicitly construct a JNode wrapping the context item: the '/' here is redundant and means the user almost certainly doesn't understand what they are doing.

Another consideration is that if we restrict leading / to work only with XTrees, then we reinstate a lot of static type checking capability that we have currently lost.

The machinery for generating test cases from spec examples is failing in the case of the csv-to-xml() tests.

The test generator produces an expected result assertion of

<assert-xml ignore-prefixes="false"><![CDATA[<substitute-for-unparseable-result-xml xmlns="http://www.w3.org/2010/09/qt-fots-catalog"/>]]></assert-xml>

The same problem occurs with some fn:analyze-string tests.

The failure basically means that parse-xml() on the expected test results has failed. (Stylesheet generate-qt3-test-set.xsl line 122).

I think that the problem is that the build is running Saxon in schema-aware mode, and this applies to all XML parsing including the parse-xml() function, so we're getting a schema validation failure where we really don't want to be validating in the first place.

Unfortunately we can't selectively switch parse-xml() validation off unless we move to a later (and not yet stable) Saxon version, and rewriting the whole stylesheet to not be schema-aware would be painful.

A first cut at providing a functional approach to XNode and XTree construction.

At this stage I'm interested in comments on the general approach, not the fine detail (some of which, e.g. namespace inheritance, still needs work.)

Companion PR to #2051 .

I have opted for only two examples, hoping they catalyze the imagination of what is possible. Comments welcome.

Fix #2066

CSS changes to improve alignment of complex signatures, e.g. fn:round

Fix #2066

Revised design for xsl:array based on usage experience.

Fix #2007

Fix #2080

With let $($x, $y, $z), $z bids to the rest of the sequence.

With let $[$a, $b, $c], FOAR0001 is raised if the array is too short.

Note, XPath and XQuery should be reviewed separately as the source text for let expressions is different.

1. When binding to a sequence, the last variable binds to the rest of the sequence.
2. When binding to an array, an FOAR0001 occurs if there are more variables than array members.

Fix #2080.

1. Use non-optional types such as xs:boolean for options parameters
2. Use regular error codes for bad options
3. Drop error code relating to the discontinued method option.

Fix #2082

Proposed revision of the rules for get() in node tests.

Mainly editorial clarification; but also changes the rules for the focus - the expression is now evaluated with absent focus to ensure an error in preference to unexpected results.

Fix #2112

Clarifies that the results are in document order and deduplicated.

Fix #2084

Fix #2087

Fix #2102

Drops the parentheses in map(), array(), function(*)

The documentation for get() says for XNodes…

A selector can also take the form get(Expr). The contained expression Expr is evaluated with the focus of the containing axis step (so its value is independent of the specific XNode being tested). The result of the expression after atomization must be a sequence of zero or more xs:QName values (otherwise a type error [err:XPTY0004] is raised). An XNode satisfies the selector if its node kind is the principal node kind of the axis and its node name is among the values returned by the selector expression.

…and for JNodes…

If the selector takes the form get(Expr), then the contained expression Expr is evaluated with the focus of the containing axis step (so its value is independent of the specific JNode being tested). A JNode satisfies the selector if its ·selector· property is equal to one or more of the values returned by the selector expression, under the rules of the fn:atomic-equal function.

Nitpicking:

• With the existing rule for JNodes, I assume that no match would be returned for EXPR := [ 'a', 'b' ]. I would thus propose to atomize EXPR first and compare it afterwards (see below).
• “A JNode satisfies the selector if [the value of] its ·selector· property”

Next, I would like us to unify the rules for XNodes and JNodes. The rationale (besides “simpler rules are simpler to explain”):

1. XPath is well-known for being forgiving. Maybe we can maintain that tradition for name tests, by tolerating input other than QNames.
2. By using identical rules for JNodes and XNodes, it will be easier to process input that mixed XNodes and JNodes. An example:

(<xml>ignored</xml>, { 1: 'one' }, [ 'one' ])/get(1)

I would propose to simplify the joint rules to the following XPath expression:

some(
  data(EXPR),
  atomic-equal(?, if(. instance of node()) then node-name() else jnode-selector())
)

Finally, I assume that the focus information can be utilized in the get expression, right? Is it correct to assume that all of the following expressions will return <a2/>?

let $xml := <xml><a3/><a2/><a1/></xml>
let $name := #a2
return (
  $xml/get(#a2),
  $xml/get($name),
  $xml/get(node-name()[. = $name])
  $xml/get(xs:QName('a' || position())),
  $xml/get(if(position() = 2) { $name } ),
  $xml/get(xs:QName(`a{ last() - 1 }`))
)

Fix #2095

This PR is purely editorial: it adds notes and examples showing where jnode-content() is (or is not) called implicitly.

Fix #2098

Fix #2103

Responding to an action at today's meeting, this adds a note to the effect that although types can contain cyclic references, instances can not.

The constructor of fn:schema-type-record is currently shown with a type of

fn(xs:anyAtomicType?) as xs:anyAtomicType?

However this does not cover list type constructors, returning multiple occurrences. It should thus be

fn(xs:anyAtomicType?) as xs:anyAtomicType*

A test exists that requires this: schema-type-005

Related (but not identical to) https://github.com/qt4cg/qtspecs/issues/2095:

As Michael indicated in https://github.com/qt4cg/qtspecs/issues/2095#issuecomment-3069173742, the implicit unwrapping of accessed/iterated JNode results may already be defined in the current spec, but it may need to be further clarified. Examples:

let $jnode := { 'array': [ 1, 2 ] }/array
return (
  (: FLWOR expressions :)
  for member $m in $jnode return $m,
  (: Functions :)
  array:size($jnode),
  (: Lookups :)
  $jnode?1
)

Similar to fn:name and other accessor functions, the new JNode functions (fn:node-content, fn:node-position, fn:node-selector) should be gifted with a 0-arity variant.

The current presentation of the data types is inconsistent (https://qt4cg.org/specifications/xpath-datamodel-40/Overview.html#types-hierarchy):

It includes GNode, XNode, and JNode, attribute, document etc. (without parentheses), but function(*), array(*) and map(*). Shouldn’t we remove the parentheses from function(*) etc., or add them to the other types?

Or maybe we should even change function(*) to Function, etc.

Labels | Types -- | -- GNode | gnode() XNode | node() JNode | jnode() attribute | attribute(), attribute(*), attribute(a), … document | document-node(), document-node(*), … function() | function(*), function(xs:int) as xs:int, … array() | array(*), array(xs:int), … map(*) | map(*), map(xs:int, xs:int), …

The spec defines various built-in named record types (https://qt4cg.org/specifications/xpath-functions-40/Overview.html#id-built-in-named-record-types):

key-value-pair
load-xquery-module-record
parsed-csv-structure-record
random-number-generator-record
schema-type-record
uri-structure-record

Suggestions:

1. The spec says in https://qt4cg.org/specifications/xquery-40/xquery-40.html#id-named-record-types:

Named record types implicitly create a constructor function that can be used to create instances of the record type.

I would propose excluding the constructors for built-in types. It will save us a lot of bulky code in the implementations, for functions that will hardly be used, as they all refer to return types of existing functions. Ironically, the only exception might be key-value-pair, but it is redundant anyway (we already have map:pair… unless it is dropped, together with the record type, by #2092).

2. A type is missing for the result of fn:divide-decimals, and we should suffix key-value-pair with -record (provided we will keep it in the spec).

With #2083, some XQFO functions was generalized for JNodes:

fn:distinct-ordered-nodes
fn:generate-id
fn:root
fn:siblings
fn:transitive-closure

Others are still pending (to be completed):

fn:has-children
fn:innermost
fn:outermost
fn:path

We should…

1. analyze the rules of the existing functions (for example, what happens if both XNodes and JNodes are used in fn:transitive-closure?)
2. add more functions.

As a late-breaking change to PR #2083, the item type syntax for matching JNodes has been changed to jnode-type(), to avoid a clash with the name of the function fn:jnode.

We might prefer to resolve this clash in a different way.

Internal questions and feedback on the union operator, triggered by the JNodes proposal:

1. Will { 1: 2 } union { 3: 4 } be allowed, or will it be { 1: 2 }/. union { 3: 4 }/. ?
2. Combining maps and arrays: one might expect { 1: 2 } union { 3: 4 } to result in { 1: 2, 3: 4 }.
3. If union et al. are enhanced anyway, couldn’t they be generalized for sequences? (1, 2) union 3 → (1, 2, 3).

…which I answered as follows:

1. I guess no; the conversion to JNodes is needed.
2. …a good reason why we should not implicitly coerce maps/arrays to JNodes.
3. For atomic-only sequences, it could be equivalent to fn:distinct-values((A, B)). For heterogenous sequences, it gets tricky: How should <a>1</a> union 1e0 be combined? Similar to how functions like fn:min are defined, it could be the first item that determines how the remaining input is combined (but the operation would not be commutative anymore; A union B might yield different results than B union A).

It seems that introduce jnode by extending node is more consistent than introducing jnode as a data type that is a sibling to node.

I would like to hear comments about pros and cons of this approach.

@ruv wrote:

I wonder what if jnode was a subtype of node

@michaelhkay wrote:

Then all operations on node would become available for jnode, including many that obviously don't make sense, for example getting the in-scope namespaces, applying schema validation, etc etc.

This does not seem to be a problem. There are many features that make sense for one node kind (or type) and don't make sense for other.

For example,

• fn:name() does not make sense for document-node(), comment(), text() (but applies to them and return the empty string);
• fn:in-scope-prefixes() applies to only a node that is an element().
• the constructor document { } does not accept an attribute node.

Thus, nodes that are a subtype of jnode can have their own restrictions.

jnode as subtype of node

There can be four direct subtypes of jnode:

• map-node
• array-node
• map-entry
• array-member (or maybe let's call it array-entry)

So, jnode is a union of them: jnode = map-node | array-node | map-entry | array-entry.

Nodes of the type map-node and array-node are similar to document-node. Their parent is always ().

The child axis of a jnode can contains only nodes of the type map-entry | array-entry.

An advantage of this approach is that there is no need to introduce XNode and GNode, and corresponding confusion, like say that node() matches XNode, but not GNode (in the general case).

This probably also allows us to specify XSLT for jnodes more seamlessly.

Fix #2035

By playing around with the new JNodes syntax, I noticed that the somewhat bulky function fn:jnode-content needs to be used a lot to process the result of a path traversal, basically every time when coercion is not possible or does not make sense (iterations, any operation based on item()*).

An example:

{ 'Catania': { 'nomi': ('Alfredo', 'Andrea') } }

1. Count the number of persons in Catania:

./Catania/nomi => jnode-content() => count()

2. List all persons in upper case:

for $nome in .//nomi => jnode-content()
return upper-case($nome)

To prevent users from selectively resorting to the shorter lookup syntax…

.?Catania?nomi => count()

for $nome in .?Catania?*?nomi
return upper-case($nome)

…we could include another pseudo-function that returns the content/value instead of a JNode:

./Catania/content(nomi) => count()

for $nome in .//content(nomi) 
return upper-case($nome)

One drawback would be that this would violate the current principle that the righthand side is only a filter.

Allows conditional and repeated entries in a map constructor.

Fix #2003

The way the XQFO functions are structured is becoming increasingly arbitrary. Examples:

• Processing sequences: fn:doc
• Processing nodes: fn:string
• Processing QNames: fn:in-scope-prefixes
• Parsing and serializing: fn:xsd-validator

For at least 10% of the functions, it will be difficult to find a good categorization (categorization is a challenging topic in itself), but maybe we can improve the status quo.

We can certainly tackle this at a later stage; this issue is only about reminding us of the task.

I propose dropping the three functions map:pair, map:of-pairs, map:pairs, together with the built-in record type fn:key-value-pair.

With the introduction of JNodes, I think these are redundant.

• In place of map:pairs($map), use $map/*.
• In place of map:pair($key, $value), use {$key : $value}/*
• In place of map:of-pairs, use map:build($jnodes, jnode-selector#1, jnode-content#1)

I'm proposing to keep the map:entry, map:entries, and map:merge trio which also do much the same thing.

Similarly, I propose dropping array:members and array:of-members, and "value records":

• In place of array:members($array), use $array/*
• In place of array:of-members, use array:build($jnodes, jnode-content#1)

(Alternatively: keep the functions but define them to return and consume JNodes, rather than key-value and value records.)

Fix #2076

Hopefully this is an improvement!

We say that fn:serialize() returns a string, so I'd expect it to be UTF-8 (or UTF-16, or whatever the implementation's common character set is for strings). I don't see any mention of the meaning of encoding.

Maybe the presentation of XNode and JNode properties could be aligned.

The XDM uses square brackets for XML node properties:

Document node properties are derived from the infoset as follows:

base-uri     The value of the [base URI] property, if available, […]

…and the character ¶ for JNode properties:

JNode has the following properties:     ¶parent: a JNode […]

Would it make sense to always use ¶ or square brackets, or are the kind of properties we talk about too different in order to be aligned? In the latter case, we may need to clarify this in the spec, or add some words on (possibly non-existing) GNode properties.

1. “Regular files” (QT4CG-128-01)

Add POSIX reference.

2. Permissions (QT4CG-128-02)

All functions should be checked with regard to permission handling. Due to the variety of file systems and the programming languages that operate on them, it may turn out that file:not-found and file:io-error is all we can offer.

3. file:is-absolute

The rule says: “A path is absolute if it does not need to be combined with other path information, such as the current directory, to locate a file.”

Thus, file:is-absolute('/') must not return true on Windows systems, as the drive letter is missing.

Other rules of the spec that refer to absolute file paths should reflect this.

4. file:resolve-path

Additionally, Rule 1 “If $path is an absolute path, it is returned unchanged.” contradicts the final sentence, which states that a separator must be added to directory paths:

…to be continued.

The (proposed) spec currently says (for the adaptive output method)

A JNode is serialized by serializing its ¶value property.

I propose changing the output to

JNode(k:v)

where k is the serialization of the selector property and v is the serialization of the value property.

The rule for the JSON output method remains the same.

The reason for the change is so that people can see when a query returns a JNode as distinct from returning its value.

The data model allows the ¶value property of a JNode be (or contain) a JNode. But can it actually happen, and if so, what are the consequences?

I think it can happen. Although fn:JNode can't be applied directly to a JNode, it is possible to construct a map or array in which the entries/members are (or contain) JNodes. We can then wrap such an array or map in a JNode using the fn:JNode function, and the child axis applied to this containing array will return JNodes that have JNodes as their ¶value properties.

While the results may be confusing, I don't think they are harmful (and someone may find an imaginative way of making use of such a structure). For the time being therefore, I propose to allow it, perhaps with an explanatory note to point out any dangers.

Should atomization of a JNode unwrap multiple layers? We currently say that if a JNode J has a ¶value V, then the atomization of J is the atomization of V. I see no particular reason to change that rule, but again, it's an edge case we might draw attention to.

The FOS schema does not allow an fos:example containing an fos:test to omit the fos:result element. I’ve inserted

<fos:result>���</fos:result>

as a placeholder to mark the problem. I also had to move some of the fos:errors sections to a different location.

@ChristianGruen apologies for just pushing these in. I wanted to get the specs building again.

We have two conflicting statements in the spec.

§4.6.4 says: "The step expression S is equivalent to ./S. Thus, if the context value is a sequence containing multiple nodes, the semantics of a step expression are equivalent to a path expression in which the step is always applied to a single node."

§4.6.5 says: "When the context value for evaluation of a step includes multiple nodes, the step is evaluated separately for each of those nodes, and the results are combined without reordering."

I'm not sure which is intended: does S means. / S, or . ! S?

This proposal (which has the JNodes proposal as its baseline) is a first cut at defining generalised steps and path expressions that handle both XNodes and JNodes in a uniform way.

The proposal adds functionality to path expressions (using "/") but does not yet remove the corresponding functionality from lookup expressions (using "?") - that will follow in a subsequent draft.

The changes are largely confined to XPath section §4.6.

Obviously there is much scope to add notes and examples. There is also a need to reorganise sections so concepts are introduced before they are referenced.

In most functions, the options parameters have types such as xs:boolean and xs:string. But in parse-html, they are xs:boolean? and xs:string?

We may need to look at expressions of the following kind…

let $($a, $b) := (1 to 6) ! string()
for $i in 1 to 3
return ($a, $b, $i)

…which currently yield an exception:

Improper use? Potential bug? Your feedback is welcome:
Contact: basex-talk@mailman.uni-konstanz.de
Version: BaseX 12.1 beta
Java: Eclipse Adoptium, 17.0.14
OS: Windows 11, amd64
java.lang.ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 32
	at org.basex.query.var.QueryStack.set(QueryStack.java:124)
	at org.basex.query.QueryContext.set(QueryContext.java:577)
	at org.basex.query.expr.gflwor.Let$LetEval.next(Let.java:147)
	at org.basex.query.expr.gflwor.GFLWOR$1.next(GFLWOR.java:79)
	at org.basex.query.scope.MainModule$1.next(MainModule.java:65)
	at org.basex.query.QueryContext.next(QueryContext.java:395)
	at org.basex.query.QueryContext.lambda$6(QueryContext.java:629)
	at org.basex.query.QueryContext.run(QueryContext.java:770)
	at org.basex.query.QueryContext.cache(QueryContext.java:609)

With the new LetSequenceBinding, LetArrayBinding and LetMapBinding clauses, single items of an evaluated expression can be partially bound:

let $($a, $b) := (1, 2, 3)
let $[$a $b, $c] := [ 1, 2, 3 ]
let ${$a, $b} := { 'a': 1, 'b': 2, 'c': 3 }

It would be helpful to be able to also bind all remaining items, for example with the three-dot syntax. If we allow this syntax for maps, the result could be a submap with all entries except for the ones that were bound by previous bindings:

(: $first: 1, $remaining: (2, 3) :)
let $($first, $remaining...) := (1, 2, 3)

(: $first: 1, $remaining: (2, 3) :)
let $[$first, $remaining...] := [ 1, 2, 3 ]

(: $a: 1, $remaining: { 'b': 2, 'c': 3 } :)
let ${$a, $remaining...} := { 'a': 1, 'b': 2, 'c': 3 }

A QName has three components, prefix, uri, and local, and it's sometimes useful to be able to specify all three.

I suggest extending the definition of EQName to allow the format Q{uri}prefix:local, where the optional prefix is documentary only, but is present in the resulting QName value.

There may be a need to look at places where EQNames are used and to describe the consequences of using this format.

Per chair request, I'm raising an issue on PR #2031 (on issue #2025), even though it has not been adopted by the CG.

I strongly support the JNodes proposal. But in its current state, I have concerns about the fundamentals. If I am right, or only partly right, adjustments will have cascading effects.

I realize that throughout the specs we use the "Definition" rubric loosely, but, for the points I raise below, I would ask that we at least aspire to a more robust definition of "definition." I draw on the classic Aristotelian model, where a good definition should specify the definiendum's genus ("definiendum" = "the thing to be defined"), then supply only those predicates needed to distinguish the definiendum from other species of that genus. The classic example is the definition of "human being" as a "rational animal." The animal is the genus, and the adjective "rational" delimits human beings from non-human being animals. No need to get hung up on details -- that's the gist of the what informs my comments below.

The PR proposes the new top-level structure:

• GNode
  • XNode
  • JNode

The first term is defined: 

[Definition: The term generic node or GNode is a collective term for XNodes (more commonly called simply nodes) representing the parts of an XML document, and JNodes, often used to represent the parts of a JSON document.]

This definition attempts to define things outside the scope of the definiendum. It is presented here as a kind of abstract umbrella category for more specific things. Not a big deal; we carry on:

[Definition: An XNode, more commonly referred to simply as a node, represents a construct found in an XML document. There are seven kinds: document nodes, element nodes, attribute nodes, text nodes, comment nodes, processing instruction nodes, and namespace nodes] [Definition: A JNode represents an encapsulation of a value in a tree of maps and arrays, such as might be obtained by parsing a JSON document. XDM maps and arrays, however, are more general than those found in JSON.]

Each of these two definitions says not what the definiendum is, but rather what it does ("represents": like a member of parliament represents constituents? -- confusing). It also includes buffer words that introduce intermediaries between the definiendum and the thing you would think most immediate to it: "construct," "found," "encapsulation," "value."

More difficult is the fact that, like GNode, an XNode is an abstract category and not a thing in itself. But a JNode, we learn later, is not an abstract category, but an actual thing, with properties. So at the top level of the taxonomy, we have an inconsistency, between abstract categories that have no instantiation, versus those that do, and the two are put in parataxis.

The definition of JNode is not well formulated. It restricts itself to "a value in a tree of maps and arrays" but not to maps and arrays themselves. Does the quoted phrase mean a map entry or an array member? Or the value within said entry or member? 

Slight tangent: in the specs' definition of "value," the term is not really defined, but simply said to be synonymous with "sequence." But in practice the word substitution doesn't work. More often, the specs use "value" in a more restricted, common-sense meaning, to describe a two-term relationship. A thing "owns" a value and some datum inhabits the role of that thing's value. X has value Y. Y is value of X. We run into problems with the ambiguous word "value." Currently a JNode encapsulates a value (see above). But it also has the property (we learn later) of value. So the value has a value?

The JNode definition is sharpened, not in the data model, where it should be, but in the opening sentence of XSLT section 20: "A JNode is a wrapper around a map or array, or around a value that appears within the content of a map or array." This raises the question, what is a wrapper? And content? But it also raises the question about the relationship between JNode and map and between JNode and array, and the juxtaposition of JNode with XNode accentuates the difference. We would never say that an XNode is a wrapper around an element, an attribute, etc. The inconsistency is of a piece with the confusion I've pointed out above concerning the taxonomy of the data model.

Before I propose a solution, I need to probe a similar problem that already exists in the specs:

[Definition: A function is an item that can be called. ]

The word "called" is set in boldface, as if it is a technical term defined elsewhere. It is not, and is rarely used in the specs. What is it for something to be callable? Non-callable? To my mind, we do not have a proper definition of "function," and it is fair game for adjustment. As we have done. In 4.0 we have promoted the function and its proper parts into the topmost level of the data model taxonomy (with adjustments to a few definitions).

[Definition: An array item (also called simply an array) is a function item that represents an array.]

This suffers from the same flaws as GNode and XNode ("represents"), and is tautologous. In the version 3.1 definition of "map" we had the same problem, but the version 4.0 definition at least avoids the tautology.

So, to sum, we have definitions that aren't, inconsistency in our data model taxonomy, and a variety of other problems.

A different approach

We all intuit that the new taxonomy GNode - (XNode | JNode) is meaningful, useful, and important. Arrays and maps really are trees as much as they are functions.

Let all three terms GNode, XNode, and JNode be defined as abstract categories.

Just as XNode is subdivided into specific xnodes, let JNode subdivide into four specific jnodes:

1. map
2. map entry
3. array
4. array member

Adopt the same approach we do for xnodes, and define each of the four on its own terms. Define map - map entry and array - array member along lines similar to the approach adopted in 6.6 to define element - attribute (quite analogous!). We have wrestled over having to have both sequence and selector properties. But with this new approach, we are not stuck. Only map and array jnodes require a sequence property. Map entry and array member jnodes require only the selector property, not the sequence.

This approach is extensible. Suppose we have a proposal for a new JNode. It's a blork, and every blork has one or more cheegs, each one of which has one or more drazers. We simply define three more JNodes: blork, cheeg, drazer. We make sure that the properties for each are suited to what they are individually (the same way we do for the 7 types of XNodes).

One more step, the most controversial: drop Map Items and Array Items from the Function Items category. Yes, there is a fundamental way in which maps and arrays behave like (non-map/array) functions, but there are also equally fundamental ways in which maps and arrays behave like XNodes. If we do not need to define maps and arrays subordinate to XNodes/nodes, then why should we define them subordinate to functions? JNodes have dual citizenship.

An alternative taxonomy is to drop the concept of GNode altogether, and let there be four kinds of item: 

• item 
  • anyAtomicType 
  • XNode 
    • attribute 
    • document 
    • element 
    • text 
    • comment 
    • processing-instruction 
    • namespace 
  • JNode/JFunction 
    • map 
    • map entry 
    • array 
    • array member 
  • function(*)

Includes general refactorings.

Closes #2016

Based on user feedback that I got, it seems that the solution to expand/collapse a sub-TOC could be more intuitive.

@ndw Would it be possible to remove the rightmost arrow icon, and to simply expand the subentries when a TOC entry is clicked? This would also decrease the number of icons, and it would remove the current, somewhat confusing behavior that a TOC entry is also expanded/collapsed when the empty area to the right of the arrow is clicked.

We could keep the arrow on top level to be able to expand/collapse all entries.

• The »Summary of Changes« sections contain outdated information that refer to the presentation of the specs. Maybe we don’t really need them:
  • Use the arrows to browse significant changes since the 3.1 version of this specification.
  • Sections with significant changes are marked Δ in the table of contents. New functions introduced in this version are marked ➕ in the table of contents.

…to be continued.

Christian (I'm not sure where....) has proposed using an operator other than ?, for example ?/, for paths involving JNodes.

This would eliminate some of the non-orthogonalities introduced in the interests of backwards compatibility. For example, the result could always be in document order with duplicates eliminated (which is not currently true for A?B). A?/B could then be truly synonymous with A?/child::B, with no ifs and buts. The result would always be a sequence of JNodes, they would only be unwrapped if used in a context (such as arithmetic) where coercion forces the unwrapping.

This might (perhaps?) also enable a tidier syntax for filtering by type: the current syntax A?~[sequenceType] is rather clunky.

The JNode model as currently proposed doesn't handle sequences very elegantly: specifically maps and arrays whose entries/members contain values of more than one item.

Also (but separate) handling of empty maps, arrays, and sequences isn't ideal.

Consider the map {"a": 1, "b": [2], "c": (3, 4), "d": ([5], [6]), "e": (7, [8]), "f": []}

Applying child::* to this map gives you six JNodes, as you would expect, with selector properties "a", "b", "c", "d", "e", "f". After that, things get complicated.

• The JNode child::a has a value of 1, and no children.

• The JNode child::b has a value of [2], and has one child, with selector=1, value=2, position=1

• The JNode child::c has a value of (3,4), and no children.

• The JNode child::d has a value of ([5], [6]), and has two children. The first child has selector=1, position=1, value=5; the second has selector=1, position=2, value=6. Each of these two children itself has one child.

• The JNode child::e has a value of (7, [8]), and has one child. The child has selector=1, position=2, value=8.

• The JNode child::f has a value of [] and no children.

There is logic to this, but it isn't easy to explain. I don't at the moment have any clear ideas for improving matters, but raise the issue in the hope that we can come up with ideas.

If we decided to introduce a custom JPath expression for JNodes (see #2054), we could possibly use the classic lookup operator to access the properties of JNodes:

For example, the expression…

let $data := { 'name': 'Achab' }
return $data/name

…would result in a JNode…

JNode(
  value     := 'Achab'
  parent    := JNode($data)
  selector  := 'name'
  position  := 1
)

…and we could use $name?selector to retrieve the selector. This way, we could possibly go without the specific functions fn:JNode-content, fn:JNode-selector and fn:JNode-position.

In principle, the classic lookup operator could be further extended to access properties of XNodes.

Proposes a new fn:update function that can handle both JNodes and XNodes.

(this is a branch on a branch, so I don't know how well the diff'ing will work; but look in F&O for the fn:update function)

Small edits to map:build

• fix parameter name
• remove surplus blank lines
• add example with multiple keys returned by key function (a map of sequences)

#1970, Closes #2068.

As this PR includes changes that should have been part of #1970 (and minor other fixes), I will immediately merge the PR. If someone objects, I will be happy to revert the change.

• #2017: Key should also be mandatory for array:sort-by

…to be continued.

If a function parameter type (e.g. in map:build) has a long type that wraps to multiple lines the parameter name and default are center aligned. A similar issue will happen with the type if the default value wraps.

Setting vertical align to top will fix this.

Closes #1970

Editorial. The only controversial change may be to rename the second parameter of map:build from $keys to $key.

Closes #2056

Closes #1996

Closes #2017

Closes #2058

Closes #2059

QNames that are output with the adaptive serialization method could be prefixed with a # character:

serialize((#a, #xml:a), { 'method': 'adaptive' })

(: Result :)
#Q{}a
#Q{http://www.w3.org/XML/1998/namespace}a

The literal QName syntax could be useful for annotations:

(: Current RESTXQ syntax :)
%rest:query-param('search', '{$search}')
(: Alternative new syntax :)
%rest:query-param('search', #search)

To remove the current exotic status of literal QNames, we could allow the syntax at some more places, like:

• Catch clauses: catch #err:XPTY0004 { ... }
• Element/attribute tests: element(#a:b)

With lookups, it is simple to use dynamic keys:

$map?$name

An equivalent solution is missing for path expressions. The current approach is to check the name with a predicate:

$node/*[node-name() = $name]

We could provide a more compare concise syntax by extending element tests (and, similarly, attribute tests):

$node/element($name)

The new literal QName syntax simplifies things further:

for $name in (#h1, #h2, #h3)
return $node/element($name)

The REx-generated XQuery parser has failed to raise the expected syntax error for test case nscons-046.

This is caused by REx' unability to precisely handle differing whitespace allowances of multiple viable parsing alternatives. In the actual case,

<foo>{namespace # ...

there are two possible interpretations for namespace # that still need to be distinguished:

• a NamedFunctionRef refering to some arity of a function named namespace,
• a CompNamespaceConstructor using a MarkedNCName.

Now the problem is that the former allows implicit whitespace to follow, while the latter does not. A REx parser accepts whitespace here and thus fails the test case.

While this is REx' problem, and can be worked around by creating multiple # tokens with different lexical lookahead, it would be nicer to avoid this situation. This would possibly also simplify the work for other parsers.

May I ask to allow implicit whitespace in both the MarkedNCName and QNameLiteral productions? This would be in line with variable declarations and references, which also allow implicit whitespace between $ and EQName.

Redrafting of PR 1942, after discussion, and extension to XQuery

Fix #37 Supersedes #1942

This PR implements the decisions of today's discussion to the best of my understanding. I don't think further discussion is needed, but it does merit careful checking.

Edit: ?/ is preferred over \; see comments.

I feel that there is one aspect about the exciting new JNode proposal that may develop into a permanent crutch. It is the lack of symmetry between simple lookups and lookups with axes:

a?b
a?child::b

This particularly strikes me as this inconsistency does not exist in XPath.

What about the idea to keep XPath 3.1 lookups unchanged and simple – for the vast number of use cases that do not require navigation – and to introduce a new “JPath expression” instead that will exclusively produce JNodes?

The JStep separator that I would recommend for it is the backslash character \. It bears even more resemblance to the XPath step separator than the often questioned question mark. By using a new syntax, I believe we would have much more freedom in designing a clean solution for navigating maps and arrays that is more consistent as well as more similar to classical node paths.

Some syntactical examples:

let $countries := {
  'Japan': {
    'cities': [ { 'Fukuoka': { 'population': 1600000 } } ]
  }
}
return (
  $countries\*,
  $countries\\cities
  $countries\Japan\cities[.\\population > 1000]\..,
  $countries\\*[.\ancestor::Japan]
)

Among other advantages that I see, the traversal of arrays of maps would also be less controversial (#115).

Since the fn:collection function can raise errors, perhaps it should have a corresponding function to check if the collection is available?

For reference, see the other *-available functions:

• collation-available
• doc-available
• unparsed-text-available

Also, eXist has an implementation-specific xmldb:collection-available function: https://exist-db.org/exist/apps/fundocs/index.html?q=xmldb:collection-available. It's typically used to determine if a collection already exists before creating or deleting it.

A proposed signature, based on those linked above, could be:

fn:collection-available( $source as xs:string? := () ) as xs:boolean

The function fn:collation-available defines a $usage parameter, but few information is given on how to interpret the collation URI to return the correct result. In addition, there are no test cases.

Do we believe that the additional parameter offers enough advantages, or should we simplify the function?

I propose an enhancement of xsl:for-each-group to support clustering.

To start off with a simple use case, suppose one has the following population, <xsl:variable name="ages" as="xs:integer*" select="5, 24, 9, 5, 6, 8, 36, 38, 28"/> and one wishes to cluster the figures like so, in four groups: (5, 5, 6, 8, 9), (24), (28), (36, 38).

One is tempted to create an <xsl:for-each-group> with group-by="((. - 1) to (. + 1))". But this does not work. If @composite is absent or is no, eighteen groups are created. If @composite is yes, eight groups are created. In both cases, the results are not significantly close to the desired output.

I propose a new @group-by-cluster. The following code

<xsl:for-each-group select="$ages" group-by-cluster="((. - 1) to (. + 1))">
    <xsl:sort select="current-grouping-key()"/>
    <group key="{current-grouping-key()}" count="{count(current-group())}">
        <xsl:copy-of select="current-group()"/>
    </group>
</xsl:for-each-group> 

would produce this

   <group key="4 5 6 7 8 9 10" count="2">5 5 6 8 9</group>
   <group key="23 24 25" count="1">24</group>
   <group key="27 28 29" count="1">28</group>
   <group key="35 36 37 38 39" count="1">36 38</group>

There are numerous use cases for the proposed new feature. Here are a few:

• Clustering map or spatial coordinates
• Grouping disparate rectangles from OCR output
• Reconciling triples in linked open data (RDF) that use different IRIs synonymously
• Detecting typologies within in large corpora of documents that have periodically repetitive formulaic paragraphs.
• Discovering networks of connected things, e.g., networks of email correspondence or publication citations   Currently, the clustering I describe above is feasible in XSLT, but it requires creative strategies, usually a combination of preprocessing and the creation of specialized helper functions to recursively iterate over multiple grouping keys to create group numbers. These are challenging to write and debug, and one loses identity in a preprocessed copy of the original.

By putting clustering into a @group-by-cluster construct, users benefit not only from convenience but also from performance, as a processor might bring novel strategies for clustering.

The current-grouping-key() for a group would consist of a sequence of all members' grouping keys, duplicates removed. No two groups would have any overlap in their grouping key sequences. (That's the definition of a cluster.)

@group-by-cluster would have effect only if its value actually produced a sequence of length greater than one, and if @composite is no. (Should a user should be warned if @composite is yes?)

Fix #2040

For functions like name(), local-name() etc with as="node()? default="." in the signature, allow the context value to be an empty sequence.

Fix #2045

Discussion of untrusted execution (that is, a Processor executing code from an untrusted source), and security in general, is present in the spec, but spread out and not really connected.

Untrusted execution seems to me to be one of the biggest security issues for XSLT/XQuery and XPath in general, and I think it’s important to make the distinction between untrusted execution where an untrusted stylesheet or query is executed (perhaps via Saxon’s XsltCompiler.compile()), and when a trusted stylesheet or query causes untrusted code to be executed, as with fn:parse-xml(), fn:doc(), fn:transform() and <xsl:evaluate>.

The proposals in #2034 address one case, but have no effect on the other, and that raises the question of what would happen if an untrusted (unsafe) stylesheet executed fn:doc() with safe = true.

I would like to see the specs address the security implications of untrusted execution more explicitly, and to provide clearer guidance for implementation authors around both completely untrusted code, and trusted code which executes untrusted code.

I think that could take the form of:

• Annotation of all functions that have potentially problematic external effects (primarily file / resource access).
• Clear expectations for implementors about what (and how) security restrictions should be configurable.
• Consistent Error codes for security-related exceptions
• Consistent security-related options for functions which can cause untrusted source parsing or code execution.
• A additional security section in the spec under section 2 (Concepts / Basics) in the XSLT/XQuery/XPath specs, and section 1 of XPFO, which collects the overview and points to what to look for in the rest of the spec.

While looking at a problem in an implementation of a vendor function that read data from an external resource I was looking through the spec to see what was said about external resources and untrusted execution contexts (for example a processor executing XSLT or XPath provided by a user).

The spec makes mention of ‘available documents’, ‘available text resources’, ‘available binary resources’, ‘available collections’, and ‘available URI collections’.

The documentation for fn:doc(), fn:collection(), and fn:uri-collection() mention an error (err:FODC0002) to be raised if the URI requested is not in the relevant ‘available X’ (although there is some disagreement about what the ‘X’ should be - we have ‘available node collections’ and ‘available resource collections’ mentioned in the function docs). fn:unparsed-text() and fn:unparsed-binary() do not.

I think the documented behaviour of fn:unparsed-text() and fn:unparsed-binary() should be brought into line with the others.

I propose promoting ".." to be a primary expression (rather than an abbreviated step), and (assuming the JNodes proposal is accepted) allowing it to work on JNodes as well as XNodes.

Ignoring JNodes for now, I don't think the change would make any observable difference to the language syntax or semantics. It changes the way that a predicate is interpreted : ..[P] becomes a regular filter expression and is no longer subject to the special rules for predicates within steps; but the only difference is how position() is interpreted, and since .. can only return a singleton, this makes no difference.

For JNodes it means we will be able to write expressions such as $my-map??x[..?y = 3] rather than $my-map??x[?parent::*?y = 3]

A number of functions such as fn:name() take the context value as their default argument, with the implication that name() is exactly equivalent to name(.).

However, these functions also say that a type error XPTY0004 is raised if the context value is not a single node.

It seems to me that if the context value is an empty sequence, no type error should be raised; the effect of () -> name() should be the same as name(()).

Today's merge of qt4cg/qtspecs#2028 has added the MarkedNCName production to both the XQuery and the XPath spec. It is however only used within the XQuery spec.

Fix #2038

Fix #2041

Reported against XSLT 3.0

https://github.com/w3c/qtspecs/issues/71

Section 5.17:

• there is no changes entry flagging the fact that the coercion rules are now applied (and corresponding test cases such as contextDecl-037a do not identity a PR). The relevant PR is PR #254 .

• The statement "The context value declaration has the effect of setting the context value static type T in the static context." is incorrect. The static context no longer includes a context value static type.

• The statement "In all cases where the context value has a value, that value must match the type T according to the rules for SequenceType matching" is incorrect (or at least, misleading): as stated two paragraphs later, coercion is applied. But because there can be multiple context value declarations in different modules, specifying different types, perhaps the intent is that coercion is applied only to a value supplied in the query, and not to a value supplied externally? If so, this needs clarifying. There are apparently no tests for coercing an externally-supplied value to the required type.

Various places, for example the xsl:context-item declaration and the xsl:evaluate/@context-item attribute, should be updated to allow the there being a context value rather than a context item.

At present the context value at instruction level is always either a singleton or absent. We should consider generalizing this to align with XPath, where the -> and ?[....] operators allow the context value to be an arbitrary sequence.

An xsl:for-each-member instruction that iterates over an array and binds each member to the context value would make sense.

The new fn:apply-templates function in XSLT can invoke the "default mode", either by specifying mode="#default" or by not specifying a mode. The default mode is defined by the nearest containing instruction that has a [xsl:]default-mode attribute.

I would like to drop this dependency.

Most of the cases where a function call depends on the static context (especially an XSLT function) are cases where the relevant property is fixed for a package (e.g. the set of named keys, decimal formats, or character maps). There are places where there is a dependency on something tha can vary in a more fine-grained way - notably (a) the default collation, and (b) the set of namespace bindings, but on the whole such dependencies are undesirable (a) because they introduce opportunities for user error (e.g. when copying and pasting code) and (b) because they increase the amount of information the processor has to keep around at runtime just in case it is needed (for example, in a dynamic function call). I would therefore like to avoid introducing this dependency.

The proposed change is (a) to drop "#default" as a value of the mode option for this function, and (b) to say that if no mode is specified by fn:apply-templates the unnamed mode is used.

Fix #2036

The special condition that allows more than one operand of xsl:map to be consuming should apply only if the duplicates attribute is absent (defaulting to "error"). If duplicates are allowed then in general the result cannot be streamed.

The example of mutually-recursive record types in XPath §3.2.8.3.1 (using the schema component model as an example) is unrealistic, because an instance of this structure would be cyclic at the instance level, and therefore would be non-instantiable. In practice, the only way to represent cyclic structures using maps and arrays is by use of functions to represent some of the relationships, as we do in the schema record type returned by functions such as fn:schema-type(). We should change the example to use this technique and explain why it is being used: it would still illustrate the point; although it would be in danger of becoming excessively complicated.

This issue replaces #1955.

The first feedback that we got for the entity-expansion-limit option indicates that our current solution is neither fish nor fowl (weder Fisch noch Fleisch?):

1. With the initial suggestion in #1860, I hoped we could define sane defaults to prevent attacks caused by fn:parse-xml and fn:doc. This turned out to be difficult. Instead, we now have two specific options (allow-external-entities, entity-expansion-limit) that need to be explicitly assigned to make parsing safer.

2. In order to parse certain XML documents, like dblp..xml.gz, more than one JDK 11 limit needs to be increased:

http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit
http://www.oracle.com/xml/jaxp/properties/maxGeneralEntitySizeLimit
http://www.oracle.com/xml/jaxp/properties/totalEntitySizeLimit

As we have observed, XML parsing depends on the specific XML parsers. I believe it would be more user-friendly to replace the specific settings with a single option safe, and to let the processor decide which properties are assigned:

1. true (default): Avoid XXE and billion laughs attacks
2. false: disable safe parsing (increase limits, allow parsing of external resources)

Fix #2032

Location of typo: Array Types section.

Current text:

[ 1, 2 ] instance array(*) returns true()

Expected text:

[ 1, 2 ] instance of array(*) returns true()

Fix #2025

This is a first draft for review.

It includes changes to the data model, functions and operators, and XQuery/XPath. It does not yet include changes to XSLT.

It's a big proposal, but I think it removes more complexity from the spec than it adds. It's basically a unification of two concepts, both of which were addressing aspects of the same problem, namely that lookup expressions lose too much information. It gets rid of the pin/label mechanism, and modifiers on lookup expressions, and introduces JNodes and JAxes in their place. (Any suggestions for improved terminology are more than welcome.)

I think we get a lot more "bangs for the buck" with this solution, and it makes navigation of JSON trees work in a much closer way to familiar navigation of XML trees. It needs a lot more work on examples and explanation, of course.

Adds more explanation to xsd:validator

Extracts material from the XQuery and XSLT specs describing the validation process, moving this to a new section in F&O, to reduce duplication.

Fix #2029

See action QT4CG-119-02

In the review when the function was accepted, I was asked to supply more notes and examples indicating how the various options for assembling a schema interacted with each other.

Fix #2027

Action QT4CG-021-01 (should be QT4CG-121-01)

We now allow QNameLiterals to be used in element and attribute constructors, for example attribute #xml:space {"default"}.

For symmetry we need a similar syntax for namespace and processing-instruction constructors. These have the same ambiguity problem if the node name clashes with a reserved word such as "div", but they are constrained to be NCNames (or no-namespace QNames, depending on your perspective).

One possible solution is to use a new construct such as NCNameLiteral (but the name is wrong, unless we also allow it to be used in contexts where a Literal is allowed). Another possibility is for the grammar to allow a QNameLiteral, but for the semantics to restrict it to be in no namespace.

Fix #2022

The effect is that support for library modules is no longer optional.

I decided not to pursue merging the "schema import" and "typed data" features into one.

We have two rather separate mechanisms, both designed to solve aspects of what is essentially the same problem: lookup expressions lose too much information.

Pinning tries to solve the problem by saying that if the origin of the lookup is pinned, then the results of the lookup carry a label containing information about the key and the parent.

Modifiers like pair::* try to solve the problem by returning a map containing the key and the value as separate fields.

But pinning only solves part of the problem, in particular it doesn't prevent X?* flattening the result, and the pairs modifier only solves part of the problem, in particular it doesn't retain parentage.

I would like to try combining them and trying to create a mechanism that is better than either. I don't yet know exactly how this might work, but I'm thinking along the lines:

• Replace the concept of labelled items with labelled values (that is, a label can be attached to any value, not just an item)
• Scrap pin() as an explicit function
• A lookup expression like $X ? child::Y returns a sequence of labelled values (no flattening)
• The properties of a labelled value include: ** target - the actual value ** key - the associated key (or array index) ** parent - the containing map or array
• These properties might be made available through syntax such as $LV ? target::* , $LV ? key::*, $LV ? parent::* (or otherwise)
• ancestor and ancestor-or-self can be made available as derived properties
• Many operations when given a labelled value should automatically operate on its target, and ignore the label (rather like atomisation). Exactly which operations do this is an interesting question to which I don't yet know the answer. It's tricky because child::* returns a sequence of labelled values, and we want to be able to manipulate this in unflattened form. Perhaps child::* should instead return an array of labelled values? But then you end up with another lookup operation to extract the members of this array.

Fix #2023

In §4.14.3.1 we describe the semantics of lookup expressions. Rules 3a to 3e and 4a to 4e all start "if the KeySpecifier KS is..." and should enumerate all the possibilities for a KeySpecifier. But the case where the KeySpecifier is a VarRef is not mentioned.

Of course, X?$a is supposed to be a shorthand for X?($a), we just fail to state the fact.

In XQuery I propose:

• Dropping the "module feature" - every conformant XQ40 implementation must support library modules
• Merging the "schema-aware" and "typed-data" features into a single optional feature, aligned with schema-awareness in XSLT

I propose to move XSLT section §5.4 Patterns so it becomes §6.1, under Template Rules, where it will hopefully be easier to find.

Because this will produce a large number of diffs, I propose to make it a separate PR rather than combining it with the work I am currently doing on patterns and template rules for maps and arrays.

The section for xsl:select in the XSLT specification includes the following rationale:

An XPath expression written within an XML attribute is subjected by the XML parser to attribute value normalization, which changes the arrangement of whitespace within the value. While this will rarely affect the actual meaning of the expression, it can mean that formatting is lost. Multi-line attribute values are therefore best avoided. The loss of formatting also makes it difficult for an XSLT processor to provide precise error locations.

There are good reasons why xsl:select would be a useful instruction, but I don't think providing precise error locations is one of them. This is just circumventing a problem that is solvable today for select attributes. If an implementer wanted to supply a more precise error location in attribute values (and this would certainly help developers) they could adopt a solution similar to the Ecma SourceMap used by EcmaScript transpilers and minifiers.

In XSLT 3.0, we frequently work with multi-line select attribute values on XSLT instructions without major issues. Examples include: when calling using fold-left() functions with one or more inner functions or using multi-case if/else expressions. Using xsl:select just to get good error messages does not seem like a good trade-off for the added verbosity.

For these cases, one can either use a simple XPath linter in the XSLT editor to highlight the specific error tokens caused by basic typos and unresolved references, and then fall back on using the compiler error messages with approximate line-numbers for (the many) cases that the linter cannot pick up.

Precise XSLT Error Locations and AI Agents

Modern XSLT editors are today fully integrated with AI Agents (e.g. GitHub Copilot or AI Positron). These agents use reported error-locations to explain and suggest a fix for the XSLT problem for the user. Precise error locations are critical to the quality of the explanation and the fix. This help should be available equally for XSLT select attributes or xsl:select instructions.

Currently work In progress, committed so that the draft can be reviewed.

Changes in three main areas:

• Pattern syntax: patterns such as ?item and ?parent?item are defined to match items in a map by their key
• Built-in template rules for on-no-match="shallow-copy-all". Revisits the built in template rules for this scenario.
• General revision of the processing model for xsl:apply-templates applied to a tree of maps and arrays.

Code that calls xsl:apply-templates inevitably has expectations about the type of the result. For example someone doing

<xsl:apply-templates select="@*"/>

may have an expectation that the result will be a sequence of attribute nodes, and the code might fail untidily if it is anything else. There is currently no way of stating this expectation, or of triggering coercion on the result. We have added an attribute xsl:mode/@as but different calls on apply-templates in the same mode may have different expectations. (I'm seeing this particularly with modes that process maps and arrays)

We could add an as attribute to xsl:apply-templates to make the expectation explicit.

We should make the second parameter obligatory (fn:sort-by(1 to 3) seems confusing).

sort(keys := fn { ?key }) occurs twice in the remaining text; should be sort(key := fn { ?key }).

The EXPath File Module must be revised in several steps. First of all, several functions need to be incorporated that were added to the initial version (details: https://docs.basex.org/12/File_Functions).

Fix #2009

The rules for xsl:variable are changed so there is no attempt to construct an implicit temporary tree when the sequence constructor contains an xsl:map. xsl:array, or xsl:select instruction (perhaps mixed with other instructions).

Compatibility: note that xsl:array and xsl:select are new in 4.0, while xsl:map inside xsl:variable always throws an error in XSLT 3.0.

Justification:

• a child xsl:select element behaves like a select attribute
• if the content of xsl:variable is xsl:map or xsl:array it makes no sense to require the user to add as=map(*) or as=array(*) because the type is obvious anyway.

Completes action QT4CG-122-01

Closes #748

Issue #655 PR #795 introduced fn:sort-with.

We should define array:sort-with for consistency.

Updates the static typing rules in XSLT for new kinds of expression introduced in XPath 4.0. These rules are used in streamability analysis, but more work needs to be done to complete the streamability analysis.

Production rules are now referenced by name, as production numbers are no longer available.

This is related to issue #402.

We have generalised the meaning of union, intersect, and except, when used in XSLT patterns, so that they now mean:

• A union B - matches either A or B
• A intersect B - matches both A and B
• A except B - matches A and does not match B

With these semantics, there is no longer any sensible reason to restrict these pattern operators to apply only to node patterns. The semantics work equally well for patterns that match (for example) maps or arrays. For example

<xsl:template match="record(a, b) except record(a, b, c)">

In XSLT 3.0, an xsl:variable instruction with no select or as attribute implicitly wraps the value created by the sequence constructor in a document node. This inevitably fails if the content is a map, making it necessary to write

<xsl:variable name="m" as="map(*)">
   <xsl:map>...</xsl:map>
</xsl:variable>

I propose that this wrapping should not happen if the first item in the result of the sequence constructor is a function item (including a map or array). The practical effect on users is that they can leave out the as="map(*)" attribute in this situation.

For function items other than arrays, this is currently an error condition so there is no incompatibility.

For arrays it does represent an incompatible change -- the current rules ("Constructing complex content") say that an array is flattened. But XSLT 3.0 has no instruction to construct an array, it would have to be done using xsl:sequence; and no-one would deliberately construct an array merely in order to flatten it, so the situation is unlikely to arise in practice.

The proposal is that the decision whether or not to construct a wrapping document node should be based on the first item in the sequence. This is to allow lazy evaluation. A function item appearing later in the sequence would be handled the same way as now -- most likely an error. An empty sequence continues to result in a childless document node.

Fix #2004

This kind of code comes up a lot, and is hard to simplify except by dropping into XPath:

                <xsl:array>
                  <xsl:for-each select="?members?*[?_nodeType='MethodDeclaration']">
                     <xsl:array-member>
                        <xsl:apply-templates select="."/>
                     </xsl:array-member>
                  </xsl:for-each>
               </xsl:array>

Issue #2005 (PR #2006) make it feasible to do it all in XPath (by calling the apply-templates() function) but I don't feel that's the whole answer.

Perhaps we could define something like

<xsl:array-build for-each="?members?*[?_nodeType='MethodDeclaration']">
   <xsl:apply-templates select="."/>
</xsl:array-build>

Fix #2005

I propose introducing apply-templates() as an xslt-only function, with semantics broadly equivalent to the xsl:apply-templates instruction.

The main use case identified so far is when constructing maps and arrays, it enables the XPath syntax to be used rather than the much more verbose XSLT syntax.

Parameters:

• select: the items to be processed using matching template rules
• with-params: the parameters to be passed. Like with-params on xsl:evaluate, this means variable names exist at run-time, which is a bit of an innovation, but I think it's manageable
• mode: again, this means mode names exist at run-time, which may have consequences. There are open questions about what the default should be, or how the various options (default mode, unnamed mode, current mode) should be expressed.

In the case study https://github.com/qt4cg/qtspecs/issues/1786#issuecomment-2884424739 I encountered a use case where an instruction xsl:xpath would be useful.

<xsl:xpath>
   { "class":
       { "name": f:degenerify(name/@identifier),
          "abstract": ? abstract,
           "extends": array{? extendedTypes =!> map:merge(apply-templates()) },
           "implements": array{? implementedTypes =!> map:merge(apply-templates()) }
             ...
       }
  }
 </xsl:xpath>

The instruction is very simple: <xsl:xpath>EXPR</xsl:xpath> is equivalent to <xsl:sequence select="EXPR"/>. It's particularly useful because XPath constructors for maps and arrays are so much more concise than the XSLT equivalents. Compared with using xsl:sequence, it means that:

• XML attribute value normalization doesn't kick in, so your formatting is better protected (meaning also that the system has some chance of computing line numbers correctly for diagnostics)
• You haven't tied up either single or double-quotes as an attribute delimiter; both can be freely used within the expression.
• You aren't creating the false impression that you're returning a (multi-item) sequence

Note that the content is NOT a sequence constructor; no child elements are allowed; and the content is not interpreted as a text value template. Unlike xsl:evaluate, the XPath expression is statically fixed.

(This example also introduces apply-templates as a function, but that will be a separate proposal).

If you're constructing a map using a map constructor, adding an entry conditionally can be a real pain, and typically involves a wholesale rewrite of the way the map is constructed. It would be nice to be able to mark an entry as optional so users don't have to resort to such wholesale rewrites.

The difficulty of course is finding a nice syntax: one that is both intuitively readable and grammatically unambiguous.

One possibility might be:

MapConstructorEntry ::= MapKeyExpr ":"  MapValueExpr "optional"?

with the semantics that if the "optional" keyword is present, and the result of evaluating MapValueExpr is an empty sequence, then the entry is omitted from the constructed map.

A more ambitious construct might be

MapConstructorEntry ::= MapKeyExpr ":"  MapValueExpr ("when" MapEntryCondition)?

which adds the entry to the map only if the condition is true.

For example:

let $map := {"height": string(@height), 
            "width": string(@width), 
            "weight": string(@weight) when exists(@weight)}

We could use the new QName literal syntax when serializing QNames with the adaptive method:

serialize(xs:QName('x'), { 'method': 'adaptive' })

(: current output: Q{}x :)
(: proposed output: #x or #Q{}x :)

Fix #1085

The new functionality introduced into the 4.0 version of fn:sort is repackaged into a new function fn:sort-by with a much cleaner interface; the fn:sort function reverts to its 3.1 specification.

If this PR attracts support then the corresponding change will be applied to the array:sort function.

The specifications of element-to-map() and element-to-map-plan() use different record types for the data structure representing the plan. In both cases the definition is less precise than it might be (though not wrong). The two functions should use a common named record type, which should be as precise as possible.

Fix #1992

Fix #1997

(A correction to an editorial error that made a substantive difference to the spec.)

This section reads:

If R is an [atomic type] and J is an [atomic item], then:

• If J is an instance of R then it is used unchanged.
• If J is an instance of type xs:untypedAtomic then: ** If R is an [enumeration type] then A is cast to xs:string. ** If R is [namespace-sensitive] then a [type error] [[err:XPTY0117]] is raised.
• Otherwise, J is cast to type R.

The last line (rule 3(c)) looks all wrong. If we just did a cast at this point then rules 4 and 5 would be unnecessary.

It's not easy to trace the history, but I think it went wrong when the rules for choice/union types were refactored (around 2024-04-12). The line in question appears to have originally been under a conditional "if A is an instance of xs:untypedAtomic...".

Various types of expressions are allowed as a KeySpecifier:

Lookup  ::=  ("?" | "??") (Modifier "::")? KeySpecifier
KeySpecifier  ::=  NCName | IntegerLiteral | StringLiteral | VarRef | ParenthesizedExpr | LookupWildcard | TypeSpecifier

Maybe we could add ContextValueRef and NumericLiteral to the list, to make the following expressions legal:

{ '1.5': 'one and a half' }?1.5
array { 1 to 256 }?0x80
(3, 4, 5) ! $array?.

The different variants to look up array members should be unified. For example (if I interpret the rules correctly), the following expressions can be evaluated…

[ 'a' ]?(<x>1</x>)
[ 'a' ]??(number(<x>1</x>))
[ 'a' ]??(1e0)
let $a := 1e0 return [ 'a' ]??$a

…whereas the following expressions raise errors:

[ 'a' ]?(number(<x>1</x>))
[ 'a' ]?(1e0)
let $a := 1e0 return [ 'a' ]?$a

We should probably try to make all of them legal, or try to justify what happens.

Fix #1993

The signature for map:pairs is

map:pairs(
$map	as map(*)	
) as key-value-pair*

but the test case generated in misc-BuiltInKeywords is (incorrectly)

map:pairs(map := ?) instance of function(map(*)) as fn:key-value-pair

Note the missing * at the end.

In the type fn:schema-type-record (returned by functions such as fn:schema-type), the field constructor is said to be of type fn(xs:anyAtomicType) as xs:anyAtomicType. It is also said to be "the same function as returned by [fn:function-lookup] applied to the type name (with arity one)". But that function has type fn(xs:anyAtomicType?) as T?

The correct type for the constructor field should be fn(xs:anyAtomicType?) as xs:anyAtomicType?

This PR adds six built-in named record types to the static context of every application:

Record [key-value-pair] Record [load-xquery-module-record] Record [parsed-csv-structure-record] Record [random-number-generator-record] Record [schema-type-record] Record [uri-structure-record]

These are now listed in Appendix C of F&O

Issue 835 requests a review of the names of these records; perhaps putting them in one place will make that review easier. Personally, I am happy with the names as currently defined.

Fixed invalid syntax xs:simpleType/@ref (moved to @memberTypes) in simpleType named method

Based the type fixed-namespaces-type-default on xs:token instead of xs:string to allow for whitespace normalization

Changed collation attribute on xsl:merge-key to be an avt (according to spec)

Changed attributes that were previously of type "xsl:char-optionally-expanded" to just xs:string since the spec says they can be any string. I couldn't think of a reason they should be limited to one character optionally followed by a colon and more characters, so I assumed this was some sort of artifact from the past.

Changed xsl:next-iteration and xsl:evaluate to not allow mixed content

Corrected _split_when to _split-when

Added missing shadow attributes (in two places) for allow-duplicate-names, build-tree, json-lines and json-node-output-method

Added missing shadow attribute for select on perform-sort

Added assertions for required attributes:

• xsl:use-package - name
• xsl:expose - names, component and visibility

Made the errors attribute a list of tokens instead of just xs:token. Doesn't affect validation but I think it is more clear.

Changed default value of per-mille from a tilde to ‰

Changed the default value of the on-no-match attribute from shallow-skip to text-only-copy, per the spec

Gave xsl:exclude-result-prefixes the same type as no-namespace exclude-result-prefixes, to allow for #all and #default

Gave xsl:extension-element-prefixes the same type as no-namespace extension-element-prefixes, to allow for #default

Removed the xsl:prefixes and xsl:char-optionally-expanded types since they were no longer used after the above changes

Changed the type of the visibility attribute on xsl:attribute-set, xsl:function, xsl:template and xsl:variable to xsl:visibility-not-hidden-type to exclude "hidden" per the spec

Changed the keyword value of the fixed-namespaces attribute from #default to #standard (and adjusted type names)

Fix #1983

Fix #1986

Fix #1985

Editorial.

The main effect is to centralise the descriptions of how to expand unprefixed QNames into a few named rules which can be referenced and reused throughout the spec.

I propose dropping the folliowing text in XQuery §2.4.2

None of this text says anything prescriptive, and the suggested notation of URI#local appears outdated.

The method by which an XQuery 4.0 processor reports error information to the external environment is implementation-defined.

An error can be represented by a URI reference that is derived from the error QName as follows: an error with namespace URI NS and local part LP can be represented as the URI reference NS # LP . For example, an error whose QName is err:XPST0017 could be represented as http://www.w3.org/2005/xqt-errors#XPST0017.

Note:

Along with a code identifying an error, implementations may wish to return additional information, such as the location of the error or the processing phase in which it was detected. If an implementation chooses to do so, then the mechanism that it uses to return this information is implementation-defined.

There are places in the spec that use sloppy terminology regarding namespaces. For example 2.1.3 says

the namespace URI is inferred from the prefix by examining the in-scope namespaces in the static context

But the static context does not define "in-scope namespaces", it defines "statically known namespaces"

I propose to put together an editorial PR to tidy this up.

Fix #882

Supersedes PR #1883

There has been a great deal of discussion about the relative merits of the status-quo fn:chain function and the proposed replacement fn:compose. The CG was polled on whether it preferred to have fn:chain only, fn:compose only, or both, or neither. There was no clear consensus. The only option which no-one seemed to favour was to have fn:chain only -- which is the status quo. Since no-one is happy with the status quo I am therefore proposing that we drop this function. We can then start with a clean slate.

For the record the main criticisms of the fn:chain function as currently specified were:

(a) it is more useful to have a function that combines several functions into a single function, without actually applying that function to a set of supplied arguments

(b) The function has special-case behaviour for arrays (if the input is not an array and the function has arity > 1 then the input sequence is converted to an array).

(c) The need for the function is not clearly motivated; the examples given can all be achieved in some simpler more intuitive way.

We have introduced (in 4.0) the option to specify element and attribute names in computed node constructors in the form of string literals. We should replace this with QName literals.

Resolves the syntax problem identified in #1981 by requiring a space between ( and #.

Adds more examples and notes scattered around the specs.

Unfortunately (as revealed by implementation and testing) the syntax for QName literals clashes with the syntax for pragmas in XQuery.

In the expression error(#err:XPTY0004), the longest token after error is (# which looks like the start of a pragma.

It's actually a wee bit complicated. Looking at the tokenization rules, we shouldn't be recognizing a pragma here because there is no closing #). The tokenization notes say "The lexical production rules for [variable terminals] have been designed so that there is minimal need for backtracking."; the introduction of the new syntax would mean that this is no longer the case. But regardless of the details, I think we have to change the QName literal syntax.

I propose we go for doubling the hash: error(##err:XPTY0004). We need to qualify the rules for tokenizing a pragma to say that a pragma is recognized when we see ((#, optional whitespace, EQName) - that's not unlike the rules we have for other "variable tokens".

See Saxon bugs:

https://saxonica.plan.io/issues/5852 https://saxonica.plan.io/issues/6772

regarding the recognition and generation of META elements in the HTML and XHTML header sections.

Saxon is producing HTML5 output as mandated by the 3.1 serialization spec but this is apparently either invalid or deprecated by the HTML5 specification. The 4.0 serialization spec makes some adjustments in this area but I don't think it is fully in line yet with HTML5.

One cognitive challenge with records is to internalize that records are not independent types, but only map constraints. As a consequence, no type safety guarantees exist when records are accessed and updated:

• A lookup of a non-existing key raises no error.
• A record update may result in a map that does not match the original record definition.

This makes it hard and often impossible/illegal for processors to output helpful error messages.

There are reasons why we don’t want to make records too strict: an extensible record may include keys that are not defined in the record type:

(: must not raise an error :)
declare record local:r(a, *);
let $r as local:r := { 'a': 1, 'b': 2 }
return $r?b

However, for non-extensible records, I think we should allow processors to perform stricter checks when unknown keys are looked up, or when the result of an update would conflict with the original record type:

declare record local:r(a as xs:integer);
(: unknown key :)
local:r(1)?b,
(: invalid value type :)
map:put(local:r(1), 'a', 'string')

As records are no independent types, it will be difficult to enforce errors in all cases: It would require implementations to always know that a currently processed map has once been validated against a specific record type. But in many cases, implementations may be able to preserve record types for maps that have been coerced to a record, or created with a record declaration, and propagate them to updated maps. For example, we already do so when we can statically infer that the resulting map of a map:put call will match the original record type.

The Problem

Function map:build, does not allow to explicitly define the functional dependency of a value on its key.

As result, it is unusable for creating even such simple maps as the following:

The input is:

("apple", "apricot", "banana", "blueberry", "cherry")

The $keys function is: $keys := fn($x){characters($x)} That is, every character, in every input string, is a key.

We need the values to be: if an input string contains the key two or more times, then each such string, else the empty sequence.

The expected map to be produced is:

{
  "a":  "banana",
  "b": "blueberry",
  "c": (),
  "e": "blueberry",
  "h": (),
  "i": (),
  "l": (), (: Lowercase L :)
  "n", "banana",
  "o", (),
  "p": "apple",
  "r": ("cherry", "blueberry"),
  "t": (),
  "y": ()
}

Solution

We provide a new definition of map:build - this can be a complete replacement of the current function, or could be added as a new overload. I am in the process of writing a PR, and your feedback would be appreciated.

The definition is simple:

let $mapBuild := fn(
$input	as item()*,	
$keys	as (fn($item as item(), $position as xs:integer) as xs:anyAtomicType*),
$value	as (fn($key as xs:anyAtomicType, $input as item()*) as item()*)
) as map(*)
{
  let $allKeys := distinct-values(for-each($input, $keys))
   return
     $allKeys ! map:pair(., $value(., $input)) => map:of-pairs()
}

As can be seen from executing the code below, the redefined function can be successfully used to build the "problematic" map above, and also all currently provided examples in the FO Spec for the function map:build.

let $mapBuild := fn(
$input	as item()*,	
$keys	as (fn($item as item(), $position as xs:integer) as xs:anyAtomicType*),
$value	as (fn($key as xs:anyAtomicType, $input as item()*) as item()*)
) as map(*)
{
  let $allKeys := distinct-values(for-each($input, $keys))
   return
     $allKeys ! map:pair(., $value(., $input)) => map:of-pairs()
}
 return
   let $input := ("apple", "apricot", "banana", "blueberry", "cherry"),
       $employees :=
         <employees>
           <employee name="Jim Nelson" location="New York" ssn="1234567890" salary="123456"/>
           <employee name="Ann West" location="New York" ssn="0987654321" salary="99999"/>
           <employee name="Peter Smith" location="Seattle" ssn="123454321" salary="155223"/>
           <employee name="Karen Johnson" location="Seattle" ssn="5432198760" salary="175000"/>
           <employee name="Jonh Lagarde" location="Boston" ssn="9999999999" salary="145000"/>
           <employee name="Samantha Weird" location="Boston" ssn="1111111111" salary="153000"/>
         </employees>
    return
(    
     $mapBuild(
       $input,
       fn($string, $pos) {distinct-values(characters($string))},
       fn($key, $input)
       {
         filter($input, fn($string, $pos){$key = duplicate-values(characters($string))}) 
       }
     ),
     $mapBuild((), string#1, string#1),
     $mapBuild(1 to 10, fn {. mod 3}, fn($key, $input){filter($input, fn{$key = . mod 3})}),
     $mapBuild(1 to 5, identity#1, format-integer(?, "w")),
     $mapBuild(("January", "February", "March", "April", "May", "June",
                "July", "August", "September", "October", "November", "December"),
               substring(?, 1, 1), fn($key, $input){filter($input, fn{$key = substring(., 1, 1)})}
              ),
     $mapBuild(
        ("apple", "apricot", "banana", "blueberry", "cherry"),
        substring(?, 1, 1), fn($key, $input){sum($input[$key eq substring(., 1, 1)] ! string-length(.))}
     ),
     $mapBuild(
       ('Wang', 'Liu', 'Zhao'),
       fn($name, $pos) { $name },
       fn($key, $input){index-of($input, $key)}
     ),  
     let $titles := 
       <titles>
        <title>A Beginner’s Guide to <ix>Java</ix></title>
        <title>Learning <ix>XML</ix></title>
        <title>Using <ix>XML</ix> with <ix>Java</ix></title>
      </titles>
     return
       $mapBuild($titles/title, 
                 fn($title){$title/ix}, 
                 fn($key, $input){filter($input, fn($elem){$key = $elem/ix})}
               ),
      $mapBuild(
        $employees//employee, fn{@ssn}, fn($key, $input){filter($input, fn($elem){$key = $elem/@ssn})}
      ),
      $mapBuild(
        $employees//employee, fn{@location}, fn($key, $input) {count(filter($input, fn($elem){$key = $elem/@location}))}
      ),
      $mapBuild(
        $employees//employee, fn{@location}, fn($key, $input) {max((filter($input, fn($elem){$key = $elem/@location}))/xs:decimal(@salary))}
      )
)

All results (executed with BaseX) are the expected, correct ones:

{"a":"banana","p":"apple","l":(),"e":"blueberry","r":("blueberry","cherry"),"i":(),"c":(),"o":(),"t":(),"b":"blueberry","n":"banana","u":(),"y":(),"h":()}
{}
{1:(1,4,7,10),2:(2,5,8),0:(3,6,9)}
{1:"one",2:"two",3:"three",4:"four",5:"five"}
{"J":("January","June","July"),"F":"February","M":("March","May"),"A":("April","August"),"S":"September","O":"October","N":"November","D":"December"}
{"a":12,"b":15,"c":6}
{"Wang":1,"Liu":2,"Zhao":3}
{"Java":(<title>A Beginner’s Guide to <ix>Java</ix></title>,<title>Using <ix>XML</ix> with <ix>Java</ix></title>),"XML":(<title>Learning <ix>XML</ix></title>,<title>Using <ix>XML</ix> with <ix>Java</ix></title>)}
{"1234567890":<employee name="Jim Nelson" location="New York" ssn="1234567890" salary="123456"/>,"0987654321":<employee name="Ann West" location="New York" ssn="0987654321" salary="99999"/>,"123454321":<employee name="Peter Smith" location="Seattle" ssn="123454321" salary="155223"/>,"5432198760":<employee name="Karen Johnson" location="Seattle" ssn="5432198760" salary="175000"/>,"9999999999":<employee name="Jonh Lagarde" location="Boston" ssn="9999999999" salary="145000"/>,"1111111111":<employee name="Samantha Weird" location="Boston" ssn="1111111111" salary="153000"/>}
{"New York":2,"Seattle":2,"Boston":2}
{"New York":123456,"Seattle":175000,"Boston":153000}

Does some general tidying up of the serialization text, but the main substantive changes are (a) to make HTML5 the default version, and (b) to make support for earlier versions effectively optional.

Please review carefully. Marking as editorial because I'm not sure any test cases need to change, but I might be wrong.

Fix #1889

Fix #1661

See also #747

As discussed in the issue, I wasn't happy with the idea of changing the coercion rules to allow strings to be provided where a QName is expected, because of the need to keep the namespace context around at run-time, and because of potential confusion about exactly what namespace context is used.

Instead I have gone back to the idea of introducing QName literals, using the simple syntax #EQName.

Examples:

error(#err:XPTY0004)
node-name($node) = #xml:space
format-number($num, #de)
load-xquery-module($module)?variables?(#myvar)
transform({'initial-template':#xsl:initial-template})
{'last': 'Kay', 'first': 'Michael', 'suffix':#fn:null}

Fix #1240 Fix #1972

This PR enables use of expressions such as $rectangle?area() - sum($rectangle?contents()?area()) which would previously have failed with a type error.

Fix #1973

Section §2.3.3.1 Static Analysis Phase mentions

A processor may raise a type error during static analysis if the inferred static type of an expression has no overlap (intersection) with the required type, and cannot be converted to the required type using the [coercion rules].

This should cross-refer to the more precisely defined concept of types being "substantively disjoint" - see §3.4.3.

A note in F+O under map:get states

map:get(map:get(map:get($map, 'employee'), 'name'), 'first') can be written as $map('employee')('name')('first').

That's technically correct: both these expressions will fail in the same way if $map does not contain an entry for the key employee. Unlike the lookup expression $map?employee?name?first which returns an empty sequence in this situation.

The rules for dynamic function calls (xpath, §4.5.3.1) state that $F($X) raises a type error if $F is an empty sequence.

I think it would be more useful if both map:get() and dynamic function calls were changed to have "empty if empty" semantics.

This is related to #1240 which goes further by allowing $F to be a sequence of function items.

Fix #1951

XQFO

• fn:fold-right has an obsolete change section saying that “The $action callback function accepts an optional position argument.”
• “then [the] operation will fail”
• remove whitespace before/after QName literals (#1982)
• fn:unparsed-binary: return type: xs:base64Binary?

(everyone: feel free to add notes, I’ll create a PR sometime later)

Change to use-xsi-schema-location (because the value is a boolean, not a location)

Fix #1952

Fix #1967

One of the examples for the new function fn:unparsed-binary uses the obsolete function name fn:binary-resource

Replaces #1945 which was approved by the CG, but had pull conflicts because of incidental editorial changes

Fix #1568

This is a continuation of the original issue https://github.com/qt4cg/qtspecs/issues/716, created almost 2 years ago, and having accumulated a lot of very useful discussion.

Now, when we have methods that are fields of records, it became practical to produce the record type entirely in code, and this is the base for the planned PR.

1. What it contains

• The standard record fields as originally published:

     initialized as xs:boolean,
     endReached as xs:boolean,
     getCurrent as %method fn() as item()*,
     moveNext as %method fn(*)

• The following 34 methods - this will form the signatures and formal definitions of the methods inside the documentation:

toArray := %method fn()
take := %method fn($n as xs:integer)
takeWhile := %method fn($pred as function(item()*) as xs:boolean)
skip := %method fn($n as xs:nonNegativeInteger)
skipWhile := %method fn($pred as function(item()*) as xs:boolean)
some := %method fn()
someWhere := %method fn($pred)
subrange := %method fn($m as xs:positiveInteger, $n as xs:integer)
chunk := %method fn($size as xs:positiveInteger)
head := %method fn()
tail := %method fn()
at := %method fn($ind as xs:nonNegativeInteger)
for-each := %method fn($fun as function(*))
for-each-pair := %method fn($gen2 as f:generator, $fun as function(*))
zip := %method fn($gen2 as f:generator)
concat := %method fn($gen2 as f:generator)
append := %method fn($value as item()*)
prepend := %method fn($value as item()*)
insertAt := %method fn($pos as xs:positiveInteger, $value as item()*)
removeAt := %method fn($pos as xs:nonNegativeInteger)
replace := %method fn($funIsMatching as function(item()*) as xs:boolean, $replacement as item()*)
reverse := %method fn()
filter := %method fn($pred as function(item()*) as xs:boolean)
fold-left := %method fn($init as item()*, $action as fn(*))
fold-right := %method fn($init as item()*, $action as fn(*))
fold-lazy := %method fn($init as item()*, $action as fn(*), $shortCircuitProvider as function(*))
scan-left := %method fn($init as item()*, $action as fn(*))
scan-right := %method fn($init as item()*, $action as fn(*))
makeGenerator := %method fn($provider as function(*))
makeGeneratorFromArray := %method fn($input as array(*))
makeGeneratorFromSequence := %method fn($input as item()*)
toSequence := %method fn()
emptyGenerator := %method fn()

• 90 tests/examples - with calls to all the methods - in normal and edge cases

2. Where to get the executable (with BaseX) code?

For everyone's convenience, you will find the complete executable code at the end of this issue/initial-comment. Alternatively, the code is available here: https://github.com/dnovatchev/Articles/blob/main/Generators/Code/generator.xpath

The latter will always contain the latest, up-to-date code. And, of course, please execute the code with BaseX, as I have done many times:

3. What this gives us:

1. Working with huge collections, that would otherwise be restricted by the available memory.
2. Deferred execution.
3. Handling collections containing unknown or infinite number of members

• A (next) member is produced only on request. No time is spent on producing all members of the collection.
• A (next) member is produced only on request. No memory is consumed to store all members of the collection.

4. Lazy evaluation - due to the above and also using the fold-lazy method (also described in this article)
5. Implementation of the original idea about Kollection - https://github.com/qt4cg/qtspecs/issues/910 .

4. What assistance is needed

I will greatly appreciate any recommendations on how to proceed with the actual PR:

• Can this be a single PR ?
• If this is too-big for a single PR, then how to proceed, like splitting it to pieces?
• Any observations and comments on the code itself.

5. References:

1. The original issue: Generators in XPath: https://github.com/qt4cg/qtspecs/issues/716
2. This article: Generators in XPath
3. The article defining fold-lazy : "Laziness in XPath. The trouble with fn:fold-right"

6. Complete, executable definition of the generator record

declare namespace f = "http://www.w3.org/2005/xpath-functions-2025";
declare record f:generator 
   ( initialized as xs:boolean,
     endReached as xs:boolean,
     getCurrent as %method fn() as item()*,
     moveNext as %method fn(*) (: as f:generator, :),
     toArray := %method fn()
     {
       while-do( [., []],
                function( $inArr) 
                { $inArr(1)?initialized and not($inArr(1)?endReached) },                 
                function($inArr) 
                { array{$inArr(1)?moveNext(), 
                        array:append($inArr(2), $inArr(1)?getCurrent())
                       } 
                 }         
       ) (2)
     },
     
     take := %method fn($n as xs:integer) 
     {
      let $gen := if(not(?initialized)) then ?moveNext()
                    else .
       return
         if($gen?endReached or $n le 0) then $gen?emptyGenerator()
          else
            let $current := $gen?getCurrent(),
                $newResultGen := map:put(., "getCurrent", %method fn(){$current}),
                $nextGen := $gen?moveNext()
             return
               if($nextGen?endReached) then $newResultGen
                 else
                   let
                       $newResultGen2 :=  map:put($newResultGen, "moveNext", %method fn() {$nextGen?take($n -1)}) 
                     return
                       $newResultGen2
      },
      
      takeWhile := %method fn($pred as function(item()*) as xs:boolean)
      {
        let $gen := if(not(?initialized)) then ?moveNext()
                      else .
         return
           if($gen?endReached) then $gen?emptyGenerator()
            else      
              let $current := $gen?getCurrent()
                return
                  if(not($pred($current))) then $gen?emptyGenerator()
                  else
                    let $newResultGen := map:put(., "getCurrent", %method fn(){$current}),
                        $nextGen := ?moveNext()
                     return
                        if($nextGen?endReached) then $newResultGen
                        else
                          let $newResultGen2 :=  map:put($newResultGen, "moveNext", %method fn() {$nextGen?takeWhile($pred)}) 
                           return $newResultGen2  
      },
     
     skipStrict := %method fn($n as xs:nonNegativeInteger, $issueErrorOnEmpty as xs:boolean) 
     {
            if($n eq 0) then .
              else if(?endReached) 
                     then if($issueErrorOnEmpty)
                           then error((), "Input Generator too-short") 
                           else ?emptyGenerator()
              else 
                let $gen := if(not(?initialized)) then ?moveNext()
                             else .
                  return
                    if(not($gen?endReached)) then $gen?moveNext()?skipStrict($n -1, $issueErrorOnEmpty)
                      else $gen?emptyGenerator()                 

     },
     skip := %method fn($n as xs:nonNegativeInteger) 
     {
       ?skipStrict($n, false())
     },
     
     skipWhile := %method fn($pred as function(item()*) as xs:boolean)
     {
        let $gen := if(not(?initialized)) then ?moveNext()
                      else .
         return
           if($gen?endReached) then $gen?emptyGenerator()
            else
              let $current := $gen?getCurrent()
               return
                 if(not($pred($current))) then $gen
                  else $gen?moveNext()?skipWhile($pred)                    
     },
     
     some := %method fn()
     {
       ?initialized and not(?endReached)
     },
     
     someWhere := %method fn($pred)
     {
       ?filter($pred)?some()
     },
     
     subrange := %method fn($m as xs:positiveInteger, $n as xs:integer)
     {
       ?skip($m - 1)?take($n - $m + 1)
     },
     
     chunk := %method fn($size as xs:positiveInteger)
     {
        let $gen := if(not(?initialized)) then ?moveNext()
                      else .
         return
           if($gen?endReached) then $gen?emptyGenerator()
           else
             let $thisChunk := $gen?take($size)?toArray(),
                 $cutGen := $gen?skip($size),
                 $resultGen := $gen => map:put("getCurrent", %method fn(){$thisChunk})
                                    => map:put("moveNext", %method fn(){$cutGen?chunk($size)})
              return $resultGen
     },
     
     head := %method fn() {?take(1)?getCurrent()},
     tail := %method fn() {?skip(1)},
     
     at := %method fn($ind as xs:nonNegativeInteger) {?subrange($ind, $ind)?getCurrent()},
           
     for-each := %method fn($fun as function(*))
     {
      let $gen := if(not(?initialized)) then ?moveNext()
                    else .        
       return
         if(?endReached) then ?emptyGenerator()
          else
           let $current := $fun(?getCurrent()),
                $newResultGen := map:put(., "getCurrent", %method fn(){$current}),
                $nextGen := ?moveNext()
            return
              if($nextGen?endReached) then $newResultGen
                else
                  let $newResultGen2 :=  map:put($newResultGen, "moveNext", %method fn() {$nextGen?for-each($fun)}) 
                     return
                       $newResultGen2                    
      },
      
      for-each-pair := %method fn($gen2 as f:generator, $fun as function(*))
      {
        let $gen := if(not(?initialized)) then ?moveNext()
                    else .,
            $gen2 := if(not($gen2?initialized)) then $gen2?moveNext()
                    else $gen2
         return
            if(?endReached or $gen2?endReached) then ?emptyGenerator() 
             else  
               let $current := $fun(?getCurrent(), $gen2?getCurrent()),
                   $newResultGen := map:put(., "getCurrent", %method fn(){$current}),
                   $nextGen1 := ?moveNext(),
                   $nextGen2 := $gen2?moveNext()
                return
                   if($nextGen1?endReached or $nextGen2?endReached) then $newResultGen
                     else
                       let $newResultGen2 := map:put($newResultGen, "moveNext", %method fn(){$nextGen1?for-each-pair($nextGen2, $fun)})
                         return
                           $newResultGen2                        
      },
      
      zip := %method fn($gen2 as f:generator)
      {
        ?for-each-pair($gen2, fn($x1, $x2){[$x1, $x2]})
      },

      concat := %method fn($gen2 as f:generator)
      {
        let $gen := if(not(?initialized)) then ?moveNext()
                    else .,
            $gen2 := if(not($gen2?initialized)) then $gen2?moveNext()
                    else $gen2,
            $resultGen := if($gen?endReached) then $gen2
                            else if($gen2?endReached) then $gen
                            else
                              $gen  => map:put(  "moveNext", 
                                                %method fn()
                                                 {
                                                 let $nextGen := $gen?moveNext()
                                                   return 
                                                     $nextGen?concat($gen2)
                                                 }
                                              )                                   
        return 
           $resultGen            
      },

      append := %method fn($value as item()*)
      {
        let $gen := if(not(?initialized)) then ?moveNext()
                    else .,
            $genSingle := $gen => map:put("getCurrent", %method fn(){$value})
                               => map:put("moveNext", %method fn(){?emptyGenerator()})
                               => map:put("endReached", false())
         return
           $gen?concat($genSingle)                    
      },
      
      prepend := %method fn($value as item()*)
      {
                let $gen := if(not(?initialized)) then ?moveNext()
                    else .,
                    $genSingle := $gen => map:put("getCurrent", %method fn(){$value})
                                       => map:put("moveNext", %method fn(){?emptyGenerator()})
         return
           $genSingle?concat($gen)  
      },
      
      insertAt := %method fn($pos as xs:positiveInteger, $value as item()*)
      {
        let $genTail := ?skipStrict($pos - 1, true())
         return
            if($pos gt 1)
              then ?take($pos - 1)?append($value)?concat($genTail)
              else $genTail?prepend($value)               
      },
      
      removeAt := %method fn($pos as xs:nonNegativeInteger)
      {
        let $genTail := ?skipStrict($pos, true())
          return
            if($pos gt 1)
              then ?take($pos - 1)?concat($genTail)
              else $genTail
      },
    
      replace := %method fn($funIsMatching as function(item()*) as xs:boolean, $replacement as item()*)
      {
        if(?endReached) then .
          else
            let $current := ?getCurrent()
              return
                if($funIsMatching($current))
                  then let $nextGen := ?moveNext()
                     return
                       . => map:put("getCurrent", %method fn() {$replacement})
                         => map:put("moveNext", %method fn() { $nextGen } 
                                  )
                  else (: $current is not the match for replacement :)
                    let $nextGen := ?moveNext()
                      return . => map:put("moveNext", 
                                           %method fn()
                                           {
                                             let $intendedReplace := function($z) {$z?replace($funIsMatching, $replacement)}
                                              return
                                                if($nextGen?endReached) then $nextGen
                                                else $intendedReplace($nextGen)
                                           }
                                        )
      },
      
      reverse := %method fn()
      {
        if(?endReached) then ?emptyGenerator()
          else
           let $current := ?getCurrent()
             return
               ?tail()?reverse()?append($current)
      },

      filter := %method fn($pred as function(item()*) as xs:boolean)
      {
             if(?initialized and ?endReached) then ?emptyGenerator()
              else
                let $getNextGoodGen := function($gen as map(*), 
                                             $pred as function(item()*) as xs:boolean)
                   {
                      if($gen?endReached) then $gen?emptyGenerator()
                      else
                        let $mapResult := 
                              while-do(
                                       $gen,
                                       function($x) { not($x?endReached) and not($pred($x?getCurrent()))},
                                       function($x) { $x?moveNext() }
                                       )   
                        return 
                          if($mapResult?endReached) then $gen?emptyGenerator()
                           else $mapResult                  
                   },
                   
                   $gen := if(?initialized) then . 
                             else ?moveNext(),
                   $nextGoodGen := $getNextGoodGen($gen, $pred)
                return
                  if($nextGoodGen?endReached) then $gen?emptyGenerator()
                  else
                    $nextGoodGen => map:put("moveNext", 
                                            %method fn() 
                                              {
                                                let $nextGoodGen := $getNextGoodGen(?inputGen?moveNext(), $pred)
                                                  return
                                                    if($nextGoodGen?endReached) then $nextGoodGen?emptyGenerator()
                                                    else
                                                      map:put(map:put($nextGoodGen, "moveNext", %method fn() {$nextGoodGen?moveNext()?filter($pred)}),
                                                                      "inputGen", $nextGoodGen
                                                              )
                                               }
                                           )
                                   =>
                                     map:put("inputGen", $nextGoodGen)
        },     
        fold-left := %method fn($init as item()*, $action as fn(*))
        {
          if(?endReached) then $init
            else ?tail()?fold-left($action($init, ?getCurrent()), $action)
        },
        
        fold-right := %method fn($init as item()*, $action as fn(*))
        {
          if(?endReached) then $init
            else $action(?head(), ?tail()?fold-right($init, $action))
        },
        
        fold-lazy := %method fn($init as item()*, $action as fn(*), $shortCircuitProvider as function(*))
        {
          if(?endReached) then $init
          else
           let $current := ?getCurrent()
             return
               if(function-arity($shortCircuitProvider($current, $init)) eq 0)
                 then $shortCircuitProvider($current, $init)()
                 else $action($current, ?moveNext()?fold-lazy($init, $action, $shortCircuitProvider))
        },
        
        scan-left := %method fn($init as item()*, $action as fn(*))
        {
          let $resultGen := ?emptyGenerator() 
                                => map:put("endReached", false())
                                => map:put("getCurrent", %method fn(){$init})
           return
             if(?endReached) 
               then $resultGen => map:put("moveNext", %method fn(){?emptyGenerator()})
               else
                 let $resultGen := $resultGen => map:put("getCurrent", %method fn(){$init}),
                     $partialFoldResult := $action($init, ?getCurrent())
                   return
                     let $nextGen := ?moveNext()
                      return
                        $resultGen => map:put("moveNext", %method fn()
                                              { 
                                                  $nextGen?scan-left($partialFoldResult, $action)
                                               }
                                              )            
        },
      
        scan-right := %method fn($init as item()*, $action as fn(*))
        {
          ?reverse()?scan-left($init, $action)?reverse()                         
        },
        
        makeGenerator := %method fn($provider as function(*))
        {
         let $gen := if(not(?initialized)) then ?moveNext()
                    else .,
              $nextDataItemGetter := $provider(0),
              $nextGen := if(not($nextDataItemGetter instance of function(*))) then $gen?emptyGenerator()  
                           else $gen?emptyGenerator()
                            => map:put("numDataItems", 1)
                            => map:put("current", $nextDataItemGetter())
                            => map:put("endReached", false())
                            => map:put("getCurrent", %method fn() {?current})
                            => map:put("moveNext",  
                                       %method fn() 
                                        {
                                          let $nextDataItemGetter := $provider(?numDataItems)
                                            return
                                              if(not($nextDataItemGetter instance of function(*))) then ?emptyGenerator()
                                              else
                                                . => map:put("current", $nextDataItemGetter())
                                                  => map:put("numDataItems", ?numDataItems + 1)
                                        }
                                       )
           return $nextGen                                                  
        },
        
        makeGeneratorFromArray := %method fn($input as array(*))
        {
          let $size := array:size($input),
              $arrayProvider := fn($ind as xs:integer)
                                {
                                  if($ind +1 gt $size) then -1
                                   else fn(){$input($ind + 1)}
                                }
           return ?makeGenerator($arrayProvider)
        },
        
        makeGeneratorFromSequence := %method fn($input as item()*)
        {
          let $size := count($input),
              $seqProvider := fn($ind as xs:integer)
                                {
                                  if($ind +1 gt $size) then -1
                                   else fn(){$input[$ind + 1]}
                                }
           return ?makeGenerator($seqProvider)
        },
        
        toSequence := %method fn() {?toArray() => array:items()},     
        
        emptyGenerator := %method fn() 
        {
          . => map:put("initialized", true()) => map:put("endReached", true())
            => map:put("getCurrent", %method fn() {error((),"getCurrent() called on an emptyGenerator")})
            => map:put("moveNext", %method fn() {error((),"moveNext() called on an emptyGenerator")})
        },      
     *
   );

let $gen2ToInf := f:generator(initialized := true(), endReached := false(), 
                              getCurrent := %method fn(){?last +1},
                              moveNext := %method fn()
                              {
                                if(not(?initialized))
                                  then map:put(., "inittialized", true())
                                  else map:put(., "last", ?last + 1)
                              },
                              options := {"last" : 1}
                             ),
    $double := fn($n) {2*$n},
    $sum2 := fn($m, $n) {$m + $n},
    $product := fn($m, $n) {$m * $n}
  return    
  (
    "$gen2ToInf?take(3)?toArray()",
    $gen2ToInf?take(3)?toArray(),
    "================",    
    "$gen2ToInf?take(3)?skip(2)?getCurrent()",
    $gen2ToInf?take(3)?skip(2)?getCurrent(),
    (: $gen2ToInf?take(3)?moveNext()?moveNext()?moveNext()?getCurrent(), :)
    "================",
    "$gen2ToInf?getCurrent()",
    $gen2ToInf?getCurrent(),
    "$gen2ToInf?moveNext()?getCurrent()",
    $gen2ToInf?moveNext()?getCurrent(),
    "================",
    "$gen2ToInf?take(5) instance of f:generator",
    $gen2ToInf?take(5) instance of f:generator,
    "==>  $gen2ToInf?skip(7) instance of f:generator",
    $gen2ToInf?skip(7) instance of f:generator,  
    "================",
    "$gen2ToInf?subrange(4, 6)?getCurrent()",
    $gen2ToInf?subrange(4, 6)?getCurrent(), 
    "$gen2ToInf?subrange(4, 6)?moveNext()?getCurrent()",
    $gen2ToInf?subrange(4, 6)?moveNext()?getCurrent(),
    "$gen2ToInf?subrange(4, 6)?moveNext()?moveNext()?getCurrent()",
    $gen2ToInf?subrange(4, 6)?moveNext()?moveNext()?getCurrent(),
    (: $gen2ToInf?subrange(4, 6)?moveNext()?moveNext()?moveNext()?getCurrent() :) (: Must raise error:)    
    "================",    
    "$gen2ToInf?subrange(4, 6)?head()",
    $gen2ToInf?subrange(4, 6)?head(),  
    "$gen2ToInf?subrange(4, 6)?tail()?head()",
    $gen2ToInf?subrange(4, 6)?tail()?head(),
    "$gen2ToInf?subrange(4, 6)?toArray()",
    $gen2ToInf?subrange(4, 6)?toArray(),
    "$gen2ToInf?head()",
    $gen2ToInf?head(),
    "==>  $gen2ToInf?tail()?head()",
    $gen2ToInf?tail()?head(),
    "================", 
    "$gen2ToInf?subrange(4, 6)?tail()?toArray()",
    $gen2ToInf?subrange(4, 6)?tail()?toArray(),
    "================",
    "$gen2ToInf?at(5)",
    $gen2ToInf?at(5), 
    "================",
    "$gen2ToInf?subrange(1, 5)?toArray()",
    $gen2ToInf?subrange(1, 5)?toArray(),
    "$gen2ToInf?subrange(1, 5)?for-each($double)?toArray()",
    $gen2ToInf?subrange(1, 5)?for-each($double)?toArray(),
    "$gen2ToInf?take(5)?for-each($double)?toArray()",
    $gen2ToInf?take(5)?for-each($double)?toArray(),
    "==>  $gen2ToInf?for-each($double)?take(5)?toArray()",
    $gen2ToInf?for-each($double)?take(5)?toArray(),
    "================",
    "$gen2ToInf?subrange(1, 5)?toArray()",
    $gen2ToInf?subrange(1, 5)?toArray(),
    "$gen2ToInf?subrange(6, 10)?toArray()",
    $gen2ToInf?subrange(6, 10)?toArray(),
    "$gen2ToInf?subrange(1, 5)?for-each-pair($gen2ToInf?subrange(6, 10), $sum2)?toArray()",
    $gen2ToInf?subrange(1, 5)?for-each-pair($gen2ToInf?subrange(6, 10), $sum2)?toArray(), 
    "==>  $gen2ToInf?for-each-pair($gen2ToInf, $sum2)?take(5)?toArray()",
    $gen2ToInf?for-each-pair($gen2ToInf, $sum2)?take(5)?toArray(),
    "================",
    "==>  $gen2ToInf?filter(fn($n){$n mod 2 eq 1})?getCurrent()",
    $gen2ToInf?filter(fn($n){$n mod 2 eq 1})?getCurrent(),
    "$gen2ToInf?filter(fn($n){$n mod 2 eq 1})?moveNext()?getCurrent()",
    $gen2ToInf?filter(fn($n){$n mod 2 eq 1})?moveNext()?getCurrent(),
    "================", 
    "$gen2ToInf?filter(fn($n){$n mod 2 eq 1})?take(10)?toArray()",
    $gen2ToInf?filter(fn($n){$n mod 2 eq 1})?take(10)?toArray(),  
    "================", 
    "$gen2ToInf?filter(fn($n){$n mod 2 eq 1})?take(10)?toSequence()",
    $gen2ToInf?filter(fn($n){$n mod 2 eq 1})?take(10)?toSequence(),
    "================", 
    "$gen2ToInf?takeWhile(fn($n){$n < 11})?toArray()",
    $gen2ToInf?takeWhile(fn($n){$n < 11})?toArray(), 
    "$gen2ToInf?takeWhile(fn($n){$n < 2})?toArray()",
    $gen2ToInf?takeWhile(fn($n){$n < 2})?toArray(), 
    "================", 
    "$gen2ToInf?skipWhile(fn($n){$n < 11})?take(5)?toArray()",
    $gen2ToInf?skipWhile(fn($n){$n < 11})?take(5)?toArray(),
    "==> $gen2ToInf?skipWhile(fn($n){$n < 2})",
    $gen2ToInf?skipWhile(fn($n){$n < 2}),
    "
     ==> $gen2ToInf?skipWhile(fn($n){$n < 2})?skip(1)",
    $gen2ToInf?skipWhile(fn($n){$n < 2})?skip(1),
(:    $gen2ToInf?skipWhile(fn($x) {$x ge 2}) :) (: ?skip(1) :)
    "================", 
    "$gen2ToInf?some()",
     $gen2ToInf?some(),
     "let $empty := $gen2ToInf?emptyGenerator()
      return $empty?some()",
     let $empty := $gen2ToInf?emptyGenerator()
      return $empty?some(),
    "================",
    "$gen2ToInf?take(5)?filter(fn($n){$n ge 7})?some()",
     $gen2ToInf?take(5)?filter(fn($n){$n ge 7})?some(),  
     "$gen2ToInf?take(5)?someWhere(fn($n){$n ge 7})",
     $gen2ToInf?take(5)?someWhere(fn($n){$n ge 7}), 
     "$gen2ToInf?take(5)?someWhere(fn($n){$n ge 6})",
     $gen2ToInf?take(5)?someWhere(fn($n){$n ge 6}),
     "$gen2ToInf?someWhere(fn($n){$n ge 100})",
     $gen2ToInf?someWhere(fn($n){$n ge 100}),
     "================",
     "$gen2ToInf?take(10)?take(11)?toArray()",
     $gen2ToInf?take(10)?take(11)?toArray(),
     "$gen2ToInf?take(10)?skip(10)?toArray()",
     $gen2ToInf?take(10)?skip(10)?toArray(),
     "$gen2ToInf?take(10)?skip(9)?toArray()",     
     $gen2ToInf?take(10)?skip(9)?toArray(),
     "$gen2ToInf?take(10)?subrange(3, 12)?toArray()",
     $gen2ToInf?take(10)?subrange(3, 12)?toArray(),
     "$gen2ToInf?take(10)?subrange(5, 3)?toArray()",
     $gen2ToInf?take(10)?subrange(5, 3)?toArray(),
     "================",
     "$gen2ToInf?take(100)?chunk(20)?getCurrent()",
      $gen2ToInf?take(100)?chunk(20)?getCurrent(),
      "==>  $gen2ToInf?chunk(20)?take(5)?toArray()",
      $gen2ToInf?chunk(20)?take(5)?toArray(),
     "================",
     "$gen2ToInf?take(100)?chunk(20)?moveNext()?getCurrent()",
      $gen2ToInf?take(100)?chunk(20)?moveNext()?getCurrent(),
     "$gen2ToInf?take(100)?chunk(20)?moveNext()?moveNext()?getCurrent()", 
      $gen2ToInf?take(100)?chunk(20)?moveNext()?moveNext()?getCurrent(),
     "$gen2ToInf?take(100)?chunk(20)?skip(1)?getCurrent()",      
      $gen2ToInf?take(100)?chunk(20)?skip(1)?getCurrent(),
     "================",      
     "$gen2ToInf?take(100)?chunk(20)?for-each(fn($genX){$genX})?toArray()",      
      $gen2ToInf?take(100)?chunk(20)?for-each(fn($genX){$genX})?toArray(),
     "================",  
     "$gen2ToInf?take(10)?chunk(4)?toArray()",
      $gen2ToInf?take(10)?chunk(4)?toArray(),
      "$gen2ToInf?take(10)?chunk(4)?for-each(fn($arr){array:size($arr)})?toArray()",
      $gen2ToInf?take(10)?chunk(4)?for-each(fn($arr){array:size($arr)})?toArray(),
     "================", 
     "$gen2ToInf?subrange(10, 15)?concat($gen2ToInf?subrange(1, 9))?toArray()",
     $gen2ToInf?subrange(10, 15)?concat($gen2ToInf?subrange(1, 9))?toArray(),
     "================", 
     "$gen2ToInf?subrange(1, 5)?append(101)?toArray()",
     $gen2ToInf?subrange(1, 5)?append(101)?toArray(),
     "$gen2ToInf?subrange(1, 5)?prepend(101)?toArray()",
     $gen2ToInf?subrange(1, 5)?prepend(101)?toArray(),
     "==>  $gen2ToInf?append(101)",
     $gen2ToInf?append(101),
     "$gen2ToInf?prepend(101)?take(5)?toArray()",
     $gen2ToInf?prepend(101)?take(5)?toArray(),
     "================", 
     "$gen2ToInf?subrange(1, 5)?zip($gen2ToInf?subrange(6, 10))?toArray()",
     $gen2ToInf?subrange(1, 5)?zip($gen2ToInf?subrange(6, 10))?toArray(),
     "$gen2ToInf?subrange(1, 5)?zip($gen2ToInf?subrange(10, 20))?toArray()",
     $gen2ToInf?subrange(1, 5)?zip($gen2ToInf?subrange(10, 20))?toArray(),
     "==>  $gen2ToInf?zip($gen2ToInf?skip(5))?take(10)?toArray()",
     $gen2ToInf?zip($gen2ToInf?skip(5))?take(10)?toArray(),
     "================", 
     "$gen2ToInf?makeGenerator(fn($numGenerated as xs:integer)
                                 {if($numGenerated le 9) then fn() {$numGenerated + 1} else -1} 
                             )?toArray()",
     $gen2ToInf?makeGenerator(fn($numGenerated as xs:integer)
                                 {if($numGenerated le 9) then fn() {$numGenerated + 1} else -1} 
                             )?toArray(),
     "================", 
     "$gen2ToInf?makeGeneratorFromArray([1, 4, 9, 16, 25])?toArray()",
      $gen2ToInf?makeGeneratorFromArray([1, 4, 9, 16, 25])?toArray(),
      "$gen2ToInf?makeGeneratorFromSequence((1, 8, 27, 64, 125))?toArray()",
      $gen2ToInf?makeGeneratorFromSequence((1, 8, 27, 64, 125))?toArray(), 
     "================", 
     "$gen2ToInf?take(10)?insertAt(3, ""XYZ"")?toArray()",
      $gen2ToInf?take(10)?insertAt(3, "XYZ")?toArray(),
      "$gen2ToInf?take(10)?insertAt(1, ""ABC"")?toArray()",
      $gen2ToInf?take(10)?insertAt(1, "ABC")?toArray(),
      "$gen2ToInf?take(10)?insertAt(11, ""PQR"")?toArray()",
      $gen2ToInf?take(10)?insertAt(11, "PQR")?toArray(),
      "==>  $gen2ToInf?insertAt(3, ""XYZ"")?take(10)?toArray()", 
      $gen2ToInf?insertAt(3, "XYZ")?take(10)?toArray(),
     (: , $gen2ToInf?take(10)?insertAt(12, "GHI")?toArray() :)  (:  Must raise error "Input Generator too-short." :) 
     "================", 
     "$gen2ToInf?take(10)?removeAt(3)?toArray()",
      $gen2ToInf?take(10)?removeAt(3)?toArray(),
      "$gen2ToInf?take(10)?removeAt(1)?toArray()",
      $gen2ToInf?take(10)?removeAt(1)?toArray(),
      "$gen2ToInf?take(10)?removeAt(10)?toArray()",
      $gen2ToInf?take(10)?removeAt(10)?toArray(),
      "==>  $gen2ToInf?removeAt(3)?take(10)?toArray()",
      $gen2ToInf?removeAt(3)?take(10)?toArray(),
      (: , $gen2ToInf?take(10)?removeAt(11)?toArray() :)        (:  Must raise error "Input Generator too-short." :) 
     "================",
     "$gen2ToInf?take(10)?replace(fn($x){$x gt 4}, ""Replacement"")?toArray()",
      $gen2ToInf?take(10)?replace(fn($x){$x gt 4}, "Replacement")?toArray(),
      "$gen2ToInf?take(10)?replace(fn($x){$x lt 3}, ""Replacement"")?toArray()",
      $gen2ToInf?take(10)?replace(fn($x){$x lt 3}, "Replacement")?toArray(),
      "$gen2ToInf?take(10)?replace(fn($x){$x gt 10}, ""Replacement"")?toArray()",
      $gen2ToInf?take(10)?replace(fn($x){$x gt 10}, "Replacement")?toArray(),
      "$gen2ToInf?take(10)?replace(fn($x){$x gt 11}, ""Replacement"")?toArray()",
      $gen2ToInf?take(10)?replace(fn($x){$x gt 11}, "Replacement")?toArray(),
      "$gen2ToInf?take(10)?replace(fn($x){$x lt 2}, ""Replacement"")?toArray()",
      $gen2ToInf?take(10)?replace(fn($x){$x lt 2}, "Replacement")?toArray(),
      "==> $gen2ToInf?replace(fn($x){$x gt 4}, ""Replacement"")?take(10)?toArray()",
      $gen2ToInf?replace(fn($x){$x gt 4}, "Replacement")?take(10)?toArray(),
      "$gen2ToInf?replace(fn($x){$x lt 3}, ""Replacement"")?take(10)?toArray()",
      $gen2ToInf?replace(fn($x){$x lt 3}, "Replacement")?take(10)?toArray(),
    (:  
      Will result in endless loop:
      
      , "==>  ==>  ==>  $gen2ToInf?replace(fn($x){$x lt 2}, ""Replacement"")?take(10)?toArray() <==  <==  <==",
      $gen2ToInf?replace2(fn($x){$x lt 2}, "Replacement")?take(10)?toArray() 
    :)
    "================",
    "$gen2ToInf?emptyGenerator()?reverse()?toArray()",
    $gen2ToInf?emptyGenerator()?reverse()?toArray(),
    "$gen2ToInf?emptyGenerator()?append(2)?reverse()?toArray()",
    $gen2ToInf?emptyGenerator()?append(2)?reverse()?toArray(),
    "$gen2ToInf?take(10)?reverse()?toArray()",
    $gen2ToInf?take(10)?reverse()?toArray(),
    "================",
    "$gen2ToInf?take(5)?fold-left(0, fn($x, $y){$x + $y})",
    $gen2ToInf?take(5)?fold-left(0, fn($x, $y){$x + $y}),
    "================",
    "$gen2ToInf?take(5)?fold-right(0, fn($x, $y){$x + $y})",
    $gen2ToInf?take(5)?fold-right(0, fn($x, $y){$x + $y}),
    "================",
    "$gen2ToInf?emptyGenerator()?scan-left(0, fn($x, $y){$x + $y})?toArray()",
    $gen2ToInf?emptyGenerator()?scan-left(0, fn($x, $y){$x + $y})?toArray(),
    "$gen2ToInf?take(5)?scan-left(0, fn($x, $y){$x + $y})?toArray()",
    $gen2ToInf?take(5)?scan-left(0, fn($x, $y){$x + $y})?toArray(),
    "================",
    "$gen2ToInf?makeGeneratorFromSequence((1 to 10))?scan-right(0, fn($x, $y){$x + $y})?toArray()",
    $gen2ToInf?makeGeneratorFromSequence((1 to 10))?scan-right(0, fn($x, $y){$x + $y})?toArray(),
    "================",
    let $multShortCircuitProvider := fn($x, $y)
        {
          if($x eq 0) then fn(){0}
            else fn($z) {$x * $z}
        },
        $gen-5ToInf := $gen2ToInf?for-each(fn($n){$n -7})
     return
     (
       "let $multShortCircuitProvider := fn($x, $y)
        {
          if($x eq 0) then fn(){0}
            else fn($z) {$x * $z}
        },
            $gen-5ToInf := $gen2ToInf?for-each(fn($n){$n -7})
          return
            $gen2ToInf?take(5)?fold-lazy(1, $product, $multShortCircuitProvider),
            $gen-5ToInf?fold-lazy(1, $product, $multShortCircuitProvider)",
       $gen2ToInf?take(5)?fold-lazy(1, $product, $multShortCircuitProvider),
       $gen-5ToInf?fold-lazy(1, $product, $multShortCircuitProvider)
     )
   )

Change to schema-for-xslt40

Fix #1957 (xsl:output disallow mixed content)

Add support for xsl:import-schema/@role

Fix #1958

As this feature request has been reported back to us more than once, I want to raise the question here if we want to introduce a function that inverts the result of fn:map-to-element back to an XML representation – provided that a conversion plan exists.

Many results would certainly be lossy, but as the plan is now available separately, it would be possible to roundtrip a lot of data with regular structures, without having to write custom conversion code.

By the simple expedient of adding

<e:attribute name="*" required="yes">
   <e:data-type name="expression"/>
</e:attribute>

to the syntax summary we get

With some additional prose, would that be sufficient?

This is an attempt to complete action QT4CG-116-03.

Part of the confusion in the rendering is that it’s partly done by CSS and partly done by JavaScript and those had gotten out of sync. Given that the JavaScript code has to do some of the work, I changed things so it does all of the work.

I also discovered that the weird Firefox bug where the font size changed was, wait for it, caused by the particular codepoint being used for “changed”. So I, uh, changed it.

We now get ✚ for new sections, ✭ for changed sections, and both when there are both new and changed sections. I spent a bit of time trying to find a third symbol, but gave up.

The markup in the XSLT specification isn’t quite the same as in other specifications, so the results are a tiny bit odd in places. There are some sections that get marked “both” in the ToC but when you expand the ToC, there’s only one mark. I haven’t tried to work out what’s going on there yet.

Provides an XSLT package that uses named record types and methods to implement an atomic set data type, as an example of how abstract data types can now be implemented.

If the key is already present, the processor combines the new value for the key with the existing value as determined by the and duplicates option.

The declaration (line 1412) says mixed="true".

See issue #1954

The PR removes the requirement for private variables and functions declared in library modules to be in the module namespace. There has never been any sensible reason for this restriction.

The restriction is retained for public variables and functions; one could argue that it is unnecessary in that case also, but it does no harm and enforces good coding discipline.

The current rule for the entity-expansion-limit option is:

The processor should impose a limit on the number of entity references that are expanded, or on the size of the expanded entities, depending on the options available in the underlying XML parser; the limit should be commensurate with the value requested, but the precise effect may be . implementation-dependent. If the XML parser does not offer the ability to impose a limit, or if the value is zero, then entity expansion should if possible be disabled entirely, leading to a dynamic error if the input contains any entity references. A negative value should be interpreted as placing no limits on entity expansion.

By default, Java uses 64000 as limit. An explicit value less than or equal to 0 indicates no limit (https://docs.oracle.com/javase/tutorial/jaxp/limits/limits.html, https://docs.oracle.com/en/java/javase/17/docs/api/java.xml/module-summary.html). I don’t know about other languages.

I would like to…

1. change the option to an expand-entities Boolean, or
2. change the rules and make 0 disable the limit.

Any favorites?

It would be nice to have some way of indicating that some of the fields in a record are (in some sense) private, intended for internal use.

I'm not proposing full encapsulation - the instances of a record type are maps, and can be manipulated by functions such as map:keys(), map:get(), and map:put() which expose all the keys.

Rather I'm proposing a convention that makes it difficult to access the fields "accidentally" using lookup expressions: a bit like naming the fields using a leading underscore, but something a bit stronger. Analogous to reflection in Java, which allows you to break encapsulation with a bit of effort.

I'd suggest making the keys for these "private" fields QNames rather than strings:

• In the record declaration, we allow a field name to be a QName rather than a string: record(private:data as item()*, long, lat).
• QNames can't be used directly in a lookup; to access the field, you need to know what namespace "private" is bound to, which doesn't need to be published information (though it is of course discoverable)
• Internally the implementor of this interface can bind a QName to a private variable and use this:

declare %private variable $private:data as xs:QName('http://my.private.namespace/', 'data')

and then access it using $record?$private:data

I propose that when a named record type is declared in XQuery or XSLT, the generation of a constructor function should be optional.

Perhaps in XQuery it should only happen if there is an annotation %constructor, and in XSLT if there is an attribute constructor="yes". I think it's better for the default to be "no constructor" because it's better to make the existence of a constructor explicitly visible.

There are cases where you don't want a system-generated constructor primarily because you want to provide your own constructor which perhaps accepts the data in a slightly different form, or perhaps imposes constraints like cross-validation of supplied arguments.

Functions (such as fn:doc and fn:parse-xml) that have a boolean option xsi-schema-location should change this to use-xsi-schema-location to make it clearer that the expected value is a boolean and not a schema location.

(Comment made in passing at the last meeting).

A few minor comments on the method attribute (that also apply to XSLT 3.0):

1. A note in section 25.1 says "In the case of the attributes method, cdata-section-elements, suppress-indentation, and use-character-maps, the effective value of the attribute contains a space-separated list of EQNames." The effective value of the method attribute should not be a list of values, just one value, right?

2. The XSD schema requires that the method attribute contain a colon if it is not one of the 6 "built-in" values. (The type xsl:method restricts xsl:EQName with the pattern "\c*:\c*"). Now that it can be an expanded QName, it should also allow for Q{...}.

3. A note in section 3.2 says: "Extension attributes may also be used to influence the behavior of the serialization methods xml, xhtml, html, or text, to the extent that... If a serialization method other than one of these four is requested (using a prefixed QName in the method parameter) then...".
   a. This lists only 4 methods, Should "json" and "adaptive" be added to the list and "four" changed to "six"? b. Should "using a prefixed QName" be changed to something like "using an EQName with a non-absent namespace"?

4. In the note about error XTSE1570 in Section 26.2. "The value must (if present) be a valid EQName. If it is a lexical QName with no a prefix, then it identifies a method specified in [XSLT and XQuery Serialization] and must be one of xml, html, xhtml, or text. a. It only lists the 4 methods, leaving out "json" and "adaptive" b. Typo - "no a prefix" should be "no prefix"

Thanks!

Fix #1704

The main substantive change is that unparsed-text() now explicitly discards any leading BOM.

Other functions that involve decoding of octets to strings are updated to reflect the changes that we made to reference the concept of "permitted characters".

My feedback is based on the latest version PR (#1906):

1. Boolean types

I think we should be careful about changing data to a representation that differs from the input data. If the input contains 0 and 1, it seems too invasive to me to return a boolean. Many users will not be aware that those numbers are valid candidates for Boolean conversions in XPath. That’s why I would still pledge for adapting the type rule detections, and placing numeric before boolean (related: 5).

Things are even more awkward (if I got the rules right) when working without a conversion plan:

(: Query :)
element-to-map(<x><a>1</a><a>2</a></x>)

(: Result :)
{ "x": [ true(), 2 ] }

2. Explicit types

I still feel uneasy that we ignore the specified type if it does not match – even more because XML is known for its rigor that documents must be well-formed to be accepted. I agree we should allow users to be lax about their generated output – by deliberately omitting types – but if type hints are supplied, I think we should take them serious.

An example:

element-to-map(
  <a>2</a>,
  { 'plan': { 'a': { 'layout': 'simple', 'type': 'boolean' } } }
)

3. Numeric casts

If the prescribed type is numeric and the value is castable as xs:numeric, then it is output as an instance of xs:integer, xs:decimal, or xs:double depending on the lexical form of the value, following the same rules as for XPath numeric literals.

Unless we use the same rule somewhere else in the spec, I would definitely vote for making things easier and choose consistency. xs:numeric(<a>1</a>) returns a double value, so I think we should do exactly the same here. If the result will be serialized as JSON, everything will be a number anyway.

4. Normalized space

2. If empty($EE/(* | text()) …the layout is empty

I still believe empty($EE/(* | text()[normalize-space()]) would be a better choice. The error sections for both empty and list state that “whitespace-only text nodes are discarded.”, so it is not clear to me why the rules for whitespace text nodes differ for these layouts.

5. 18.5.2 Creating a conversion plan

The current rules do not mention yet that child keys need to be added for list and list-plus.

In general, I would appreciate if redundancy could be removed. I’m still struggling finding all relevant information without resorting to the tests. For example, I think that due to the new XQuery code, a lot of informal and possibly lossy rules can be dropped.

6. Function signatures: document-node(), element()

Both functions should accept only elements, or accept both document nodes and elements. Maybe it’s better to only accept elements; it would resemble the name of the function.

7. deep-skip option

I wouldn’t be able to tell how a shallow-skip option could work, so maybe skip is sufficient?

8. Streamability

• The conversion is not streamable.

It is not clear (to me) what this means. Is this XSLT-related? Maybe a reference would be helpful, or we should drop the phrase if it’s not relevant anymore?

My observation was that fn:element-to-map can be implemented without keeping the full document in main-memory, so maybe we should let the processors decide what to do?

9. JSON

Example output: …shown as serialized JSON. The result is always shown as a singleton map…

This sounds contradictory, as there are no maps in the JSON terminology (but objects). Maybe there is no need to mention the JSON serialization, as the presented results are maps & arrays that can be run as XPath expressions out of the box.

10. Layout rules: errors

The error rules for empty and empty-plus say: “If any other child nodes are present, this layout fails.”. For the simple and simple-plus layouts, it is “If any child elements are present, this layout fails.”.

Am I right to assume that in both cases it’s only child element that result in a failure?

My feedback is based on the latest version PR (#1906) and the latest test cases (https://github.com/qt4cg/qt4tests/pull/223). I decided to list my observations in this repository, as I am not sure whether it’s the tests or the spec that may possibly need to be revised:

1. element-to-map-017:

As discussed in the last meeting (see also https://github.com/qt4cg/qtspecs/pull/1906#issuecomment-2821502378), the xsi:type attribute should already be ignored in the choice of the conversion plan, in order to choose a plan that does not include attributes:

element-to-map(parse-xml('<a xmlns="http://a.com/" xsi:type="xs:integer"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:xs="http://www.w3.org/2001/XMLSchema">2</a>')/*)
Result: { "Q{http://a.com/}a": 2 }
Expect: { "Q{http://a.com/}a": { "#content": "2" } }

2. element-to-map-401:

I would expect the record layout to be also applied for the b child node:

element-to-map(parse-xml('<a id="3"><b/></a>')/a, {'plan': {'a': {'layout': 'empty'},
  '*': {'layout': 'record'}, '@id': {'type': 'numeric'} }})
Result: { "a": { "@id": 3, "b": { } } }
Expect: { "a": { "@id": 3, "b": "" } }

3. element-to-map-420 and others:

As discussed in the meeting, list layouts require a child key:

element-to-map(parse-xml('<a id="zz"><b/><b/><c/></a>')/a, {'plan':{'a':{'layout':'list'}}})
Error : XPTY0004: Missing key 'child' (node: a).
Expect: FOJS0008

4. element-to-map-511:

I would expect the mixed layout to be also applied for the a child node:

element-to-map(parse-xml('<a>The <i>short</i> introduction</a>')/a, {'plan': {'a': {'layout':'simple'}, '*': {'layout':'mixed'} }})
Result: { "a": [ "The ", { "i": [ "short" ] }, " introduction" ] }
Expect: { "a": array { ("The ", { "i": "short" }, " introduction") } }

Fix #1936

We have two new great features in XPath 4.0 - the record and %method entries of a map.

It is natural to combine the two and have a record, one of whose entries is a method.

Unfortunately, at present there is no such example in the relevant Specs, and this leaves the reader trying to guess even the syntax of the specific record definition.

Therefore, we need at least one such example, to help readers and implementors in this.

BaseX did a great job in implementing both records and %method map-entries, and I constructed the following example, which is syntactically correct, but results in error (due to function coercion - explained in a separate issue here: https://github.com/qt4cg/qtspecs/issues/1938).

This code:

declare namespace t = "my:t";
declare record t:location 
   ( longitude as xs:integer,
     latitude as xs:integer,
     myFun2 as %method fn() as xs:integer,
     *
   );
   
   let $r := t:location(longitude := 25, latitude := 10, myFun2 := %method fn() {?longitude + ?latitude}
                        )
     return ($r, $r?myFun2())

When executed with BaseX, raises an error: [XPDY0002] .: Context value is undefined.

We need the specification to provide a similar example, and what the result should be.

In particular:

1. Must the return type of the method myFun2 (above) be specified or can it be omitted? BaseX raises a syntax error if this is specified as: myFun2 as %method fn() : "Expecting 'as', found ',' ?
2. If the method entry is specified as having a particular type (such as: as xs:integer) then can the corresponding function, provided in the construction of this record omit the function type (such as myFun2 := %method fn() {?longitude + ?latitude}) , even though it is clear that the type of the result is xs:integer ?

Fix #1568

I'm struggling a bit with try/catch/finally. Is there an implied constraint on the order of evaluation? The spec appears to suggest so:

its expression will be evaluated after the expressions of the try clause and a possibly evaluated catch clause.

What does evaluated after actually mean in a functional language? I can't see how to reconcile this with the general principles of the language regarding lazy evaluation etc. Is there some kind of exception to the general rule that you only have to evaluate as much of an expression as is needed to work out what the result is going to be?

It seems to me that the hidden unstated purpose of "finally" is to execute expressions that have side-effects, and without a proper semantic framework for handling side-effects, this is going to get us into trouble.

Resubmitted because of some clerical error...

Closes #37.

This currently only supports XPath. I'm working on the wording for XQuery.

Purely editorial. No issue raised.

Closes #1939

The finally clause is required to return an empty sequence. If not, it raises XQDY0153. This should be a type error rather than a dynamic error, so that it can be raised statically when appropriate.

Also noted in passing: "the the" in item 4 of the "changes" section of §4.20).

@ChristianGruen, in BaseXdb/basex#2420, brought up a test case similar to this:

declare record local:r(
    f as fn() as item()
);

local:r(%method fn() {.})
? f()
=> map:keys()

With a method passed to the constructor, and retrieved by the lookup operator, one would expect the function call to return the map, and the result to be f, the set of keys of the map. With BaseX's current implementation however, it fails with [XPDY0002] .: Context value is undefined.

The reason for this failure is function coercion. The record constructor asks for a more specific type than that of the supplied function, so it is subject to function coercion. This creates a new function item, which preserves the method annotation, and which effects a call to the original one. The lookup operator then is applied to the newly created function item, which as a method is equipped with the map as its context item. But when the original function gets called, that context item is not propagated to it. So there is a (rightful?) complaint about an undefined context.

I may be missing something here, but I do not see anything in the spec that makes the context item available to the coerced function, so I think that the described behavior is in fact conformant to the spec.

But as it contradicts the original expectation, I would be grateful for a clarification.

Fix #1936

The attributes in attribute group "literal-result-element-attributes" should be in the XSL namespace, and most of them are, except for xsl:default-mode, xsl:default-validation and xsl:expand-text. Those 3 are missing their form="qualified" attribute, so they would default to being in no namespace.

I know it's non-normative but it would be good to make the correction. (This is also an issue for the XSD for XSLT 3.0).

The doc-available function needs to make it clear what happens when invalid options are supplied.

Clearly invalid options such as xinclude="yes-or-no" should be an error (rather than resulting in a return value of false, as the current spec might suggest).

It's less clear what should happen if say you request schema validation with a non-schema-aware processor. I think this probably calls for returning false rather than an error.

At meeting 116, the question was raised: why don't we support RELAX NG validation?

I think that's a good question. Further, I think we should, if we can work out the technical details and arrive at consensus.

The good news is that it's a lot simpler than XSD validation. For those not familiar with RELAX NG, the 50,000 foot summary is that it's (more-or-less) regular expressions over trees. The grammar defines a number of patterns, including at least one designated as a start pattern. If the document matches (any one of the) start pattern(s), then it's valid. A trivial example looks like this:

start = doc

doc =
    element doc {
        attribute date { xsd:date },
        p+
    }

p =
    element p {
        text
    }

(RELAX NG also has an XML syntax, but the "compact syntax" is isomorphic and many people find it easier to read.)

A couple of things to note: the "p" in "p+" in the doc pattern is a reference to the "p" pattern, not to the element named "p". And although the date attribute has to conform to an xsd:date, that does not do any type assignment. RELAX NG allows user-defined data types; I suggest we make that an implementation-defined feature. Since no type assignment is performed, it doesn't really matter.

How might we add support for RELAX NG validation? A sketch...

1. Add to the static context a set of RELAX NG patterns. Initally empty, these are patterns that can match the document element during RELAX NG validation. (The union of all of the "start" patterns from all of the imported RELAX NG grammars.) There's no user-access to these patterns, so we don't technically need to add them to the data model, though I suppose we could.

2. In XQuery, allow schema import to import RELAX NG grammars. This has no effect except that the start patterns defined in that grammar are added to the start patterns in the static context. It is an error to specify “fixed” or “default element namespace” if the imported schema is a RELAX NG grammar.

3. Add “relax-ng” as a ValidationMode in ValidateExpr. It is an error to also specify a “type”.

   If RELAX NG validation is requested, the patterns in the static context are used to attempt to validate the document. If one succeeds, the validated document is returned. If none succeed, that’s an error.

4. In XSLT, allow schema import to import RELAX NG grammars.

• Details about role, TBD
• Details about literal schema elements, TBD

5. In XSLT, on elements that have a validation attribute, allow the value “relax-ng” with semantics analagous to the validate expression in XQuery.

6. In the F&O functions that have “dtd-validation” and “xsd-validation” options, add a boolean “relax-ng” validation option. If true, validation is done with the patterns in the static context.

There's one small wrinkle, the RELAX NG DTD Compatibility specification defines some annotations that allow a RELAX NG grammar to return default attributes. That means RELAX NG validation can return a different document than was validated, but I'm not sure how important that support is in 2025 if there were strong objections.

This proposal makes schema validation (as performed by the XQuery validate expression) available as a function. This allows additional options to be defined without extending the grammar, it makes it easier to incorporate validation within a pipeline of function calls, and it makes validation available from XPath.

If the proposal is accepted I would propose doing some editorial reorganisation so that the current XQuery and XSLT text describing the semantics of validation are directed to the definition of this function, reducing duplication in the specs.

Fix #1271

Adds an example demonstrating passing of parameter through a chain of xsl:next-match instructions

Improves the description of the semantics of the xsd-validation option on parse-xml() and doc(). Also brings the two functions into line by adding the xsi-schema-location option from doc() to parse-xml().

Fix a simple typo.

Fix #1724

This PR doesn't do what issue https://github.com/qt4cg/qtspecs/issues/1844 suggests, namely dropping the mapping arrow. Instead it picks up a couple of points made in passing in that issue:

(a) drops remaining references to the obsolete =?> operator

(b) simplifies the grammar for arrow expressions

(c) improves the way arrow expressions are described, including their relationship to pipeline expressions.

Fix #1907

Fix #1907

This PR doesn't do what issue #1844 suggests, namely dropping the mapping arrow. Instead it picks up a couple of points made in passing in that issue:

(a) drops remaining references to the obsolete =?> operator

(b) simplifies the grammar for arrow expressions

(c) improves the way arrow expressions are described, including their relationship to pipeline expressions.

Fix #1923

Minor editorial adjustment needed at XPath specs, 4.8 Arithmetic Expressions. The definition of UnionExpr is mentioned in the EBNF snippets at the top, but that definition is not discussed, nor should it be. This snippet should be dropped.

Fix #1921

Simple editorial bug fix.

In XSLT §3.5.1 The semantics of VersionTo and VersionFromTo are described as if the keyword to is always followed by a VersionPrefix, whereas the syntax allows a choice of a VersionPrefix or a PackageVersion.

This problem is present in XSLT 3.0.

See also Saxon bug https://saxonica.plan.io/issues/6746

Note also the absence of tests for the form to VersionPrefix.

The function fn:parse-xml is nondeterministic: Every function call may return a different node instance. Most other parse functions (fn:parse-json, fn:parse-csv, fn:csv-to-xml, etc) are deterministic, and I believe we should change that and make them nondeterministic as well.

We could also make fn:json-doc nondeterministic. If we don’t, we should probably add a stable option.

Closes #1905

Fix #1891 Fix #1799

partial fix for #1889

Fix #1891 Fix #1799

partial fix for #1889

Fix #1896

Replaces PR #1909

Adds error conditions for unpacking an integer that is too large for the implementation

Closes #501

1. Reinstates the non-capturing group syntax (?: xxx )
2. Clarifies that a zero-length matching segment does not overlap an immediately preceding adjacent (but non-zero-length) segment.

Adopted from #501:

In https://github.com/qt4cg/qtspecs/pull/493, a function/expression was suggested to re-throw errors:

try {
  (: wild stuff :)
} catch * {
  module:log($err:description),
  fn:throw($err:map)
}

Existing errors map can be modified before rethrowing them:

try {
  1 div 0
} catch * {
  module:log($err:description),
  fn:throw($err:map => map:put('description', 'Arithmetic error'))
}

I would like to share these observations that I made while working on recent changes of regular expression handling per #1856.

The section that mentions potential rewrites of \b and \B misses to consider the start and end of the string, as well as the empty string. It should rather read:

\b can be rewritten to an equivalent form in terms of lookbehind and lookahead assertions:

(?:(*positive_lookbehind:\w)(?:$|(*positive_lookahead:\W))|(?:^|(*positive_lookbehind:\W))(*positive_lookahead:\w))

A similar rewrite is possible for \B, but it must additionally take care of the empty string.

For fn:analyze-string, it might be useful to add a clarifying remark about empty matches at the end of the result (see qt4cg/qt4tests#224).

The specification of fn:analyze-string contains a duplicated word, the the. The same also occurs in several other places in the documents.

To follow: options for collection() and uri-collection().

Add error condition.

Fix #1902

I also did some work on removing errors and warnings from the EXPath binary build. There are a couple of outstanding issues I'm not sure how to fix:

(a) The function bin:bin had the incorrect id value func-bin-binary instead of func-bin-bin. I've corrected it, but the database of section ids needs updating.

(b) In database.xml, the EXPath binary spec is identified as document-summary/@uri = "https://qt4cg.org/specifications/EXPath/binary-40/". But the actual location of the specification is "https://qt4cg.org/specifications/expath-binary-40/"

(c) There are tags such as <code>bin:index-of-range</code> which the stylesheet is trying to interpret as function names rather than error codes. They actually refer to obsolete error codes so we can't use <errorref>

Fix #1520

We should ignore the %method annotation for wildcard lookups:

let $data := { 'fn': %method fn() { . } }
return $data?*

I cannot see when this makes sense as it only seems to work if the wildcard lookup returns a single item (→ “selects a key/value pair whose value part is a singleton method”). In addition, it makes streaming of wildcard results troublesome.

If we believe we should support context value bindings for wildcards, I think it would be better to apply it to each item of the returned value, instead of the value as a whole.

The PR drops the "uniform" option of elements-to-maps into a separate function elements-to-maps-conversion-plan, which can be used to analyze a corpus of data and generate a conversion plan for use by elements-to-maps. This is useful when the conversion is to be applied to documents that are not part of the corpus, for example when new documents arrive for conversion every day and need to be converted in a consistent way. It also provides a more general mechanism for users to override the system decisions on what layouts to use for what elements.

The PR is not entirely complete at this stage: the technical detail is all there, but examples need to be reviewed. Comments are welcome at this stage.

There are a few other minor changes. The most notable are:

• More consistent fallback when an inappropriate layout is chosen. If the layout does not allow attributes, then attributes are discarded; if there is any other mismatch, the converter falls back to serialized XML layout.
• Better handling of boolean and numeric element and attribute content.

XQFO:

• Buggy examples/results: map:put, map:of-pairs, fn:scan-left
• duplicates: the the, …
• Boolean defaults: true()/false() vs. true/false

…to be continued

Update the table and explanatory notes.

Fix #1832

I have labeled #982 (which included position arguments) to be closed to focus on the remaining todos:

1. The types of the $action parameters of fn:fold-right and fn:scan-right should be aligned. In particular, item()* and item() of the scan function should be swapped: → #1919

fn:fold-right(
  $input   as item()*,	
  $init    as item()*,	
  $action  as fn(item(), item()*) as item()*	
) as item()*

fn:scan-right(
  $input   as item()*,	
  $init    as item()*,	
  $action  as fn(item()*, item()) as item()*	
) as array(*)*

2. The result of the last example of fn:scan-left is syntactically wrong. → #1919
3. The equivalent array functions are still missing (if we still believe we want to include them).
4. We need tests.

If binary:unpack-integer or binary:unpack-unsigned-integer generates a value that exceeds the range supported by the implementation, err:FOAR0002 should be raised.

Related: https://github.com/expath/expath-cg/issues/116

Issue #1363 generated a large amount of discussion on how to handle absent keys in map:get() and out-of-range indexes in array:get().

I felt that one of the simplest proposals was to change the $fallback argument to be a simple default value, rather than a function. This eliminates some of the more "clever" use cases, but these can always be achieved in other ways, as the discussion thread demonstrates. Meanwhile reducing $fallback to a simple default value makes life easier for the 90% of cases where this is all that is needed (especially for arrays, when the desire is to return a default value rather than throwing an error).

This PR therefore implements that simple proposal.

Fix #1363

Continues #1862:

In the last meeting, we discussed whether the order of record entries should be considered in instance checks. After further reflection and attempts to implement it, I believe this will make things much easier in the long term:

In 3.4.1 Item Coercion Rules, the coercion of records was added as a second exceptional case: The coercion may change the item in question even if the upstream instance check is succesful. This leads to additional action that I believe could simply be avoided if the successful instance means that no further action is required. I think it will also reduce possible cost that was indicated in https://github.com/qt4cg/qtspecs/issues/1862#issuecomment-2709104860.

Note: This issue clearly focuses on implications for the implementation. From a user perspective, I assume it will hardly ever make a difference whether we consider order or not. My assumption is that nearly all records will have the expected order anyway, or they will match the order once the first coercion has taken place.

All this takes time to specify. I will be glad to make an attempt and write the PR.

Fix #1624

Fix #1876

One of the properties of function items is the parameter names.

This property is unused; there is nothing that depends on the value of this property, and no way of discovering the value, and it isn't defined for all function items, e.g. maps and arrays, or functions returned by functions such as fn:op. It causes complications, such as whether two functions can have the same identity if they have different parameter names. I propose to drop it.

Of course, there are open issues that suggest allowing parameter names to be used in dynamic function calls. But I see little chance of coming up with a design that achieves this, because in general when you're given a function item to call, you have no idea what the parameter names are, and the person supplying the function item has very little control over what the parameter names will be.

Supplies rules for how fn:function-identity() should handle maps and arrays.

Also makes the point that labels are ignored. There's a general statement to the effect in XDM that labels are ignored except where otherwise specified, but it's useful to avoid any doubt here.

Fix #1881

Re-submitted the same as PR 1890. Added some new examples to fn:chain.

I cannot imagine how we got a merged PR that included broken markup, but it's probably made a mess of the diffs recently.

This PR #1890 rebased off master to test if it makes for cleaner diffs.

Maybe we can align the HTML versions that fn:parse-html needs to support with the remaining specification. It currently says:

Valid values an implementation must support for the html method are: 3, 3.2 for HTML 3.2 W3C Recommendation, 14 January 1997 4, 4.01 for HTML 4.01 W3C Recommendation, 24 December 1999 5.0 for HTML5 W3C Recommendation, 28 October 2014 5.1 for HTML 5.1 W3C Recommendation, 1 November 2016 5.2 for HTML 5.2 W3C Recommendation, 14 December 2017 LS for HTML Living Standard, WHATWG 5 may be equivalent to any of 5.0, 5.1, 5.2, or LS

In the XQFO and Serialization specs, only HTML 4.0/4.01 and HTML 5 are mentioned.

@rhdunn Do you have an opinion on this?

Related: #1889

Added 6 more examples and tests All are correctly executed.

The serialization spec says (HTML Output Method: the version and html-version Parameters):

If the html-version serialization parameter is not absent, the requested HTML version is the value of the html-version serialization parameter; otherwise, it is the value of the version serialization parameter.

fn:serialize defines the following defaults:

• html-version: 5
• version: 1.0

I wonder whether these rules cover all possible cases:

1. Is it correct that HTML will be serialized as HTML5 if no options are supplied?

(: html-version=5 :) 
serialize(<html/>, { 'method': 'html' })

2. If only version is supplied, is it correct that it is ignored because of html-version defaulting to 5?

(: html-version=5 ? :) 
serialize(<html/>, { 'method': 'html', 'version': '4.01' })

3. If no, i.e., if { 'version': '4.01' } is expected to overwrite the default for html-version, how can we know at which stage the default values are to be considered?

In addition, the serialization specification mentions versions HTML 4.01 and HTML5 various times, but it seems to be up to the implementation to decide which HTML versions to support. However, we seem to have test cases for 4.0 and 5. Would it make sense to define a miminum set of versions that need to be supported?

Finally, for some reason, the html-version parameter was defined to be a decimal, whereas version is defined as a string (since XQFO 3.1). Maybe this leads to the surprising result that Saxon seems to accept the option { 'version': '4.0' }, but rejects { 'html-version': 4 }.

First draft, for initial feedback.

Notes:

• Because the CG has little energy/resources to develop the EXPath Zip module, I have situated the question of archive (compressed or not) in the URI scheme itself. There are dozens of archives, dozens of URI schemes. The only case where I have found overlap is in the jar: scheme/archive. Yes, I've seen zip: used as an alias for jar:, but it's not an official IANA URI scheme. This may need discussion.
• I have opted to bind @priority to a non-zero integer. This is the first time the constraint for the union of positive and negative integers has been placed on an XSLT attribute, so I may not have correctly set up element-catalog.xml.
• I have opted to not make attribute values format, name, and version as criteria for the priority package location (new term), so that developers can be warned when the package is at odds with the declaration. To make them criteria would mean that inconsistencies between the declaration and the referenced packages would remain undetected.
• I adopted the terms "URL" and "entry" based upon the IANA nomenclature for the jar: scheme.
• I may have overthought the distinction between archive and non-archive URIs. Feedback is appreciated.
• Error code 3000 has been broken up into different possible errors.
• Suggestions on the type and number of tests that need to be written for the test suite are welcome.

Fix #1870

The grammar for regular expressions in the regular expression section of F&O is currently defined as a code block. Making it use the grammar markup used to define the pattern, XPath, and XQuery grammars would:

1. give the grammar a unified appearance with the other grammars;
2. allow grammar elements to be cross referenced and linked back to the grammar.

Issue #119 proposes extending maps to allow arbitrary values as keys. This is very difficult to achieve, (a) because the fact that keys are atomic items is deeply embedded in the design of a number of functions and operations on maps, and (b) because it's very hard to define an equality function that suits everyone.

The way we tacked variable equality semantics for strings was via the collation-key() function, which takes a string and a collation as input and produces an opaque key value, which can be used as a key in maps, and which reflects the desired equality semantics.

We could extend the same idea to values other than strings. In particular, we could define a deep-equality-key() that can be calculated for any sequence, and that takes all the matching options of the deep-equal() function as a parameter. (We could then redefine deep-equal(a, b, options) to mean deep-equality-key(a, options) eq deep-equality-key(b, options)).

The main drawback is that the deep-equality-keys for large node trees or maps would be rather long strings. People might use the functionality without realising the expense.

Another problem is that one of our options in deep-equals() is a callback function for item equality, and we couldn't replicate this when computing a key. But this callback is the only way we have, for example, to compare nodes by identity rather than by content.

Note that an internal deep-equality-key concept (or at least a deep-equality hashcode) is needed anyway for efficient implementation of deep-equals where order is deemed irrelevant. Without it, the function becomes O(n^2). Quite independently of this proposal, we should perhaps have an explicit option on deep-equals() to compare nodes by identity.

Drops the existing fn:chain function and replaces it with a new fn:compose function.

This combines two separate changes:

(a) whereas fn:chain applies a sequence of functions to an input, fn:compose returns a composite function that can be used repeatedly with different inputs.

(b) the fn:compose function is restricted to arity-1 functions, which leads to a much simpler specification that still handles the vast majority of practical use cases.

In particular, note that if the sequence of functions to be applied is statically known, then it can always be written out explicitly; the real use case for this function is when the sequence of functions is constructed dynamically. And in this situation, fn:chain in its current form can easily fail because of problems with the arity of the functions included in the chain.

This is intended to be purely an editorial rewrite, it does not change the functionality.

Replaces #1296.

Addresses #982, but we still need to add corresponding functions for arrays.

The data model spec says that function identity is not defined for maps and arrays.

The specification of fn:function-identity() fails to mention this fact.

Tidies up the text and adds examples

Add options to control entity expansion and XInclude processing.

Fix #1857 Fix #1860

Fix #1851

Allow ?variety to be absent e.g. for xs:anySimpleType

Define namespace-sensitive by an xtermref to the definition in the XP/XQ spec.

Fix #1866

We could combine the competing $replacement and $action parameters:

replace(
  'this is a test',
  '(\w)(\w+)?',
  fn($s, $g) { upper-case($g[1]) || lower-case($g[2]) }
)

Original comment: https://github.com/qt4cg/qtspecs/issues/1863#issuecomment-2711149296

Fix #1861

Fix #1862

Fix #1869

I am pretty sure the first reaction will be DONT!, but for the sake of consistency it seems important enough for me to bring this up:

Could we rename array “members” to “values”?

Some advantages that I would see:

• We could treat arrays and maps more similarly.
• We already have a values lookup key specifier for arrays.
• No 3.1 array function contains the string “member”, so we will not introduce any backward inconsistencies.
• All 4.0 features that use this string could be safely renamed.

Of course the term “values” is a very common one, but we have to decided to stick with “map values” – and arrays and maps are very similar.

Finally, I noticed that also the term “member” has different meanings in the spec and is not exclusively used for arrays (e.g. in the rules for fn:innermost, fn:format-integer or for members of union types).

Suggestions (based on #1338, related: #1868)

1. In symmetry with the pairs lookup specifier, we should add array:pairs and an inverse array:of-pairs function.
2. In symmetry with the values lookup specifier, we should add array:values and map:values functions, to retrieve the values of maps and the members of arrays as a sequence of arrays.
3. In return, array:members and array:of-members seem redundant, and we should drop them.
4. In analogy with the keys specifier and map:keys, we should add array:keys (which returns a dense integer range).

Background

With version 4.0, we are adding a lot of promising and powerful new map and array features. This is a big step forward, compared to the obvious limitations of 3.1.

Some aspects of the 3.1 design have made it difficult (or impossible) to fully adjust array and maps, but (in my opinion) the old overall concept was impressively consistent – and it is definitely a big challenge to achieve a 4.0 design that is not too fragmented.

To me, this becomes particularly evident in the case of arrays. The following example sums up the items of all members of an array. For the cumbersome 3.1 solution…

for $pos in 1 to array:size($array)
return sum($array($pos))

…we now have several (roughly?) equivalent options to do this:

1. for member $m in $array return sum($m)
2. array:members($array) ! sum(?value)
3. $array?pairs::* ! sum(?value)
4. $array?values::* ! sum(.)

The examples above imply that:

• for 1., an array member is a sequence;
• for 2., an array member is a map;
• for 3., an array has pairs (but there is no array:pairs);
• for 4., an array has values (but there is no array:values).

I find the name $zero for this parameter unhelpful and confusing.

I suggest $accum, short for "accumulator" or "accumulated result".

With https://github.com/qt4cg/qtspecs/pull/987, a rule was added to fn:duplicate-values:

For any set of values that compare equal, the one that is returned is the one that appears first in $values.

I think we should adapt the behavior to return the duplicates in the order they appear, not the original values:

• A common use case for this function is to find the first duplicate in a list.
• If we return the original values in the correct order, we need to parse the full sequence before we can know which will be the first result. A worst-case example:

(0x7FFFFFFFFFFFFFFF, 1, 1 to 0x7FFFFFFFFFFFFFFF)
=> duplicate-values()
=> head()

Currently array:members(["a", "b"]) returns

{'value': "a"},
{'value': "b"}

I suggest that it should instead return

{'key': 1, 'value': "a"},
{'key': 2, 'value': "b"}

The extra information is useful for any operation that wants to take account of positions as well as values. For example, rearranging an array into multiple columns. Using the names "key" and "value" also means that the data is suitable for converting an array to a map by means of map:of-pairs.

The function array:of-members() should change to accept record('value', *) (making the record type extensible) so that the key part is ignored if present.

Following up on issue 1341, we decided to drop the position argument from the 4 fold functions.

Most of the changes in this PR are dealing with the collateral damage - changes to "formal equivalents" of other functions that previously relied on fold-left having the position available to the callback function.

The grammar check done by RExification of XQuery and XPath 4.0 Grammars has detected a bunch of LALR(2) conflicts caused by the recent addition of TypeSpecifier to the KeySpecifier production.

In fact these are ambiguities between following being used as a QName (via EQName, TypeName), or as a keyword:

• array
• attribute
• comment
• document-node
• element
• empty-sequence
• enum
• fn
• function
• item
• map
• namespace-node
• node
• processing-instruction
• record
• schema-attribute
• schema-element
• text

E.g. element in

$A?~element()

can be parsed as an element test, element(), or as a type name element followed by a PositionalArgumentList. This is similar to what the "reserved-function-names" constraint covers, but that does not apply here because there is no function name involved.

The SequenceType in TypeSpecifier, enclosed in extra parenthese, does not present a problem, so my proposal is to drop ItemType from TypeSpecifier and rewrite the production to

TypeSpecifier
         ::= '~' '(' SequenceType ')' 

• In https://github.com/qt4cg/qtspecs/pull/1735#issuecomment-2715090198, it was decided to remove the position argument from fn:fold-left and fn:fold-right. → #1867
• As maps are ordered now, we should add the position argument to iterative map functions (e.g., map:for-each; basically all functions for which equivalent sequence and array functions exist).

Fix #1456

Technically identical to PR #1778, but reworked because it had become impossible to resolve the merge conflicts.

Many systems using regular expressions support case conversion in the replacement strings.

For example,

sed -e 's/[aA]*/\L\u&/'

given AAA as input, produces Aaa.

It’s not 100% clear to me its worth adding, since an action function can do the same thing with more or less work, but for reference,

\U turns the replaced text into upper case until \E, \L, or the end of the replacement string \L turns the replaces text to lower case in the same way \u and \l affect the single next character and operate independently of \U, \L, \E.

I wrote up some more precise spec text and can make a pull request; the case in the sed example above is common in text conversion projects but slightly tricky to get right with a function,

          fn { upper-case(substring(., 1, 1)) || lower-case(substring(., 2) }

This is simple, but consider \2 \L\u\1\3 as a function, where \1 may be empty.

Overall i don’t have strong feelings either way, except that supporting them may help people migrate from other systems or languages. \E feels uncomfortably procedural. In Perl and libpcre i think, \E also turns of \Q (which disables all metadata characters up until \E).

Like < and > in patterns, \L and friends can be emulated with some care, but that’s true of a lot of regular expression syntax, and one point of the shorthands (as i see it) is to move the feature towards being accessible by people with less of a programming background.

I think we should make the order of record entries part of instance checks and coercion rules:

1. It will be less confusing for users if records have a well-defined order (similar to objects in OOL), in particular if records are serialized.
2. It will be much easier for implementations to access record entries by their internal index if the order is statically known. There will still be opportunities for optimizing lookups in arbitrary maps (index-based access has generally become easier with maps being ordered).

Problem

The <xsl:next-match> instruction is useful when writing local templates to customize the behavior of an imported XSLT. Unfortunately, there is a limitation due to the fact that <xsl:next-match> does not pass along parameters unless the parameters are defined as tunneling or the parameters are explicitly coded using <xsl:with-param>.

The fact that <xsl:next-match> does not automatically pass along parameters can be surprising or lead to cumbersome workarounds, and limits how <xsl:next-match> can be used when writing local templates to customize the behavior of imported XSLT.

• In situations where parameters are defined in an imported XSLT it might not be feasible to change parameters to tunneling.

• In situations where a variety of parameters might be in scope when a template that uses <xsl:next-match> is invoked, currently each parameter needs to be explicitly coded using <xsl:param> and <xsl:with-param> in <xsl:next-match>, even though the parameters might not be relevant to the purpose or logic of the template. This may lead to fragile and less maintainable code and increases the cognitive load for developers, especially when working with complex, multi-layered stylesheets.

This proposal aims to simplify the use of the <xsl:next-match> instruction while being backwards compatible.

Proposal

• Add an option to <xsl:next-match> to enable passing along all parameters. This option might take the form of a new optional attribute on <xsl:next-match> named with-all-params (this name is similar to the existing element name <xsl:with-param>) that takes a yes/no (or Boolean) value and defaults to no (false).

• An instruction <xsl:next-match with-all-params="no"/> would operate the same as <xsl:next-match/> currently does.

• An instruction <xsl:next-match with-all-params="yes"/> would operate the same as <xsl:next-match/> currently does with the difference that all parameters that were in scope when the current template was invoked will remain in scope for the next matching template.

• An instruction <xsl:next-match with-all-params="yes"> that contains <xsl:with-param> should operate the same as described in the preceding paragraph with the difference that parameters defined by <xsl:with-param> will also be in scope for the next matching template.

• If a parameter defined by <xsl:with-param> within <xsl:next-match with-all-params="yes"> has the same name as a parameter that was in scope when the current template was invoked, then the effective value of that parameter should be the value defined by <xsl:with-param>. This will allow a template to override parameters when necessary.

To summarize, <xsl:next-match with-all-params="yes"> should invoke the next matching template and automatically pass along all parameters that were in scope when the current template was invoked, and optionally allow using <xsl:with-param> to set additional parameters or modify parameter values.

Example

Given this input document:

<!-- input.xml -->
<section>
    <p>hello</p>
</section>

This stylesheet import.xsl provides a set of base templates. The template matching element "p" uses <xsl:next-match/> in it's current (default) operation.

<!-- import.xsl -->
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="3.0" expand-text="yes">
    
    <xsl:output indent="yes"/>
    <xsl:mode on-no-match="shallow-copy"/>
    
    <xsl:template match="section">
        <section>
            <xsl:apply-templates>
                <xsl:with-param name="a" select="'a'"/>
                <xsl:with-param name="b" select="'b'"/>
            </xsl:apply-templates>
        </section>
    </xsl:template>
    
    <xsl:template match="p">
        <xsl:param name="a"/>
        <xsl:param name="b"/>
        <xsl:param name="c"/>
        <p>a {$a}</p>
        <p>b {$b}</p>
        <p>c {$c}</p>
        <xsl:next-match/>
    </xsl:template>
    
</xsl:stylesheet>

This is the output of the above stylesheet import.xsl and the input document:

<section>
   <p>a a</p>
   <p>b b</p>
   <p>c </p>
   <p>hello</p>
</section>

This stylesheet before.xsl imports the stylesheet import.xsl and defines a template to customize how <p> elements are processed. The parameter $a needs to be intercepted and forwarded even though this template is not doing anything with $a. The parameter $b is overridden, and the parameter $c is added within <xsl:next-match>.

<!-- before.xsl -->
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="3.0">
    
    <xsl:import href="import.xsl"/>
    
    <xsl:template match="p">
        <xsl:param name="a"/>
        <p>customization</p>
        <xsl:next-match>
            <xsl:with-param name="a" select="$a"/>
            <xsl:with-param name="b" select="'buzz'"/>
            <xsl:with-param name="c" select="'c'"/>
        </xsl:next-match>
    </xsl:template>
    
</xsl:stylesheet>

This stylesheet after.xsl does the same thing as the previous stylesheet but uses with-all-params="yes". The template does not need to intercept and forward the parameter $a because this is handled automatically by with-all-params="yes". The parameter $b is overridden, and the parameter $c is added within <xsl:next-match> in the same way as the previous stylesheet. Although this is a small example in which the parameter $a is the only savings, the benefit of with-all-params="yes" can be significant in scenarios where there are more parameters.

<!-- after.xsl -->
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="4.0">
    
    <xsl:import href="import.xsl"/>
    
    <xsl:template match="p">
        <p>customization</p>
        <xsl:next-match with-all-params="yes">
            <xsl:with-param name="b" select="'buzz'"/>
            <xsl:with-param name="c" select="'c'"/>
        </xsl:next-match>
    </xsl:template>
    
</xsl:stylesheet>

The two stylesheets above (before.xsl and after.xsl) should produce the same output.

<!-- output.xml -->
<section>
   <p>customization</p>
   <p>a a</p>
   <p>b buzz</p>
   <p>c c</p>
   <p>hello</p>
</section>

The text doesn’t say much about what DTD validation means. Is my assumption correct that it boils down to a SAXParserFactory.setValidating call in Java?

What about DTDs in general? Given the following snippets (using the default false for DTD validation)…

<!-- xml.dtd -->
<!ENTITY arrow "→">

parse-xml(`
  <!DOCTYPE xml SYSTEM 'xml.dtd'>
  <xml>&arrow;</xml>`
)

…should the result be <xml/>, <xml>→</xml>, or an error? In other words, should the (potentially external) xml.dtd resource be resolved and interpreted?

Maybe we should introduce an additional DTD option (or options?) to control the loading of external DTDs and the handling of entities, for example:

http://apache.org/xml/features/nonvalidating/load-external-dtd
http://xml.org/sax/features/external-general-entities
http://xml.org/sax/features/external-parameter-entities

Thoughts are welcome.

Per #1280, fn:apply has been changed to allow the number of arguments to be greater than the arity of the function.

fn:chain is defined in terms of fn:apply, and it also refers to the error code err:FOAP0001 which belongs to fn:apply.

However the condition for the error differs between the two:

• fn:chain

  An error [err:FOAP0001] is raised if the arity of any function $f in $functions is different from the number of members in the array that is passed to fn:apply.

• fn:apply

  A dynamic error is raised if the arity of the function $function is greater than the size of the array $arguments ([err:FOAP0001]).

Also in B Error Codes, err:FOAP001 still asks for the arity to exaclty match the number of arguments:

err:FOAP0001, Wrong number of arguments.

Raised when fn:apply is called and the arity of the supplied function is not the same as the number of members in the supplied array.

Should not the description of fn:chain and the error summary be adapted to the changed behaviour of fn:apply?

An initial draft of the xsl:record instruction, for discussion