Under xsl:iterate, there are rules defining what it means for an instruction to be in a tail position in a sequence constructor. In these rules xsl:switch should be treated the same way as xsl:choose.

Currently predicate callback functions used by things like fn:filter have to return a boolean; they can't rely on EBV semantics. For example you have to write fn{boolean(self::p)} rather than fn{self::p}.

It's probably with self tests that this is most noticeable, because self is so often used in a boolean context.

Of course the current rule gives stricter type-checking which will presumably catch some user errors. But it seems an unnecessary inconsistency.

• fn:splice examples expanded to illustrate integer steps other than 1.
• fn:unparsed-text-lines updated to reflect recent decisions about line handling in fn:unparsed-text.
• Other clarifications or corrections.

Let me know if any of these edits are misfires.

Edit (2023-01-04): See https://github.com/qt4cg/qtspecs/issues/917#issuecomment-1875712638 for the most promising suggestion resulting from the discussion in this thread.

Inspired by #720 and concerns regarding usability and performance, it may be a big step, but couldn’t we define records as subtypes of maps?

• The main difference would be that updates on records are only allowed as long as the resulting map matches a record definition.
• This would allow us to return much better error messages, and to prevent users from deconstructing their own data structures.
• We could still benefit from the existing map functions… provided that we believe it's an advantage. A stricter solution would be to disallow optional map entries completely (and treating records as a separate type).
• From a technical point of view, data with a fixed structure can be optimized much better than a structure that changes dynamically.

This proposal allows functions within maps to access the containing map using the variable $this.

The proposal needs editorial work to integrate it fully into the text, but it is intended to be sufficiently complete to enable a full technical review.

Fix #720

In 4.6.2.5 Inline Function Expressions we refer to the "implementation" property of a function item; this property has been renamed as the "function body".

From reading up through chap. 13. The change in the title of chap. 10.2 is more accurate, and avoids the repetition of the title of chap. 10 itself.

In the preamble of XQFO chap. 13, several paragraphs are spent introducing a tree structure example, defining variables $po, $item1, $item2, and $item3. The prose leads the reader to expect frequent invocation of this tree example, but chap. 13 never uses it.

Chapter 13's tree example is referred to (sans link) and summarized in the preamble of chap. 14. In that chapter it is used only once, in a fn:count example that really doesn't rely upon anything special in the tree example.

Variables $item1 and $item2 are invoked only once more: in chap. 4, for fn:number. It's rather out of the blue, because neither the function definition nor chapter 4's preamble say anything about what the variables mean.

My recommendation is to drop this material all together, and for the functions fn:count and fn:number replace the examples with simpler examples.

OTOH, I might have come across an incomplete implementation, and the editors might prefer to make more thorough use of this tree example. I don't know.

Editorial; examples fixed

I have an open action: QT4CG-052-06: MK to consider the editorial question of “promotion” for the symmetric relations.

I think the point that led to this was the fact that the word "promotion" seems inappropriate for cases like (string/uri) where the implicit conversion can take place in either direction.

I'd like to propose a fix to this that is not merely editorial. I propose that we allow any cast from one numeric type to another in the coercion rules. For example, if the required type is decimal, then a double or float can be supplied. Since, for many implementations of xs:decimal, this can be done losslessly, it makes at least as much sense to convert from double to decimal as from decimal to double.

The word "promotion" is in fact used (in relation to the coercion rules) only in a table heading, and we can change this heading to "implicit casting".

Appendix B.1 currently says:

B.1 Type Promotion

[Definition: Under certain circumstances, an atomic value can be promoted from one type to another.] Type promotion is used in a number of contexts:

It forms part of the process described by the [coercion rules], invoked for example when a value of one type is supplied as an argument of a function call where the required type of the corresponding function parameter is declared with a different type. It forms part of the process described in [B.2 Operator Mapping]), which selects the implementation of a binary operator based on the types of the supplied operands. It is invoked (by explicit reference) in a number of other situations, for example when computing an average of a sequence of numeric values (in the fn:avg function).

and I suggest we retain the term only for the second case, operator mapping. This differs from the coercion rules in that there are two operands and the effect is always to convert one to the type of the other. This affects numeric types only (not string/uri or binary), and it will continue to promote decimal to double, decimal to float, and float to double.

Where functions (fn:avg, fn:sum, math:pow) refer to the promotion rules, I suggest that we spell out the conversions that happen explictly, since it's not entirely obvious how the rules should be extrapolated. (For example, fn:avg doesn't make it entirely clear what should happen if the first item is a decimal, the second is a float, and the third is a double. Are you expected to "look ahead" to see what types are present, rather than evaluating the average incrementally?)

The base for this issue is the email sent by @ndw to public-xslt-40@w3.org on Dec. 13th 2023, fully quoted below:

Hello all,

After a couple of weeks of discussion[1][2] about naming things, there seem to be a some quite different perspectives on the problem.

As background, let’s remember that we have a language (or a set of languages) that evolved over time. We couldn’t anticipate in version 1.0 what we would have in 4.0. We added new features in 2.0 and 3.0 that weren’t anticipated in previous versions either.

We live with decisions (some the result of long and hard battles within the working group(s)) like the fact that sequences don’t nest so all individual items are also sequences of length one.

The context for each addition to the language has been roughly: how can we add new, useful features with a minimum of backwards incompatibility.

It’s a natural consequence of this sort of evolution that there are rough edges. Why does fn:count returns the number of items in a sequence but always returns 1 if the argument is an array? Because an array is an item and an item is a sequence of length one.

(It doesn’t help that the vision of what the X* languages should be has changed over time. What started out envisioned as a tool for transforming documents from one format to another for presentation on the web or in print has grown into something that at least some members of the group view as first class, functional programming languages. That’s not bad, but it puts entirely different stresses on the design, I think.)

As we add new functions (specifically, in the case of recent discussions, but I expect the same perspectives apply more generally), I think one perspective is roughly this:

How can we name and organize the functions so that users are least likely to be surprised and most likely to be able to figure out how to solve a particular problem?

Taken to an extreme, this perspective isn’t about changing the semantics of the functions at all, it’s “just” about naming them. Is fn:get() better (easier to understand, less confusing) than fn:items-at?

I think another perspective is roughly this:

We have a messy design. It would be better if we could refactor the design so that it was more harmonious and logical. We don’t need four different, closely related functions to get items out of different sorts of data structures, we need a set of abstractions that make it obvious that only one function is necessary.

Taken to an extreme, this perspective is about reshaping the whole language so that a single, obvious set of function names emerges naturally from the carefully constructed abstractions.

I don’t think anyone holds exactly one perspective (discussions about renaming often involve some level of discussion about semantics, for example) and I’m attempting to polarize the perspectives a little bit in an effort to shine light on a larger problem, not to be divisive.

With my chair’s hat on, the main problem I see with the first perspective is that naming is hard, often personal and emotional, and will never be wholly logical (so there will always be more to discuss, so the “problem” is never resolved). It’s not quite fair to say it’s a distraction from the “bigger” issues we need to resolve, but it does take a lot of time.

I see the appeal of the second perspective. If we had a green field, we’d do things differently. I think we might all agree that, ideally, fn:count should return the number of items in a sequence, the number of items in an array, and the number of key-value pairs in a map. But it doesn’t and it can’t without fundamentally breaking things. I don’t think we’d get agreement to break fn:count, so what can we do?

A proposal to fundamentally redesign the data model would be a tough sell, I think.

One thing we could do is define a new namespace “gn” with functions that work more logically, that treat sequences, arrays, and maps, as collections and operate on them uniformly.

I suppose we could reconstruct the whole set of functions in this new namespace and focus our efforts there, perhaps going so far as to deprecate the current fn: namespace in favor of this new one. But could we get consensus to do that? Would users thank us?

I dunno. Innovations welcome.

                                    Be seeing you,
                                      norm

This issue addresses the 2nd alternative formulated briefly by Norm as:

One thing we could do is define a new namespace “gn” with functions that work more logically, that treat sequences, arrays, and maps, as collections and operate on them uniformly.

I suppose we could reconstruct the whole set of functions in this new namespace and focus our efforts there, perhaps going so far as to deprecate the current fn: namespace in favor of this new one. But could we get consensus to do that? Would users thank us?

Here are some of the obvious advantages of having a uniform Kollection concept that covers: arrays, sequences, maps, ... and possibly future new, specific, collection-like datatypes as sets:

1. Uniform definition and understanding of a single data type - the Kollection.

2. O(N) functions only, compared to O(M * N) at present. Here N is the number of functions needed for each of the current collection-like data types (Arrays, Sequences and Maps) and M is the number of collection-like data types (currently 3).

3. The users will need to know about and understand just the single Kollection data type and its functions, not 3 or more similar collection-like data types and 3 or more number of similar (but different) functions. Minimizing by a factor of 3 the amount of factual knowledge that a user needs is something HUGE and extremely positive.

4. Allowing users to say "Good Bye" to the unclear and treacherous flat-sequence concepts we have as legacy from XPath 1.0.

5. Staying aligned to the examples of other modern programming languages such as C# with its IEnumerable interface. It is good to know that this has already been done in other shining programming languages, thus a nay-sayer will not be able to argue that this is not doable or, if done, would be negative to the language and its users.

6. Freeing enormous resources and time for the members of the Community Group so that they can spend this on more valuable avenues, than trying to find similar and best names to M similar functions each defined to one of the M current collection-like data types.

Now, to dispel some plausible myths before they start circulating here:

1. Myth 1: This will break backwards-compatibility? No, as proposed by Norm, all the functions operating on the generalized collection data type can be in a separate, new namespace and thus no existing user-code is affected.

2. Myth 2: If a sequence containing a single Kollection still has count() of 1, then what is the use of the Kollection data type? Actually, as proposed by Norm, the Kollection data type and its functions reside in their own namespace. Doing things using only functions from this new namespace eliminates the possibility of using fn:count as it resides in the different, currently existing standard function namespace.

3. Myth 3: This will be too-complex for the users and the users will not embrace it, so let us not waste time designing it. Wow, there were such prophets saying exactly the same about LINQ in 2005. As it often happens, the future proved them wrong. Users clearly and overwhelmingly "voted with their code" incorporating LINQ in almost all everyday applications and code repositories.

4. Myth 4: Banning the current functions operating on sequences, arrays and maps would be a huge burden to the users, and would intervene negatively with their programming. In fact, nobody would be banning any of the existing functions. Users can continue to use them forever. The acceptance of the uniform and generalized Kollection data - type can happen gradually with time, as was the case with the addition of LINQ to C#.

In #520, the concept of function identities was introduced. This is what the current draft says:

XDM, 2.9.4 Function Items

identity: an abstract property that can be used to test whether two variables refer to the same function or to different functions. This property is exposed only for this purpose.

Note: Currently, the concept of function identity is used for two purposes: firstly, when functions appear in the arguments supplied to the fn:deep-equal function; and secondly, in establishing whether the arguments and results of a function are "the same" when deciding whether the function is deterministic.

Note: Function identity is not currently defined for maps and arrays, because in the circumstances where function identity would otherwise be used, maps and arrays are compared by examining their content.

XQFO, 1.8.4 Properties of functions

5. […] the two function items have the same function identity. The concept of function identity is explained in Section 2.9.4 Function Items.

XQFO, 14.2.8 fn:deep-equal

c. $i1 and $i2 have the same function identity. The concept of function identity is explained in Section 2.9.4 Function Items.

XQFO, 17.1.1 fn:function-lookup

The function identity is determined in the same way as for a named function reference. Specifically, if there is no context dependency, two calls on fn:function-lookup with the same name and arity must return the same function.

While I definitely believe in the concept, I believe the documentation is still cryptic, or even impossible, to understand, at least without reading #520 or consuming the existing QT4 test cases. Here are some questions that I’m trying to answer:

• Does “abstract property” mean that the property will not be materialized in an implementation, or does it mean that the property too vague to be precisely defined?
• We should try to specify what “refer to the same function” means. Are function properties that allow us to at least safely identify a subset of functions the same? For example, will true#0 and true#0 always be identical? The test cases imply this, whereas #520 doesn’t.
• In XQFO 17.1.1, there’s a hint that context-dependency influences the decision if functions are identified as equal. Does this mean that name#0 and name#0 cannot be equal? Or can, or will, they be equal if the context is identical?
• The term “deterministic” does not appear anywhere else in the XDM spec, so one is inclined to think of the XQFO nondeterminism. It is then unclear whether two instances of, for example, map:entries and fn:parse-xml can be “the same” if the parameters are the same.
• In XQFO 17.1.1, “must return the same function.” is also unclear: What exactly is meant by “same function”? Is it a function that creates an identical result (thus, excluding nondeterministic functions like fn:parse-xml)?
• We should explain better what was the motivation to include the context-dependency of functions in the definition. It would certainly be more intuitive if both deep-equal(name#0, name#0) and deep-equal(name#1, name#1) returned true.

I’m sorry for not offering good answers in return. I could try to describe what we’ve implemented so far – mostly inspired by the test cases – but I’m not sure if it meets the requirements.

Related: #333

As already suggested in https://github.com/qt4cg/qtspecs/pull/798#pullrequestreview-1709271106 (and by first user feedback), the upcoming PR renames the option unordered to ordered, with true as default.

Fix #898

Changes non-normative text for the doc() and document-uri() functions to make it clearer that the consequences of the normative rules are not quite as previously stated.

In addition: Lexicographic order; formatting.

Closes #900. In addition, the equivalent expression for array:sort was fixed.

Closes #895

The sort functions now accept multiple collations, keys, and orders, and this needs to be reflected in the parameter names (which are still singular).

I may misunderstand something but I always find the use of types and "as" to be counter intuitive (I'd prefer to be able to run an xslt 3+ script in some sort of 'strict' mode that was a bit more rigid, but thus simpler) e.g.

consider

      <xsl:variable name="foo1">
         <foo/>
      </xsl:variable>

question - what is the type of foo1? answer - (according to my saxon/oxygen setup the answer is) "document-node"

consider

      <xsl:variable name="foo2" as="element(foo)">
         <foo/>
      </xsl:variable>

question - is this code valid then (I would as someone not used to xslt 2+ assume not)? answer - yes

but surely this code is identical to foo1, so the 'type' of variable is actually changing the interpretation of the expression.

For me that's quite confusing

It would appear that these 2 values are not two different views (interfaces) of the same underlying value, else this

<xsl:variable name="foo3" as="element(foo)" select="$foo1"/>

would be valid.

It isnt (i.e. this doesn't appear to be some subtle OO style scenario where an evaluation can have multiple interfaces, here 'document-node' and 'element' are presumably disjoint types).

For me conceptually types are descriptions of expressions, they have no behaviour, yet here they appear to (to me) effect the interpretation, not simply describe it.

For me, I'd prefer a 'strict' mode where either

      <xsl:variable name="foo2" as="element(foo)">
         <foo/>
      </xsl:variable>

is a type error, because the expression is clearly a document-node OR some other mechanism to clarify the ambiguity without this conceptual wrinkle.

An expression should either have 1 interpretation, or if its ambigious, that should be an error, I don't think the language should default to prefer one over another.

P.S. why doesnt this work? I genuinely don't know how to explicitly declare something as a document-node.

  <xsl:variable name="foo1" as=document-node()>
     <foo/>
  </xsl:variable>

The specification of document-uri() states that the returned URI must be useable as input to the doc() function and that it always round-trips, so doc(document-uri($X)) is $X is guaranteed true.

A consequence of this rule is that you can't have two documents in the same execution scope with the same document-uri() property.

Enforcing this rule causes a lot of trouble. For example at the API level, the user can set two stylesheet or query parameters to two different documents that are associated with the same URI. Another example, two XSLT packages in the same stylesheet can call doc() on the same URI and get different documents back because they have set different validation and whitespace-stripping options. Use of fn:transform() causes further complications when documents are passed across the boundary (in fact that's where we first encountered the problem). And collection() brings further complications.

In order to conform to the rule in the spec, we changed Saxon a while back so the only documents that are guaranteed to have a document-uri() property are those that were read using the doc() function - and even then, things like validation and whitespace variations are troublesome. This causes users much confusion, partly, because of the change from earlier Saxon releases, but more because there are many situations where they would expect document-uri() to give a useful result and it doesn't.

I think we can fix this simply by removing the guarantee. That will cause less inconvenience to users than the current rule. We could perhaps modify it to say that in the case of a document returned from the doc() function, its document-uri() property will be such that a call to doc() with that URI will return the same document, but in the case of documents derived from other sources (for example collection(), or a result from fn:transform) there is no such guarantee.

Fix #894

Fix #887

We need a consistent approach for defining types of optional function arguments. In most current cases, if a function argument is supplied, it must be non-empty:

map:get(
  $map       as map(*),
  $key       as xs:anyAtomicType,
  $fallback  as function(xs:anyAtomicType) as item()*  := void#1
) as item()*

fn:starts-with-sequence(
  $input        as item()*,
  $subsequence	as item()*,
  $compare      as function(item(), item()) as xs:boolean  := fn:deep-equal#2
) as xs:boolean

In some cases, it’s optional:

fn:replace(
  $value        as xs:string?,
  $pattern      as xs:string,
  $replacement  as xs:string?                                                   := (),
  $flags        as xs:string?                                                   := '',
  $action       as (function(xs:untypedAtomic, xs:untypedAtomic*) as item()?)?  := ()
) as xs:string)

(: #874 :)
fn:subsequence(
  $input   as item()*,
  $start   as xs:double?                                     := (),
  $length  as xs:double?                                     := (),
  $from    as (function(item(), xs:integer) as xs:boolean)?  := (),
  $while   as (function(item(), xs:integer) as xs:boolean)?  := (),
  $until   as (function(item(), xs:integer) as xs:boolean)?  := ()
) as item()*

As a result, map:get($map, fallback := ()) is invalid, while replace($string, $pattern, action := ()) would be valid.

I think it’s better to enforce non-empty arguments (provided that a single item is expected).

Follow-up to issue #888

There are a number of situations in which error behaviour is insufficiently specified:

• Consider the partial function application fn:contains(?, 23), where the second argument is an integer, not a string. Does the partial application fail, or does it result in a function item that fails when dynamically evaluated? The spec does not say.
• Consider the expression function-lookup(fn:name, 0) evaluated when there is no context item in the dynamic context. Does the call on function-lookup fail, or does it return a function item that fails when dynamically evaluated? The spec does not say, though there are numerous test cases in the function-lookup test set that suggest the latter.
• The same is true for the expression fn:name#0 evaluated when there is no context item in the dynamic context.
• Consider a user-defined function my:f with a parameter that has a default value of "."; consider both a function reference my:f#0 and a partial application my:f(?) evaluated when there is no context item. Again, I think the spec is unclear on the error behaviour.

It feels to me that the right thing to do in all these cases is to raise the error early: that is, to fail at the point where a function item is being constructed, not at the point where the function item is subsequently evaluated. However, this disagrees with QT3 expected test results for tests such as fn-function-lookup-267 and function-literal-267.

Inspired by #866:

We should extend fn:compare to support arbitrary atomic types. The comparison rules…

• would be unchanged for strings,
• would rely on fn:numeric-compare for numbers, and
• would rely on the existing op: functions for the remaining types.

For example, the rule for dates would be:

• 0 is returned if op:date-equal(A, B) is true,
• -1 is returned if op:date-less-than(A, B) is true,
• 1 is returned otherwise.

Some types will be rejected (xs:duration, xs:QName, xs:NOTATION, Gregorian types).

In addition, I would vote for making fn:numeric-compare and fn:atomic-equal private. I don’t see a benefit to expose them; I rather expect people to be confused.

Related to #888. The examples in 4.6.2.2 Evaluating Dynamic Function Calls…

…are misleading; all of them raise XPDY0002 if no element is bound to the global context value. Maybe shop could be replaced with $shop or doc('wares.xml')/shop.

Improved, I think.

Trying to track down the spurious diffs that we see in PRs.

I'll have to merge this to test it, so there will be a few random merges here. Sorry.

The term "named function reference" is used for a construct like name#1.

Although we define it as "A named function reference is an expression (written name#arity) which evaluates to a [function item]", the term "named function reference" perpetuates the incorrect assumption that it is some kind of literal or constant denoting a function item.

Of course, in many cases it can be treated as just that. But not when the function is context-dependent, for example name#0 or lang#1.

The term is also questionable because one would assume that a "named function reference" is a reference to a "named function", but there is no such concept as a "named function".

So what might be a better name? What the expression actually does (when evaluated) is to search the static context for a function definition whose name and arity range correspond, and then construct a function item that captures the relevant part of the dynamic context in its closure. It's hard to encapsulate all of that in a simple name for the construct, but I would suggest named function generator. This is sufficiently close to the current term to be recognisable, but tries to capture the fact that it's not just a constant or literal, it's an expression that activately does something when evaluated; and it's reasonably accurate in that the result of the evaluation is a function item that has a non-absent name.

I propose to reclassify XPDY0002 (context item is absent) as a type error rather than a dynamic error.

I don't propose to change the error code.

The only practical distinction is that this will allow the error to be reported statically when it can be detected statically, for example if the user writes something like

function($x as node()) {
  starts-with(name(), 'x')
}

At present Saxon will give you a compile-time warning for this, followed by a run-time error if the code is actually executed; this is the required behaviour for dynamic errors.

The change does mean that in a case like this example, it will no longer be possible to catch the error using try/catch. However, type errors can only be reported statically if the code is bound to fail at run-time, and catching errors that occur every time is not especially useful.

In the XPath/XQuery book, §4.6.2.4,

let $f := <foo/>/fn:name#0 return <bar>/$f()

should be

let $f := <foo/>/fn:name#0 return <bar/>/$f()

Digging a bit deeper, this reveals that we are not properly tagging and syntax-checking code examples in the spec.

We have made xs:hexBinary and xs:base64Binary comparable and we now allow implicit coercion between the two types.

I've been assuming, though I'm not sure we ever discussed it, that this automatically means that the two types can be "atomic equal" from the point of view of entries in maps: that is, a hexBinary representation of a particular binary value can no longer coexist in a map with a base64Binary representation of the same binary value.

If we were starting from scratch this would clearly make sense, but it has some messy implications:

• It's a backwards incompatibility; in 3.1 you could construct maps that you can no longer construct in 4.0
• It potentially affects interoperability of 3.1 and 4.0 applications. For example, an XQuery 4.0 application invoking an XSLT 3.0 transformation via fn:transform might get back a map that's not a valid map in 4.0.

In effect, this is not just a change to the behaviour of one function/operator, it is a data model change, because it changes the value space of the map(*) data type.

And more parochially, I freely admit, there's a lot of internal complexity trying to maintain a code base that supports both the 3.1 and 4.0 rules simultaneously.

Is this a feature that benefits users sufficiently to justify the transition complexities? Note that we can still support "eq" between the two data types without supporting fn:atomic-equal.

…to create a random universally unique identifier (UUID), represented as 128-bit value.

Should ideally be nondeterministic, or we may need to do something that’s similar to fn:random-number-generator.

Fix #862

The return type is given as map(*). We could make it more precise with a record type.

The same goes for a number of other function signatures that currently use map(*) as an argument or result type.

Perhaps we should also define a more precise type for options parameters.

I thought I had a great opportunity to use fn:chain the other day, and then found it didn't do what I wanted.

I wanted to negate a predicate: in pseudo-code

items-where($seq, not contains(?, "e"))

and I thought I could do this by chaining contains and not. But it doesn't work that way: fn:chain applies a sequence of functions to an argument, it doesn't compose a sequence of functions to yield a new function.

I wonder if a function that composes functions would be more useful, so I could write

items-where($seq, compose((contains(?, "e"), not#1)))

Fix #866

The proposal introduces a new fn:numeric-compare function that differs from lt/eq primarily in that decimals are compared retaining their full precision, rather than converting them to doubles which may lose precision. This makes the comparison fully transitive which makes it safe to use in all sorting algorithms.

The new comparison semantics are exploited in max(), min(), and sort(), and indirectly in highest() and lowest(); they are also referenced for comparing numeric values in XSLT xsl:sort (and therefore indirectly in xsl:merge) and in XQuery order by.

An effect of the change is that max() and min() applied to a sequence of integers now return an integer, not a double.

Closes #872.

Closes #844. The items keyword in the function names (excluding items-at) has been changed to subsequence.

See #878 for the controversial discussion on what to do with subsequence-(after|before|starting-where|ending-where).

Copied from https://github.com/qt4cg/qtspecs/issues/844#issuecomment-1841415417:

I'm thinking again about integrating the items-* quartet into a heavily overloaded subsequence function.

Must supply zero or one of:

• $start - the start position
• $from - a predicate, such that the start is the first item to match the predicate
• • defaulting to 1

And zero or one of:

• $length - the number of items to include
• $while - a predicate, the subsequence takes items so long as the predicate is true
• $until - a predicate, the subsequence takes items up to and including the first for which the predicate is false
• • defaulting to the end of the sequence.

This omits the "items-after" combination, but that one is easily achieved using tail(subsequence(from:="xxx")).

The rules for op:hexBinary-less-than() appear to define a recursive octet-by-octet operation, but I think it flounders in rule 3, where it does not ask for rule 2 to be applied seriatim to each octet pair, but asks for an en masse comparison of two octet sequences.

Compare to 5.3.2, Unicode Codepoint Collation, which describes a similar recursive item-for-item comparison. Interesting formal differences (e.g., unordered list versus ordered list).

fn:deep-equal() is similar, but it is also much more complex. Nevertheless, the way it breaks down the problem at the outset, to dispense immediately with the recursive factor, and deal simply with the rules for equality, is IMO admirable.

It would be nice if there were a bit more consistency in the prose and presentation of recursive rules. Do others agree, and are there other functions/operations that should be considered in this question? I'm thinking immediately only of comparator functions/operations, not functions that use recursion to filter or create. (There may be parallels, but let's start with those functions that are most similar.)

Currently fn:in-scope-namespaces(), fn:in-scope-prefixes(), and fn:namespace-uri-for-prefix are filed under XQFO chapter 10, which purports to deal exclusively with QNames. But these three functions have no direct bearing on QNames in either input or output.

Two options occur to me:

1. Move sections 10.2.6-8 to fall after 13.3.
2. Rename chapter 10 to "Functions related to QNames and namespaces". Create a 10.3 that pertains exclusively to namespaces. Move to this new subchapter 10.2.4, 10.2.6-8, as well as 13.3 fn:namespace-uri() .

Or some variant of the above.

At any rate, I think the current placement doesn't properly expose these functions to the browsing reader.

Hopefully nothing controversial here. Edits are motivated by consistency and clarity.

Following discussion under issue #844, I decided to explore the possibility of extending subsequence() with optional parameters, with the aim of making the quartet of items-before/after/starting-with/ending-with unnecessary.

This is the spec that results. I feel it's a good trade-off; by adding three optional parameters to fn:subsequence, we can eliminate 4 functions that we are having trouble finding names for. The examples feel to me to be intuitive and readable; and there is more capability in the new function than we had before, for example by combining a predicate for the start position with an integer for the length.

I haven't explored arity-2 callbacks - these certainly need some notes and examples.

Fix #865

This PR:

• Adds a non-normative appendix to XPath and XQuery comparing and contrasting the different ways of doing equality comparisons
• Changes fn:atomic-equal so it no longer refers to fn:deep-equal (the recursion terminated, but was confusing to follow)
• Removes text in XQuery describing the non-transitivity of group by clauses, which is now a solved problem
• Corrects the description of backwards incompatibilities relating to numeric comparisons in the F+O spec.

I think that fn:items-at should be changed to fn:get:

• In #843, we try to harmonize the function names across sequences, maps, and arrays. We have array:get and map:get to retrieve single entries of the input, but we have fn:items-at for sequences.
• fn:items-at allows you to supply more than a single position, but items-at($seq, (1, 3, 2) can easily be rewritten to (1, 3, 2) ! get($seq, .) – similar to (1, 3, 2) ! array:get($array, .) and (1, 3, 2) ! map:get($map, .).
• With #844, fn:items-at would be the only function left with items in its name.

The function signature would be as simple as:

fn:get(
  $input  as item()*,
  $at     as xs:integer	
) as item()

Obviously, most people will still use $input[$at] – but the same applies to arrays and maps (and other functions like fn:head). One of the advantages of fn:get is that you can pass on the context item as position argument.

Fix #867

The fourth example of fn:for-each-pair is wrong:

for-each-pair(
  (1, 8, 2),
  (3, 4, 3),
  fn($item1, $item2, $pos) {
    $pos || ': ' || max(($item1, $item2))
  }
)

Result: ("1: 1", "2: 4", "3: 2")

The results as given return the min of the pair, not the max.

With string-join, you can create a string for multiple strings, optionally interspersed with a separator. array:join can be used to create an array from multiple arrays.

fn:intersperse, which has been added to the XQuery 4 draft, does something very similar, and early feedback indicates that the function is useful, but easily to overlook due to its name.

I propose to unify the functions by…

1. renaming fn:intersperse to fn:join;
2. adding a parameter to array:join: $separator as array(*)* := (); and
3. allowing a separator sequence for fn:string-join: $separator as xs:string* := ().

Examples

Query | Result | Info -- | -- | -- string-join(('1','2','3'), '-') | '1-2-3' | existing syntax array:join([[1],[2],[3]], ['-']) | [1,'-',2,'-',3] | new join((1,2,3), '-') | (1,'-',2,'-',3) | now: intersperse((1,2,3),'-') string-join(('1','2','3')) | '123' | existing syntax array:join([[1],[2],[3]]) | [1,2,3] | existing syntax join((1,2,3)) | (1,2,3) | now: intersperse((1,2,3))
(or just (1,2,3)) string-join(('1','2','3'), ('-','+') | '1-+2-+3' | new array:join([[1],[2],[3]], ['-','+']) | [1,'-','+',2,'-','+',3] | new join((1,2,3), ('-','+')) | (1,'-','+',2,'-','+',3) | now: intersperse((1,2,3),('-','+'))

Section 1.5 of F+O introduces the signature proforma notation, and indicates that default values may be included for parameters.

It does not however say how the default value is interpreted. For example, with the signature

fn:starts-with-sequence(
    $input as item()*,  
    $subsequence as item()*,   
    $compare as function(item(), item()) as xs:boolean := fn:deep-equal#2
) as xs:boolean

There is nothing to tell us that the expression fn:deep-equal#2 is evaluated with the static and dynamic context of the caller (or of the function reference).

Note that this is different from the similar notation used for function declarations in XQuery, where the static context for the default fn:deep-equal#2 would be taken from the function declaration in the Query prolog.

We have addressed the question of non-transitivity of equality matching in distinct-values(), and in XSLT and XQuery grouping, but the same issue exists for sorting. Currently fn:sort, as well as XSLT and XQuery sorting, rely on the "lt" operator for comparing values including mixed numerics such as doubles and decimals. Because this promotes to double, it is capable of losing precision, and is therefore non-transitive. Most sort algorithms rely on the supplied comparison function being transitive, and if it isn't, then undefined failures may occur including non-termination.

One particular quirk (which led me here) is that fn:highest and fn:lowest start by using fn:sort semantics to put the values in order, and then rely on fn:deep-equal semantics to find the values that are "equal highest" or "equal lowest". But fn:sort and fn:deep-equal have different ways of deciding whether two values are equal: decimal 1.2 and double 1.2 are equal for fn:sort, but not for fn:deep-equal.

We need to explain more clearly that we now have different rules for comparing numeric values in different circumstances. My understanding of the situation is:

For eq, =, etc, we continue to use the XPath 2.0/3.0/3.1 rules for backwards compatibility reasons: for example comparison between decimal and double is done by converting the decimal to a double. This has known problems in terms of transitivity, but we have retained the rules because we identified that too many compatibility problems would be introduced by changing them.

For deep-equal, distinct-values, XSLT and XQuery grouping, etc, we have switched to the rules that were introduced for comparing map keys in 3.0, now available through the fn:atomic-equal function. Under these rules, doubles are promoted to decimals for comparison.

We should probably include a table showing which rules are used where.

A good example to use is (1e-3 = 0.001). This is true in both 3.1 and 4.0. But under the rules for maps in 3.1, and the new rules for distinct-values in 4.0, these two values are considered distinct.

In fn:fold-right (and thus array:fold-right) it's not clear how the position parameter works.

It appears to start at 1, and then to be decremented, which seems a little weird.

Working out what happens seems to involve reverse engineering the code, which isn't ideal. It's useful to have a formal definition of the function using code, but it shouldn't be necessary to reverse engineer 20 lines of difficult recursive code in order to get a feel for what the function does.

The only example given doesn't add any clarification.

Fix #742

In XPath 4.0 there is a new concept of Implausible Expressions.

There are several different sections about different types of implausible expressions:

• 2.4.6 Implausible Expressions - only a single example, and it is not in an Examples sub-section and is difficult to locate.
• 3.8.1 Implausible Coercions -has 3 examples and seems OK.
• 4.7.4.3 Implausible Axis Steps - has no visible examples.
• 4.15.3.4 Implausible Lookup Expressions - has no examples.

Another problem is that the definition of "implausible" seems not precise and subjective (what is the meaning of "there is a high probability that they were written incorrectly"):

"implausible Certain expressions, while not erroneous, are classified as being implausible, because there is a high probability that they were written incorrectly."

Proposed fixing actions:

1. Provide a more precise and non-subjective definition of the concept.

2. Provide many examples of implausible expressions - both in the central section 2.4.6 and in all other sections dealing with more specific types of implausible expressions.

I don't think that the semantics of the expression $E??KS are clearly enough defined.

The effect of the deep lookup expression E??KS is obtained by evaluating E, establishing its recursive content C, removing any item that is not a map or array to yield a sequence D, and then evaluating the shallow lookup expression D?KS, but with one exception: if evaluation of any shallow lookup fails, then the error is not propagated, but instead its result is taken to be an empty sequence.

1. The definition of "recursive content" needs to be tightened up.
2. It needs to be more clearly stated which errors we ignore, and which we don't. For example, what if the key specifier evaluates to a non-singleton sequence?
3. In the case of $E??* in particular, I don't think it makes much sense to exclude items that are not maps or arrays. I think the expectation in this case is to return the full recursive content.
4. As currently defined, if $M is a map, the the result of $M??* includes the map $M itself. I don't think this matches expectations. Certainly, with the parallel expression $node//*, the result does not include $node.

We have added the text

If the context value is anything other than a single item, the semantics of the expression ?KS are defined to be equivalent to the expression . ! ?KS. The remainder of this section therefore explains the semantics on the assumption that the [context value] is a single item, referred to as the context item.

Consider the case where the context value is a sequence of two maps (map{'x':1, 1:'p', 2:'q'}, map{'x':2, 1:'P', 2:'Q'}) and the expression is ?(?x). What is the context for evaluation of the key specifier (?x)? I would have expected that we evaluate the key specifier once, in the outer context, so the key specifier value is (1,2) and we therefore take the entries with keys 1 and 2 in both maps, giving a result of ('p', 'q', 'P', 'Q'). But the cited paragraph suggests we evaluate KS separately for each item in the context value, and this return entry 1 of map 1 and entry 2 of map 2, giving a result of ('p', 'Q').

I know it's an edge case and it's unlikely in practice that people will write context-dependent key specifiers, but the rules need to be clear. I thought we had previously decided that the key specifier expression should be evaluated in the outer context.

The new syntax for type-qualified wildcards has problems when used in a chained lookup expression, for example

[[1,2], [3,4], 5, 6]?*::array(*)?1

because the "?" that follows array(*) is interpreted as an occurrence indicator. This can be avoided by using parentheses, but it's too much of an elephant trap - a better solution is needed.

Apart from id, Haskell has const, which accepts 2 arguments, but only returns the first. Thanks to the introduction of default arguments, it’s straightforward to extend fn:identity to be able to accept 2 arguments:

fn:identity(
  $input    as item()*,
  $ignored  as item()*  := ()
) as item()*

Fix #856

The errors section of fn:deep-equal still says

A type error is raised [[err:FOTY0015] if either input sequence contains a function item that is not a map or array.

This is no longer the case (and error FOTY0015 is now obsolete and should be removed from the appendix)

Closes #844

During discussion of the ?? operator it was pointed out that we need more examples and explanation, especially of how to handle cases where the "flattening" behaviour of the operator is inconvenient. This applies equally to paths using the existing ? operator - $x?y?z and $x??z both have this problem. For example, there is no way of filtering the result of$x?y?z or $x??z to select only members of size 3.

Fix #852

tc-inclusive($node/$step(.)), $step)

should read

tc-inclusive($node/$step(.), $step)

Editorial; closes #822 (visit this issue for a list of the changes)

Now that fn:parse-html has been added to the specification, we need test cases for all provided options and input types (including binary input).

Looking at the current set of test cases, it seems unrealistic to use older libraries such as TagSoup for this function. I wonder if we should support ·implementation-defined· parsing algorithms at all. What do others think?

Next, is there any implementation available that supports all given method/html-version variants?

Fix #847

The description of build-uri was written with the expectation that if a key was in the map, it's value should be used. I don't really want to replace every occurrence of

if `x` is present in the map

with

if `x` is present in the map and its value is not the empty sequence

So I've attempted to justify that globally with the following paragraph at the beginning of the description:

The components are derived from the contents of the $parts map. To simplify the description below, any key whose value is the empty sequence is ignored; this is equivalent to the key not being present in the map.

I'm not hugely proud of that bit of prose though. Suggestions for improvements most welcome.

• Adds tagging for fos:test elements so we know which tests depend on XQuery (rather than XPath).

• Corrects expected results for some tests

The (only) example in the spec of a build-uri() call specifies {"port":()} . But uri-structure-record has

port? as xs:string,

which means that the empty sequence is not a valid value. Either the record structure should be changed to specify the type as xs:string?, or the example should be changed.

Fix #845

The section of the XPath specification on Quantified Expressions contains paragraph starting

"The order in which test expressions are evaluated for the various binding tuples"

This is the only place in which "binding tuples" are mentioned in connection with quantified expressions, and in XPath (as distinct from XQuery) it is the only place where tuples are mentioned at all. The paragraph could easily be rewritten to avoid introducing a new concept.

Observations:

A. What about renaming fn:contains-sequence, fn:starts-with-sequence and fn:ends-with-sequence to fn:contains-items, fn:starts-with-items and fn:ends-with-items, in alignment with fn:items-at, fn:items-before, etc.? If we add equivalent functions for arrays, it could be array:contains-members, etc.

B. It seems confusing to have fn:items-starting-where and fn:items-after, instead of fn:items-starting-after. Maybe we can think of alternative (shorter) names for fn:items-starting-where and fn:items-ending-where? – I know we’ve discussed before; I raised it again due to user feedback.

In many threads (#135, others), we have discussed how to align the functions for sequences, arrays, and maps. This is an attempt to summarize the status quo, and I hope to keep it up-to-date in the coming weeks.

The 4.0 functions are the ones with the keyword new attached. If the function is followed by a question mark, there may be an existing issue for its addition, or it may be consistent to add it.

Please note that the data types have fundamental differences, so it’s not always possible to present or provide exact symmetries.

To be discussed

Functions | Array Functions | Map Functions --- | --- | --- fn:contains-subsequence
new: #94, #844 | array:contains-subarray ? | map:contains fn:ends-with-subsequence
new: #96, #844 | array:ends-with-subarray ? | – fn:starts-with-subsequence
new: #96, #844 | array:starts-with-subarray ? | – fn:distinct-values | array:distinct-members ? | – fn:duplicate-values
new: #123 | array:duplicate-members ? | – fn:empty | array:empty new: #229 | map:empty ? #827 fn:exists | array:exists new: #229 | map:exists ? #827 fn:every new | array:every ? | map:every ? fn:some new | array:some ? | map:some ? fn:highest new | array:highest ? | – fn:lowest new | array:lowest ? | – fn:index-of | array:index-of ? #260 | – fn:index-where new | array:index-where
new: #114 | map:keys($m, $pred)
new: #467 fn:items-at new: #213
→ fn:get ? #872 | array:members-at ? #825
array:get | map:get fn:intersperse new: #2
→ fn:join ? #868 | array:join | – fn:subsequence-where
new: #878 | array:subarray-where ? | – fn:substitute ? #553, #583 | array:replace new;
array:substitute ? #583 | map:replace new;
map:substitute ? #583 fn:slice new | array:slice new | – – | array:split new | map:entries new – | array:values new | map:keys; map:values new – | array:entries ? #826 | map:entries new – | array:merge ? #826 | map:merge – | – | map:entry – | array:members
new → keep? #826 | map:pairs
new → keep? #826 – | array:of-members
new → keep? #826 | map:of-pairs
new → keep? #826 – | – | map:pair
new: #508 → keep? #826

Settled

Functions | Array Functions | Map Functions --- | --- | --- fn:count | array:size | map:size fn:filter | array:filter | map:filter fn:fold-left | array:fold-left | – fn:fold-right | array:fold-right | – fn:for-each-pair | array:for-each-pair | – fn:for-each | array:for-each | map:for-each fn:head | array:head | – fn:insert-before | array:insert-before | – fn:remove | array:remove | map:remove fn:reverse | array:reverse | – fn:sort | array:sort | – fn:subsequence | array:subarray | – fn:tail | array:tail | – – | array:put; array:append | map:put fn:footnew: #250 | array:foot new: #250 | – fn:trunk new: #250 | array:trunk new: #250 | –

Improves the stylesheet that generates the test set misc/BuiltInKeywords.xml; specifically, it's smarter about generating acceptable callback functions that won't trigger an unwanted error.

(The generated test calls each function twice, once with positional arguments and once with keyword arguments, and checks that the two results are deep-equal).

Addresses #840

The newly added example

seconds-from-duration(
   xs:duration("P1Y1D")
)

Result: 1

Looks clearly wrong.

Joel, what did you have in mind?

Fix #838. Editorial.

The examples for a number of functions, such as fn:contains(), use a UCA collation URI bound to the variable $coll.

Two problems:

(a) These sections also contain prose saying "The collation used in these examples, http://example.com/CollationA is a collation in which both -and * are ignorable collation units." - but this is not the collation URI actually used. This error was already present in the published 3.1 Recommendation.

(b) The examples that use the variable $coll do not have a use attribute, which means that the test cases generated in the test suite do not declare the variable, which means that the tests fail.

Adds support for a deep lookup operator "??" as a transitive equivalent of "?", and allows a wildcard lookup X?* or X??* to be qualified with the required type of value, for example X??*::record(from, to).

The OKFN's Frictionless Data project's CSV Standard specifies some additional things we should take into account for fn:parse-csv and related functions.

Most important is the option to specify a comment line character, whose presence at the start of a line will cause it to be treated as a comment. Because of the way that rows can span lines, post-processing to extract comments might be impossible in some cases.

The use of record types as a thing that users will interact with seems to be increasing. There are a number of new types proposed and we should review their names to ensure that we are using a coherent naming scheme for all spec-defined record types, and that the names are good enough - that they explain what they are, and aren't too unwieldy.

The csv-row-record type used by the CSV XDM mapping provides its fields and an accessor function that can perform field lookup by index or column name. The column names are set when the CSV is parsed. For users who want to make use of csv-row-record (when needing to parse CSVs that fn:parse-csv itself cannot handle out-of-the-box, perhaps) we should provide a creation function that accepts the name: index map and the fields for the row and creates a csv-row-record with a correctly functioning field function entry.

Hello,

Per discussion on the list, this PR changes the way git handles line endings so that text files will always, exclusively have line endings delimited by a single lf. This should be a largely transparent change for Mac/Unix users. On Windows, it means that checked out files will have lf line termination. If your favorite editing tool can handle this, then you don't have to care. Even if your editor saves files with cr/lf line endings, git will turn them back into single lf endings when you commit your changes.

Hat tip to @ChristianGruen for keeping focus on this issue.

This PR has no technical changes, and effects a relatively small number of files. I'll leave it here for a day or two, then I'm going to be inclined to merge it. Object now, if you object :-)

Note that unlike many of the functions we have added, these are non-trivial: they cannot easily be implemented in XSLT or XQuery.

This is a first cut and I expect some refinement will be needed, but reviews are invited.

I might subsequently propose layering some XSLT syntax on top of this for convenience.

I don't really know how these crept in. Perhaps I was negligent in confirming that #828 passed tests in PR? Too late to tell now, but I think I've fixed them.

Many of the functions in this non-normative appendix are no longer needed, or can be expressed more concisely using new 4.0 language features.

In #817, it was discussed that the current EBV semantics have been inspired a lot by XPath 1.0. Today, we have numerous other data types apart from strings, doubles, booleans, and nodes, and I believe it’s time to do justice to this by getting rid of the error for unsupported data types for fn:boolean.

We currently have:

Type | Rule to compute boolean value --- | --- node() | true() xs:boolean | $item != 0 and not(is-NaN($item)) xs:untypedAtomic, xs:string, xs:anyURI | $item != ''

I have two options in mind:

1. The easiest solution, which would come closest to JavaScript, would be to return true() for all other items. This would allow us to do simple checks like:

declare function local:byte-length($data as xs:basexBinary?) xs:integer {
  (: instead of exists($data); utilizes the EXPath Binary Module :)
  if($data) then bin:length($data) else 0
};

2. If we want to be more fine granular, we could do justice to the specifics of 4 more types:

Type | Rule to compute boolean value --- | --- array(*) | array:size($item) != 0 map(*) | map:size($item) != 0 xs:base64Binary
xs:hex64Binary | bin:length($item) != 0 or
not($item = (xs:hexBinary(''), xs:base64Binary(''))

It would then be possible to write:

• if($map) instead of map:size($map) != 0 or map:exists($map) (see #827 for the naming controversy).
  Note that if($map) will also return false() is $map is an empty sequence.
• if($func) { $func(1, 2) } instead of exists($func)

In the last comments of #817, it was addressed that the behavior of existing code may change if errors are replaced by results. I hope we can live with that, as I cannot think of cases in which the EBV computation make sense for items that always raise an error.

Which option do some of you prefer?

I have added positional parameters to the following functions:

array:filter
array:fold-left
array:fold-right
array:for-each
array:for-each-pair
array:index-where
fn:every
fn:filter
fn:fold-left
fn:fold-right
fn:for-each
fn:for-each-pair
fn:index-where
fn:items-after
fn:items-before
fn:items-ending-where
fn:items-starting-where
fn:iterate-while
fn:partition
fn:some

Comments:

• For fn:every and fn:some, the additional parameter seemed useful to me, as a positional variable has been requested for quantifier expressions in the past.
• I’ve also added positional variables to folds.
• I’ve unified and simplified the formal XPath/XQuery equivalencies in the rule sets.
• I’ve dropped some XSLT equivalencies because I felt that the XQuery representations are more concise (I certainly won't mind if they're added back).

Closes #516.

We have array:empty and array:exists, but no equivalent functions for maps.

I think we have decided to live with the ambiguity (discussed in #229) that map:exists(map {}) will return false although the “map exists”. Same for arrays.

When introducing the new array features to some users, the for member syntax was welcomed by everyone.

However, there was some confusion (again, see my past feedback to the mailing list) about what the QT4 group considers to be “members of an array”, and about value records.

In particular, the “value record” representation of arrays led to questions that I didn’t have a good answer for. In particular, people didn’t understand why an array member was returned as a map, and why that map is (again) called “array member” or “value record” – a term no one associated with arrays (at least for now… which somewhat is not surprising, as it has just been introduced).

Next, due to atomization (as mentioned before), array:split allows us to omit the explicit ?value lookups that are required for array:members:

sum(array:members($array)?value)
sum(array:split($array))

I suppose I have been biased in my presentation, but I’ve failed to give good arguments to justify the current solution in the spec. The questions that I think need to be answered are:

• How will people benefit from the (usually intermediate) map representation for array members?
• What exactly do we win with array:members and array:of-members instead of using the existing array:join function, combined with the new array:split function?

Out of interest, I have rewritten the formal equivalencies for the array functions with array:split/array:join:

• array:append

array:of-members((array:members($array), map{'value':$member})) array:join((array:split($array), array { $member }))

• array:build

array:of-members($input ! map { 'value': $action(.) }) array:join($input ! array { $action(.) })

• array:filter

array:of-members(array:members($array) => filter(function($m) { $predicate($m?value) }) array:join(array:split($array) => filter(function($m) { $predicate($m?*) })

• array:for-each

array:of-members(array:members($array) ! map { 'value': $action(?value) }) array:join(array:split($array) ! array { $action(?*) })

• array:for-each-pair

array:of-members(
  for-each-pair(array:members($array1), 
    array:members($array2), 
    function($m, $n) {map{'value': $action($m?value, $n?value)}}))
array:join(
  for-each-pair(array:split($array1), array:split($array2),
    function($m, $n) { array { $action($m?*, $n?*) } }))

• array:insert-before

array:of-members(array:members($array) => insert-before($position, map{'value':$member})) array:join(array:split($array) => insert-before($position, array { $member }))

• array:remove

array:of-members(array:members($array) => remove($positions)) array:join(array:split($array) => remove($positions))

• array:reverse

array:of-members(array:members($array) => reverse()) array:join(array:split($array) => reverse())

• array:slice

array:of-members(array:members($array) => slice($start, $end, $step)) array:join(array:split($array) => slice($start, $end, $step))

• array:split

array:of-members(array:members($array) => sort($collation, function($x) { $key($x?value) })) array:join(array:split($array) => sort($collation, function($x) { $key($x?*) }))

• array:subarray

array:of-members(array:members($array) => subsequence($start, $length)) array:join(array:split($array) => subsequence($start, $length))

• array { $sequence }

array:of-members($sequence ! map { 'value': . }) array:join($sequence ! array { . })

• [E1, E2, E3, ..., En]

array:join((map { 'value': E1 }, map { 'value': E2 }, map { 'value': E3 }, ... map { 'value': En })) array:join((array { E1 }, array { E2 }, array { E3 }, ... array { En }))

• $array?*

array:members($array) ! ?value array:split($array) ! ?*

• $array?$N / $array($N)

array:members($array)[$N]?value array:split($array)[$N]?* (or array:get($array, $N))

As a side note, I noticed that the equivalence given for array:join must be buggy:

(: current equivalence presented in the spec :)
array:of-members($arrays ! array:members(.))

(: returns [ 1, 2, 3 ] :)
let $arrays := ([ 1 ], [ 2, 3 ])
return array:of-members($arrays ! array:members(.))

Concluding, If I could choose, I would tend to drop array:members and array:of-members and rename array:split to array:members.

The title says it all.

We have fn:slice and array:slice. We also do have fn:items-at, but we have somehow missed to add the corresponding array:items-at array:members-at function.

We could even think of a function map:entries-at and map:values-for-keys. The first of these would return all map entries that have as keys one of the provided as argument set of keys. The 2nd function would return all values of the map entries that have as keys one of the provided as argument set of keys.

Here is a complete XPath 3.1 implementation:

let $members-at := function(
                 $input as array( *), 
                 $indexes as xs:integer*
                ) as array(*)*
 {
     for $ind in $indexes
       return [$input($ind)]
 }

Example:

Evaluating this expression:

let $members-at := function(
                 $input as array( *), 
                 $indexes as xs:integer*
                ) as array(*)*
 {
     for $ind in $indexes
       return [$input($ind)]
 }
  return
     $members-at([1, (2, 3), (4, 5, 6)], (1, 3) )

produces the wanted result:

[1], [(4,5,6)]

Fix #799 Fix #738

Fix #712

XQuery spec:

• [x] 4.3.4 Context Value Reference → 4.3.4 Context Value References (plural, in alignment with the other expressions)
• [x] Move 4.3.4 before 4.3.3 Parenthesized Expressions (and after 4.3.2 Variable References)
• [x] 4.19 Switch Expression → 4.19 Switch Expressions
• [x] Try/Catch Expressions: There’s no CatchErrorList anymore

XQFO spec:

• [x] Unify representation of equivalent examples, implementations, …see https://github.com/qt4cg/qtspecs/pull/828#issuecomment-1807222990
• [x] Add History sections for new functions
• [x] 9^XXX outputs "4451" → "3185"
• [x] errors are signaled → raised (#783)

In XQuery, the default namespace for annotations is http://www.w3.org/2012/. It’s the only namespace for which no prefix exists, and I think we should change that. ann feels like a reasonable choice to me as we tend to have short prefixes (such as err for errors).

In the current XQuery 4, the coercion rules are applied to variable bindings of FLWOR expressions:

https://qt4cg.org/specifications/xquery-40/xquery-40-diff.html#id-binding-rules

I believe this is yet another feature that needs to be formally accepted. If it has already been done so, this issue can be closed again immediately (maybe with a reference to the related issue, or the associated QT4 meeting).

This is a placeholder issue for Syd Bauman’s suggestion on Slack to integrate Foxpath, or parts of it, in the standard.

Yes, I dare to question the semantics of effective boolean values. The reason is that I never learned to fully like them. It seems obvious where the rules come from, and why they have been reasonable in previous versions of the language. From today’s perspective, I think there’s really some need to simplify and unify the rules, and I believe it’s possible with little effort and without endangering backward compatibility (provided that we are willing to drop errors and return results).

Some examples for the somewhat strange nature of the current rules:

• boolean((<_>x</_>, <_>y</_>)) returns true, whereas boolean(('x', 'y')) raises an error.
• boolean(xs:NCName('x')) returns true, whereas boolean(xs:QName('x')) raises an error.
• boolean((<a/>, 1)) and boolean((1, <a/>)) may either return true or raise an error, depending on the implementation.

I believe it will make much more sense to

1. check all values of the input equally (in analogy to the existential semantics of general comparisons), and
2. use existence checks for more types instead of raising a clueless error.

The semantics would be tidied up a lot, it could look like this…

declare function ebv($input as item()*) as xs:boolean {
  some $item in $input satisfies typeswitch($item) {
    case xs:untypedAtomic | xs:string | xs:anyURI  return $item != ''
    case xs:numeric                                return $item != 0
    case xs:boolean                                return $item
    default                                        return true()
  }
};

…or, if we include more types, like this:

declare function ebv($input as item()*) as xs:boolean {
  some $item in $input satisfies typeswitch($item) {
    case xs:untypedAtomic | xs:string | xs:anyURI  return $item != ''
    case xs:numeric                                return $item != 0
    case xs:boolean                                return $item
    case xs:base64Binary                           return $item != xs:base64Binary('')
    case xs:hexBinary                              return $item != xs:hexBinary('')
    case array(*)                                  return array:size($item) != 0
    case map(*)                                    return map:size($item) != 0
    default                                        return true()
  }
};

(If we believe that it’s too progressive to accept all types, we could still raise an error for some specific types… although I don’t think that anyone would benefit from this choice).

As a result, EBV checks could also be used to check more than one item:

(: true if at least one tokenized string is non-empty :)
if(tokenize('a/', '/')) then ...
(: true if at least one number is unequal to 0 :)
if($numbers) then ...
(: true if at least one Boolean is true :)
if(false(), true(), true()) then ...

Nothing would change for the classical EBV checks: if($node/*), if($x = $y), if($ok), …

Regarding “1. check all values of the input equally”, one could argue that this might affect performance. I don’t actually think so: For node sequences, it will still be sufficient to retrieve only the first item. For mixed-type sequences, errors were raised in the past.

The resulting EBV could be easily combined with revised predicate semantics (#816).

Predicates provide a compact syntax for positional access to sequences but only single numbers are supported.

It would be handy to allow E[1, 2, 3] (E[3, 2, 1], E[1 to 3], etc.) as a shortcut for E[position() = (1, 2, 3)] (MarkLogic offers this possibility, if I remember correctly). We shouldn’t change the EBV syntax, and we should continue raising an error if the predicate sequence contains items other than numbers. Examples:

Expression | Result --- | --- (1 to 5)[2, 3] | 2, 3 (1 to 5)[3, 2] | 2, 3 (1 to 5)[2 to 3] | 2, 3 (1 to 5)[6, 'x'] | error (1 to 5)[1 to 5, 'x'] | 1, 2, 3, 4, 5 or error (up to the implementation, similar to sequences that start with a node)

I bet this has already been discussed in the past…

Introduces mutual promotion between xs:base64Binary and xs:hexBinary. Fix #130.

Reorganises the material on type promotion. Supersedes PR #538.

There are some details missing for the error handling of the built-in template processing for shallow-copy-all.

When the built-in template processes an array, the built-in rule constructs a value record for each member, and applies-templates to this value record, expecting the result to be a value record.

• It's not stated whether the "value record" is extensible, that is, whether it can contain fields other than "value"
• No error code is given, and it's not identified explicitly as a type error.

Similarly when a map is processed, the result is expected to be a key-value record, but the details of the error are not spelled out.

Fix #809

It has always been a challenge to teach the difference between a conversion, coercion, promotion, casts and treats. Of course, we cannot get rid of the complexity, but I think it’s a good step forward that the conversion rules have recently been unified in the specification.

Maybe we can push it even further. I would suggest…

1. renaming “Coercion Rules” to “Type Coercion” (…most other sections contain rules as well),
2. making “Function Coercion” a subsection of “Type Coercion”, and
3. making “Type Promotion” another subsection of “Type Coercion”.

I’m the wrong person to decide this, but maybe the full section can be moved to the Appendix, as it’s referenced all around the documents.

Obviously, I should have done icons for functions that have changed as well. So now I have. I'm not convinced that the 🆙 emoji is sufficiently different from the 🆕 emoji. Maybe I should try to make them different colors or something. But it's a start.

This PR adds a .gitattributes file that identifies some files explicitly as text files (and others explicitly as binary files).

After this PR is merged, I believe it will be the case that end-of-line handling will be correct on a per-platform basis. That is, if a Windows user checks out the repository, all the files will have PC-style line endings (CR followed by LF). If a Mac or Unix user checks out the repository, all the files will have Unix-style line endings (LF).

Commits should "do the right thing" to preserve the line endings appropriately.

Fingers crossed!

On the XML.com slack, Pieter Lamers observes that fn:atomic-equal looks a little out of place in chapter 18 with the other map functions.

Following a discussion on the XML.com slack, this is a little lunchtime hackery...

Any function listed in the new-functions section of the changes appendix or identified with an ednote that contains the string New in 4.0 will be marked as 🆕 in the specification. The 🆕 occurs in the ToC, the drop-down function list, and to the left of the section title.

(I'm just going to merge this as it has no spec changes and the effect won't be visible in the PR anyway.)

"Instance of the data model" generally becomes "input tree"

An instance of the data model used to hold serialization parameters is now referred to as a "parameter document".

Fix #789

As CG observes in issue 566, there are still a couple of small problems with fn:parse-uri().

1. The regular expressions used to parse the fragment identifier and query are incorrect. The URI specification allows ? to appear in a query string and # to appear in a fragment identifier, so the expressions have been rewritten to match everything after the first ? and #, respectively, even if they contain additional ? or # characters.
2. The description of how to interpret the regular expression for parsing a Windows file path preceded by slashes was incorrect. It caused the leading "/" to be lost. That's been corrected.

Stylesheet changes to improve the formatting of F&O examples. (But they still aren't perfect...!)

1. Variables used by the examples are pulled out under a separate heading.

2. If any examples include long lines, then single-column ("wide") format is used automatically.

3. In two-column layout, if the left-hand column uses "eg" format, then so does the right-hand column. (But it would be nice to get rid of the grey background for this case).

In addition to a minor typo, note the following:

• rearrangement of a rule for $input (the original syntax, mixing types and derived types, confused me)
• examples in X-from-Y duration functions to help illustrate the problem when mixing the two components of a duration.

Closes #651

Closes #801

In the specs, both “nondeterministic” and “non-deterministic” can be found. The first one is more frequent, so I guess it’s the one that’s preferred.

I’ve chosen XPST0017 over XPST0003 (it felt more intuitive to me).

Closes #660.

In fn:expanded-QName, http:/example.com should be http://example.com.

In map:merge, there's a stray "(" in

map:merge((
  ($week, map { 7: "Unbekannt" })
)

I decided to follow my initial suggestion and add unordered to fn:deep-equal instead of adding a separate function for it. My motivation:

• It’s convenient to be able to use the new option in combination with the other options.
• It will be used more often than many other options we’ve added recently.
• It seemed pretty straightforward to add, both in terms of documentation and implementation.

Closes #479

Attn @ndw -- these edits pertain to 6.6.1 fn:parse-uri and require your careful review.

• hierarchical option appears in the rules of the function but not its preamble
• "This function is described..." moved to anticipate the actual narrative.
• I was a bit thrown by the phrase "This approach...is not implementation advice" because "approach" is vague and the reader is left with the impression that a good deal of something that follows is non-normative. We offer non-normative prose in the rules, but that informal prose is normally followed by an equivalent description that is normative. I did not make any edits in this area, but I would recommend clarification as to exactly what parts of the narrative are normative and which are non-normative.
• The period-to-colon change at 27573 is to make sure the reader remains aware of the governing "If..." clause.
• I deleted a paragraph that attempted to distill the rather complicated description of fn:decode-from-uri. A mere xref provides a cleaner narrative here and a more accurate description of uri decoding, and the xref is within the same document, so should not pose a burden on the reader.
• For the query-parameters there was a discrepancy in the data model presented (in the preamble versus in the function rules), and I opted for the array-of-maps model. If it's supposed to be the other way (simple map), let me know and I'll revise.
• query-segments versus query-parameters; I went with the latter.

Let me know where things ain't right.

It would be useful to be able to write

let $n as xs:integer := some_expr ....

and also maybe

for $s as xs:string, $p as my:name return....

This might occur e.g. in the body of a function, for example... and would help type safety and debugging.

Closes #655

Closes #216. See this issue for more details on the proposed change.

Closes #704

See issue #783.

Errors are raised, not signalled or reported or generated or thrown.

Pursuant to action QT4CG-051-02, here is a proposal for fn:invisible-xml.

There are a number of different design choices that could be made. For example, one could argue that a function of the form fn:invisible-xml($grammar as xs:anyURI, $input as xs:anyURI) would be the easiest thing for users in many cases. But not in all cases. You might want versions where either the grammar or the input were strings instead of URIs.

I've attempted to craft the smallest proposal that could get the job done.

Proposes XSLT 4.0 and Serialization 4.0 changes resulting from the generalization of context item to context value in XPath 4.0.

The serialization changes are purely editorial.

In XSLT, we acknowledge the introduction of the context value in XPath but don't take advantage of it; at the XSLT level, the context value for an instruction is still always a single item. The only technical change is that we allow xsl:evaluate to pass any value as the context value to the dynamically evaluated expression (while retaining the name of the relevant attribute "context-item").

Fulfils action QT4CG-046-01

The serialization spec makes extensive use of the phrase "an instance of the data model". This phrase is defined to be a synonym of "value" or "sequence", and it seems to add a lot of words without adding any clarity. In many cases the context makes clear that it is actually referring to a tree rooted at a document node.

In section 3.1 "Setting Serialization Parameters by Means of a Data Model Instance" it would be helpful to use a more specific phrase, for example "a parameter document".

Elsewhere the phrase is often used to mean "the value being serialized", and again, it would be helpful to use a more specific phrase, perhaps a term that is quite distinctive such as "the payload". (It's sometimes referred to as "the result tree", but that assumes that the value being serialized is the result of a query or stylesheet.)

I propose a function fn:annotate() which will add annotations to a function item. It will create a new function item that differs from the original only in its annotations.

Currently annotations are a dynamic property of function items, but they can only be set in very limited ways. Allowing them to be set dynamically creates lot of opportunities. For example, there is currently no way to set annotations on a map or array, but one could define annotations, for example, to indicate that a map should hold entries ordered by key, or that an array should use a "sparse" implementation; there is scope both for spec-defined and vendor-defined (and perhaps even user-defined) annotations.

Some of the possibilities this would open up have been outlined in other issues. At present, though, we have a capability in the data model that is underexploited, and an fn:annotate() function is a conceptually simple addition, which can be justified purely on the grounds of completeness - if something exists in the data model, surely it should have getters and setters?

1. Errors are raised, not signaled
2. Cross-references point to 4.0 specs rather than 3.0/3.1
3. Some internal markup changes for tidiness

Partial fix for #783 as it affects the serialization spec.

This closes #695 .

Closes issue #777

xsd:assert is illegal in schema version 1.0; updated to specify version 1.1.

The most common phrase we use when describing an error condition is "A (dynamic|static|type) error is raised if ...".

There are other cases where we use the verb "reported" or "signaled" (and occasionally, "thrown" or "generated"). We should avoid these verbs, partly in the interests of consistency, and partly because they are misleading: an error is not reported if it is caught by a try/catch, but it is still raised.

XSLT also commonly uses the phrase "It is a (dynamic|static|type) error if ... " which is also acceptable.

I’ve eventually renamed $pairs to $input (I didn’t rename $input to $members as initially suggested, as we have $member parameters in other functions that are of type item()*, not record(value as item()*)).

Closes #469

Fix #776

You'll need to pull and rebase master after I merge this.

We have changed format-number() and some XSLT functions including system-property(), function-available() etc so that the QName-valued argument is now declared with type union(xs:string, xs:QName) rather than xs:string. I believe that there are edge cases where this is an incompatible change -- for example if the supplied value is an xs:anyURI value. I think the edge cases are probably sufficiently obscure and unlikely to occur in practice that it is sufficient to document them.

I propose a new function for the core XPath functions, here called fn:hash() for the sake of discussion. The goal is to give XPath users access to CRC, checksum, and cryptographic hash functions.

Rationale

Simple checksums functions, such as those from the Fletcher family, are relatively easy to write in a host language such as XSLT. More complex ones are far more challenging to write, and may incur serious performance penalties. For example, from the TAN function library, see the MD5 checksum/hash functions. (Yes, one day I thought it would be fun to try to implement the MD5 algorithm in XSLT.) Most programming languages in which an implementation is written have access to cryptographic libraries that are highly performative.

Hash functions are widely used, and certainly important in XML-based workflows, whether as filenames, database fields, etc.

The closest comparable existing functions is generate-id(), but this was designed as an identifier for nodes.

In short, I believe there is a significant need that outstrips current functionality.

In the draft below, I have adopted only the MD-5 algorithm as a core requirement, to catalyze discussion. I have assumed the user wants the string form of the output, not the raw bits. I have not tried to flesh out prose that would warn users away from security complacency.

For discussion, here is a list of relevant algorithms. I look forward to community feedback.

fn:hash Draft Specification

Summary

This function takes as input a string or octet sequence and returns a string representation of the results from a specified hash, checksum, or cyclic redundancy check function.

Signature

fn:hash(
   $value as union(xs:string, xs:hexBinary, 
                   xs:base64Binary)?   := fn:string(.),
   $algorithm as xs:string             := "md5"
) as xs:string?

Properties

The zero-argument form of this function is deterministic, context-dependent, and focus-dependent.

The one- and two-argument form of this function is deterministic, context-independent, and focus-independent.

Rules

If the zero-argument version of the function is used, the result is the same as calling the one-argument version, with $value set to fn:string(.).

If the one-argument version of the function is used, the result is the same as calling the two-argument version, with $algorithm set to "md5".

The effective value of $algorithm is the value of the expression fn:lower-case(fn:replace($function, '\W+', '')).

If $value is the empty sequence, or a string of zero length, the function returns the empty sequence.

If $value is an instance of xs:string it is cast to xs:hexBinary on the basis of UTF-8 encoding. If $value is an instance of xs:base64Binary it is cast to xs:hexBinary.

The function returns an xs:string representation of the bytes returned by passing the xs:hexBinary value of $value as an octet sequence through the specified hash or checksum function.

Conforming implementations MUST support md5 and the associated MD5 Message-Digest algorithm defined by RFC 6151 (update to RFC 1321). They MAY support other checksum and hash functions with implementation-defined semantics.

Error Conditions

A dynamic error is raised [err:XXXXXXX] if the effective value of $algorithm is not one of the values supported by the implementation.

Notes

• The MD5 algorithm is normally not used for cryptographic purposes. [More cautionary prose about not assuming that something can be trusted as secure.]

Examples

Expression | Result -- | -- fn:hash("abc") | 900150983cd24fb0d6963f7d28e17f72 fn:hash("ABC") | 902fbdd2b1df0c4f70b4a5d23525e932

Light edits for consistency.

• Examples in char() have been reordered so that the most interesting and useful examples are at top.
• Substring functions were difficult to read in the specs because of the overly long collation URI. These are now bound to a variable, to enhance legibility.
• Schema location fixed.
• extra examples in string-length(), to help users recognize the role of combining characters.

In the current draft for replace() the history log states that the new parameter $action has not yet had community review. This ticket provides a placeholder for attention and discussion.

If the CG has already reviewed and accepted $action, comment here and I will update the function history accordingly.

The serialisation spec has xspecref spec="XT31" references that need to be updated to XT40. But for some reason the /etc/XT40 file isn't being built, so these references fail to resolve.

I deleted my fork, got a new one and applied the latest changes. Seems this fixed the issues.

(This is related to fn:parse-uri, fn:build-uri, and fn:decode-from-uri. I'm making it a distinct issue to call it out and see if we can get consensus on the right answer. I've come to the conclusion that what I've implemented isn't justified by any specific reading of the relevant specifications, so it's wrong.)

This question is slightly tricky because encoding (or not encoding) characters can change the meaning of the URI.

If you trace your way through the ABNF in RFC 3986 you eventually get to:

   path-abempty  = *( "/" segment )
   path-absolute = "/" [ segment-nz *( "/" segment ) ]
   path-noscheme = segment-nz-nc *( "/" segment )
   path-rootless = segment-nz *( "/" segment )
   path-empty    = 0<pchar>

The various segment nonterminals boil down to some number of pchar. (The segment-nz form is used to forbid a zero length string before the first /; the segment-nz-nc form is used for a URI that does not begin with a scheme: it must have a non-zero length string before the first / that additionally must not contain a :.)

  pchar         = unreserved / pct-encoded / sub-delims / ":" / "@"

Where:

  unreserved    = ALPHA / DIGIT / "-" / "." / "_" / "~"
  pct-encoded   = "%" HEXDIG HEXDIG
  sub-delims    = "!" / "$" / "&" / "'" / "(" / ")" / "*" / "+" / "," / ";" / "="

I think it follows that all characters except the following must be encoded:

Upper-and lower-case A to Z, the digits 0 to 9, -, ., _, ~, %, !, $, &, ', (, ), *, +, ,, ;, =, :, and @.

or, conversely, that any characters other than those must be encoded.

Observe that / isn't among the characters that are not encoded. That's because the / in hierarchical URIs divides the segments. It's part of the URI syntax. That's why a literal forward slash that appears somewhere in an actual path segment must be encoded %2F and it's why causually unencoding such a character changes the URI.

Observe also that there's no provision here for encoding a space with +, even though it's fairly common. I've carried that error through to some of the test results for fn:build-uri and fn:parse-uri. I'll fix those tests.

I'm going to try changing my implementation to follow the rule above and see what happens.

If you think this analysis is incorrect, please explain where I went wrong.

All 3 editorial suggestions made in today's meeting are now reflected.

This actions the comment in #767 that the rules for the function are unclear.

The SVG element name handling is correct in the spec per the comments in https://github.com/qt4cg/qt4tests/issues/57, so don't need changing.

Purely editorial: Some words in the specification seem to be British English (organised, generalised, behaviour), whereas the majority of the text is American English. Should this be fixed?

I assume there are tools to get this straight? I am too rarely affected by it to know more about it…

I think this is sufficient to close issue 566. @ChristianGruen could you let me know if you agree?

Closes #768

I'm unclear on how %XA should be decoded. Is that %X (an error) followed by A, or is that %XA (an error)?

Also, a couple of typos:

s/to an sequence/to a sequence/
s/that are no hexadecimal/that are not hexadecimal`

Gunther Rademacher points out at https://github.com/qt4cg/qt4tests/issues/57 that the expected test results for parse-html() expect SVG elements to be output in lower case; and as far as I can see, this is consistent with the spec. But is it useful? It means we're producing XML that isn't valid according to the SVG schema, and that presumably might be rejected by tools that expect to handle valid SVG.

In passing, I note that it's very hard to work out from the parse-html() spec what the function actually does; it's somehow written as if it's obvious, but it never gets around to actually saying it explicitly. It also doesn't say what happens when the method option is omitted, as it is in these tests.

In addition, it seems that we don't have any tests that exercise the various options of the function, e.g. the ability to consue binary as well as character input.

Includes the following changes:

• Clarification of the syntax and semantics of the version number in the XQuery version declaration
• Updates to front and back matter to reflect the current status of the 4.0 specs
• Update cross-spec references to point to the 4.0 specs rather than 3.1 specs
• A DTD change from spec="SER40" to spec="SE40" to reflect the name of the /etc file generated by the gradle script.

Fix #765

The text at §5.1 does not yet acknowledge version="4.0".

We should also be a bit more precise about the syntax, which currently says that the version number is two integers separated by a dot. We should give a regular expression to be absolutely clear that we mean decimal integers, no underscores allowed.

There are a few other version references to tidy up at the same time, e.g. "XQuery is designed to meet the requirements identified by the W3C XML Query Working Group [[XQuery 3.1 Requirements]]."

With XQuery, you can import library modules as follows:

import module namespace utils = 'http://project.org/utils' at 'org/project/utils.xq`;
utils:action()

If an implementation has a mechanism to resolve a namespace URI, you can also do this:

import module namespace utils = 'http://project.org/utils`;
utils:action()

It would be desirable to simplify this further, and be able to do one of…

(: already legal, but no namespace binding takes place :)
import module namespace 'http://project.org/utils';
(: not supported yet :)
import module namespace http://project.org/utils;
(: no URI rewriting needed for relative paths :)
import module namespace org/project/utils;

utils:action()

…and…

1. rewrite namespace URIs to local paths (this is how we proceed: https://docs.basex.org/wiki/Java_Bindings#URI_Rewriting)
2. extract the prefix from the path.

I can imagine that the current ways to simplify imports depend a lot on the used implementations. Suggestions are welcome.

I think what prompted my original ticket was the use of "output". That term is used scores of times in the specs, and it always implies the output of a function within the XPath expression. Although the term is qualified in the diagnostic functions with the adjective "trace" or "log", it's tempting to think of the function's actual output. Coming back a few weeks later, I think some gentle massaging (instead of a convoluted preamble) can help readers not make the requisite adjustment, reflected in the present PR.

I have offered two examples that I hope show the practical side of these functions. I might have missed something though, so please chime in.

This covers edits to XQFO 4 through 5.3, and incorporates the suggestion in #758

Drops the $min and $max parameters, with the effect that this corresponds much more closely to the general computer-science definition of a transitive closure.

The function now computes the set of nodes delivered by the transitive closure of the supplied $step function when applied to a given $start node, rather than returning a function item that must then be applied to the chosen $start node. This is hopefully easier for most users to understand, and does not lose any useful functionality.

The $min parameter of the old function is effectively forced to its default value of 1, and the $max value to its default of infinity.

Fix #754 Fix #554

This PR addresses the main points of #554 in making the function correspond more closely to the mathematical (or at least the computer-science) definition of transitive closure. It doesn't implement other ideas in #554, like returning the depth of search alongside the actual closure. That's because I believe in the principle that wherever possible a function should do one thing in as simple a way as possible.

We should be more ambitious about ensuring consistency when serializing data back to its original representation, and how to achieve it. This is the status quo:

Format| Serialize Function | Parse Function --- | --- | --- XML | fn:serialize($input) | fn:parse-xml JSON | fn:serialize($input, map { 'method': 'json' }) | fn:parse-json JSON | fn:xml-to-json($input) | fn:json-to-xml XHTML | fn:serialize($input, map { 'method': 'xhtml' }) | fn:parse-html HTML | fn:serialize($input, map { 'method': 'html' }) | fn:parse-html CSV | still missing | fn:parse-csv and variants

In BaseX, the XML created by fn:json-to-xml can already be serialized back to JSON as follows (related: #759):

serialize(
  <map xmlns="http://www.w3.org/2005/xpath-functions">
    <number key="A">1</number>
  </map>,
  map { 'method': 'json', 'json': map { 'format': 'basic' }
})

For CSV, we’ve introduced a CSV serialization method that supports all CSV flavors we support (see CSV Module for more details):

serialize(
  <map xmlns="http://www.w3.org/2005/xpath-functions">
    <number key="A">1</number>
  </map>,
  map { 'method': 'csv', 'csv': map { 'format': 'direct' }
})

Related (parse functions): #748

We have more and more serialization parameters that are specific to JSON (related: #530, #756, #576, #641). It won’t get better with each new option we add, so maybe it’s time to introduce a custom serialization parameter for JSON (and possibly other methods):

serialize(
  map { "abc": 123 },
  map { 'method': 'json', 'json': map { 'escape-solidus': true(), 'number-format':'#' }
})

If we want to be able to use output options in the XQuery prolog, we additionally need to define a syntax for serializing the options to a single string:

declare namespace output = 'http://www.w3.org/2010/xslt-xquery-serialization';
declare option output:method 'json';
declare option output:json 'escape-solidus=yes,number-format=#';
map { "abc": 123 }

Both suggestions are inspired by our own implementation; we use it for our custom JSON and CSV options.

In the XQFO specs, 5.3.3, the description of keyword strength, option quaternary, is I think confusing/misleading:

quaternary considers spaces and punctuation that would otherwise be ignored (for example data-base=database).

I propose the following:

"quaternary always considers as significant spaces and punctuation (data-base≠database; if maxVariable is punct or higher and alternate is not non-ignorable, lower strengths will treat data-base=database)."

That's a lot more words, but I think more accurate. And it may help the reader become familiar with the other two keywords.

Editors' input is welcome.

We talked on the call today about the tension between defining multiple simple functions focussed on one task, and a small number of omnibus functions that have many different options.

I think we would all agree that multiple simple functions would be the better choice except for the problem that they all end up going into a single global namespace. So the question becomes, how can we better partition the name-space (using the term deliberately with a hyphen).

We're reluctant to use the namespace mechanism to partition our function library because namespaces are cumbersome and clutter the code with lots of boilerplate; declaring namespaces for binding function libraries also has side-effects for example on the semantics of element constructors.

One approach would be to build on the idea that @dnovatchev presented of using maps containing anonymous functions, so for example csv()?parse() would first call fn:csv() to load a family (or library) of functions, of which one is then selected for execution. This works, but I don't think it's a perfect solution; for example static analysis becomes a lot more difficult, and we don't get the benefits of default parameters and keyword arguments.

Most languages use hierarchic names with "." as a separator. Although XML names allow "." as a regular character, none of our built-in function names currently use it as such. So it would be entirely possible to adopt a convention where names like csv.parse() etc are used to name functions in a function family referred to as "csv". This wouldn't by itself require any language changes.

But if we adopted this convention, we could build on it to provide usability tweaks that make a large function library easier to manage. For example, we could put the math functions into the fn namespace with names like fn:math.sin(x), and then provide a way of binding a namespace prefix to a subtree of the fn namespace, so math:sin becomes a synonym for fn:math.sin(). The immediate benefit is that the namespace prefix doesn't need to be declared unless people want to use it. We could also then consider defining an algorithm for searching the fn namespace for abbreviated names such as sin(x), perhaps with some form of "import functions" declaration that says which subtrees of the fn namespace are to be searched.

We get a lot of complaints about the use of exponential notation when formatting large numbers with the JSON serializer.

I propose adding an option such as number-format="picture" to control this, where the picture is a subset of what is allowed for format-number().

And perhaps we should bring xml-to-json() into line. The current capability of providing a callback function is more powerful, but difficult to align with serialization.

We have no expression yet to bind a value to the context value. Such an expression would be useful, among other things, to extend the focus function to sequences (fn { . }, see #129).

Here are 3 possible constructs for that, ordered by my personal preference:

1. Value Map Expression

ValueExpr      ::=  ValidateExpr | ExtensionExpr | ValueMapExpr
ValueMapExpr   ::=  SimpleMapExpr ("~" SimpleMapExpr)*
SimpleMapExpr  ::=  PathExpr ("!" PathExpr)*

(: Example :)
//flower ~ (count(.) || ' flowers: ' || string-join(name, ', '))

The expression would be similar to the simple map expression (which we could rename to item map expression). The following equivalents would then exist for simple FLWOR expressions:

for $i in (1 to 5) return string($i)  ≍  (1 to 5) ! string(.)
let $i := (1 to 5) return count($i)   ≍  (1 to 5) ~ count(.)

fn { E } could be rewritten to fn($c) { $c ~ E }.

2. Context Value Declaration

ContextExpr  ::=  "context" "{" Expr "}" EnclosedExpr

(: Example :)
context { //flower } {
  count(.) || ' flowers: ' || string-join(name, ', ')
}

The result of the first expression defines the context value, the second expression can reference the context.

fn { E } could be rewritten to fn($c) { context { $c } { E } }.

3. Enhanced FLWOR expression (for the sake of completion)

Similar to variables, the dot could be used to bind and reference the context:

LetBinding  ::=  ("." | ("$" VarName)) TypeDeclaration? ":=" ExprSingle
ForBinding  ::=  ("." | ("$" VarName)) TypeDeclaration? AllowingEmpty? PositionalVar? "in" ExprSingle

(: Example :)
let . := //flower
return count(.) || ' flowers: ' || string-join(name, ', ')

fn { E } could be rewritten to fn($c) { let . := $c return E }.

Assessment

• The first solution looks most appealing to me. I like the analogy with the existing syntax for single items.
• We could choose the second solution if we believe that the expression will be rarely used.
• I‘ve backed away from the third solution; I think it would be too pervasive.

I have problems grasping why fn:transitive-closure returns a function. Wouldn’t it be more consistent with the remaining function set, and easier, to pass on the input as the first argument and directly create the result?

fn:transitive-closure(
  $node  as node(),
  $step  as function(node()) as node()*,	
  $min   as xs:nonNegativeInteger?       := 1,
  $max   as xs:positiveInteger?          := ()
) as node()*

It would also be easier then to use the function within chains:

$nodes =!> transitive-closure(fn { * }) => count()

Fix issue #65. Basically, fix the bug whereby xmlns="xxx" changes the default namespace for element NameTests, while retaining bug-compatibility.

Fix #706

Fix #750.

See ACTION QT4CG-048-01

The question arose, if an xsl:mode declaration specifies an expected type in @as, then all template rules in that mode are expected/required to deliver a value conforming to that type. But what about the default/fallback template rules? Surely they need to deliver a value of that type as well?

For example, suppose the mode specifies as="xs:boolean". Regardless of the value of xsl:on-no-match, none of the built-in template rules is going to deliver a boolean. You're expecting a boolean result from xsl:apply-templates, and if none of the template rules match, you're going to get something other than a boolean.

I think the answer is to say that you get a type error if the built-in template rule for the mode returns a value that's not of the required type.

I don't think this error should ever be reported statically, because the compiler has no way of knowing whether the set of explicit template rules is sufficient to cover all cases that will actually arise in source documents.

Allows for expressions interoperable between XPath and XQuery. Fix #653.

The functions for parsing input have been defined by different people, and the current state is quite inconsistent:

Function | Parameters --- | --- fn:parse-xml | $value as xs:string? fn:doc | $href as xs:string? fn:parse-json | $value as xs:string?, $options as map(*) fn:json-doc | $href as xs:string?, $options as map(*) fn:parse-html | $html as union(xs:string, xs:hexBinary, xs:base64Binary)?, $options as map(*) fn:parse-csv | $csv as xs:string?, $options as map(*)

I believe there’s some need to unify the functions, and we could at least:

• introduce a fn:XYZ-doc($href, $options) function for each input format (with at least one encoding option), and
• restrict the type of the input parameter of fn:parse-XYZ to xs:string? and always name it $value.

And I wonder if we should tag all fn:XYZ-doc functions as ·nondeterministic· (if it’s not too late)?

It's quite common to want to write a constant QName; I found myself doing this a lot, for example, in examples and test cases for the elements-to-maps() function. It's clumsy having to call xs:QName() or fn:QName() or parse-QName() for this purpose. It's particularly clumsy with map constructors where you want to write many QNames.

I propose we introduce the syntax Q"prefix:local".

The quotes can be either single or double; the prefix is optional. If there is no prefix, the result is a no-namespace QName.

The prefix (if present) must be bound to a namespace in the static context.

Character and entity references are not allowed.

Note that Q{uri}local is a NameTest, not a QName literal.

We decided to rename xsl:for-each-group/@break-when as @split-when. We should make the same change to the name of the second argument of fn:partition.

I propose adding support for inline xslt functions.

Whilst XPath supports this, Xpath functions are limited in what they can do, and how "look" e.g. returning newly constructed elements isnt possible without parse-xml-fragment.

I would suggest the syntax would be basically the same as for xsl:function except with the name omitted, e.g.

    <xsl:template name="apply-function" as="xs:integer">
        <xsl:param name="input" as="xs:integer"/>
        <xsl:param name="function" as="function(xs:integer) as xs:integer"/>
        <xsl:sequence select="$function($input)"/>
    </xsl:template>

`    <xsl:template ....>
        <xsl:call-template name="apply-function">
            <xsl:with-param name="input" select="1"/>
            <xsl:with-param name="function">
                <xsl:function as="xs:integer">
                    <xsl:param name="value" as="xs:integer"/>
                    <result>
                        <xsl:sequence select="$value * 2"/>
                    </result>
                </xsl:function>
            </xsl:with-param>
        </xsl:call-template>
    </xsl:template>
`

benefits

• less syntactic "noise" of named functions
• the ability to embed xslt functions inline inside maps (and other data types)
• functional parity with xpath (and more)
• natural generalisation to local function proposal

alternatives

• use reference to explicitly named XSLT function
• use XPath (though problematic when constructing new nodes)

Editorial: Some XQuery equivalents were buggy, and the formatting was unified.

In reviewing and accepting the spec for enumeration types, it was suggested that it might be useful to allow values other than strings.

• There's a difficulty in that not all atomic values can be represented by literals. We have the same problem with function annotations; perhaps we need to bite the bullet and define some kind of "constant atomic expression" construct.
• Aside from that, there don't seem to be any major obstacles.

We change

An EnumerationType has a value space consisting of a set of xs:string values. When matching strings against an enumeration type, strings are always compared using the Unicode codepoint collation.

to

An EnumerationType has a value space consisting of a set of atomic values. When matching values against an enumeration type, values are always compared using the fn:atomic-compare() function (as used for comparing map keys).

The subtyping rules (newly defined in terms of unions of singleton enumeration sets) seem to work in their current form, without change. enum("red", "green") is still a subtype of xs:string, because all the enumerated values are instances of xs:string.

The draft XSLT 4.0 specification (§5.3.2) proposes a new declaration xsl:function-library as a solution to the problem of having to qualify all function names except those in the core namespace. We have not reviewed this proposal.

Fulfils Action QT4CG-048-03.

Fulfils action QT4CG-047-01 : the CG decided to rename break-when as split-when. Also applies a few minor editorial corrections in the same general area.

FO: Why is fn:op under section "17.3 Dynamic loading" ?

1. Lexical substitution has little, if anything at all, to do with (dynamic) loading. Nothing is loaded from some external resource, as in the case of fn: load-xquery-module and fn:transform.

2. There is nothing dynamic about having a predefined function that has a predefined set of possible values. In fact this could be defined as a strictly/statically defined map(xs:string, function(item()*, item()*) as item()*) with all allowed possible keys and as their values - the corresponding functions.

Taking this into account it is suggested to move fn:op to a section where it truly belongs. Maybe have in this section also other features of the language that are merely lexical substitution, as for example, function(s) for the creation of type-aliases.

Fix issue #295

The main changes are:

• In place of the special self-reference syntax "..", we now allow recursive use of type aliases, allowing types to be mutually recursive
• We generalise the places that recursive references are allowed, for example the record type used by fn:random-number-generator is now legal
• Subtyping rules for recursive record types are now defined (this was previously a gap in the specification). Acknowledgements to a John Snelson blog post for pointing me in the right direction.

Fix issue #730

Note: the issue led to a wide-ranging discussion about possible enhancements to the type system, for example adding types for empty maps and arrays. I have ignored most of this, and have focussed on fixing the issue as raised (arising originally on the test suite), namely the incorrect use of V? to define a type that allows either an instance of the sequence type V or or an empty sequence.

I propose that we should add local functions to XSLT: specifically, allowing an xsl:function declaration to appear within a sequence constructor, declaring a named function that is available for use only within the sequence constructor.

At present this can be achieved by declaring a local variable bound to an anonymous function, but it's clumsy to have to use completely different syntax for local and global functions, and functions defined in this way cannot be mutually recursive.

I propose that such functions should shadow any global functions with the same name, in the same way as happens with local variable declarations. I have an open mind as to whether shadowing of functions in reserved namespaces should be allowed.

The main difficulty is the scoping rules. We don't want the problems Javascript has with "hoisting". I propose that (a) all local function declarations must appear before any instructions (or local variable declarations, but not params) within the sequence constructor, and (b) these function declarations are in-scope throughout the sequence constructor including forwards references from the body of other functions declared earlier within the same sequence constructor.

Added fn:chain.

Took much effort to ensure Unix-style line-endings are used.

I added fn:chain, which has been discussed in https://github.com/qt4cg/qtspecs/issues/517

Minor tweaks to the spec for capture=yes accumulator rules.

Fix #731

Two little things in the spec for capturing accumulators that we agreed last week:

(a) We should define an error code for use when the capture attribute is present but phase="start".

(b) The streamability rules are too strict. They say that the select attribute must be motionless or consuming, but this is not necessary, because the select attribute is applied to a snapshot tree, which is instantiated in memory and therefore does not need to be streamable.

It is stated in XPath §3.6.4.2, and probably elsewhere, that

The function signature of a map matching type map(K, V), treated as a function, is function(xs:anyAtomicType) as V?

But V is a sequence type, not an item type, so you can't just tag a '?' onto the end of it. What is intended here by V? is a sequence type that is the union of V and empty-sequence().

The specifications (XQuery and XSLT) should say something about the effect of requesting validation on a document that contains xsi:schemaLocation and/or xsi:noNamespaceSchemaLocation attributes. At present XQuery says nothing, and XSLT says very little.

XQuery 3.1 says: A validate expression can be used to validate a document node or an element node with respect to the [in-scope schema definitions], using the schema validation process defined in [[XML Schema 1.0]] or [[XML Schema 1.1]]. This doesn't really answer the question. The "with respect to" phrase could be read as implying that ONLY the in-scope schema definitions are used. Particular problems occur if xsi:schemaLocation refers to a schema document that attempts to override or redefine the schema components that have been statically imported.

XSLT 3.0 says nothing of interest about what schema (=set of schema components) is used when validation is requested, though it does mention in passing that xsi:schemaLocation attributes might be interpreted in some way by a schema processor.

If we look to the behaviour of Saxon as a reference implementation, then we'll quickly find fault. There's a configuration option to control whether xsi:schemaLocation attributes are considered or ignored; if they are considered, then the schema components referenced are added to a global pool of schema components which are used not only to validate the document in question, but to validate any subsequent documents. We're in the process of redesigning this to do something that makes more sense.

Fix #52. Implements decision made at meeting 046.

Add a note to clarify the behaviour of load-xquery-module.

Fix #725

Add a clarification note to load-xquery-module, to correct a misunderstanding by a (very knowledgeable) user: see https://saxonica.plan.io/issues/6209

The function load-query-module does not modify the static or dynamic context in any way. In particular, the variables and functions that are loaded from the query module are not added to the static or dynamic context of the calling code. They are accessible only via the map that is returned from the function call.

Updated to take account of comments

Had this been a real emergency, we would have fled in terror and you would not have been informed.

DO NOT MERGE THIS! :-)

Maybe I'm more cleverer today than I was last time I looked into this.

It has become idiomatic to use maps, and record type definitions, to declare a collection of functions; so for example the random-number-generator object offers a "method" next() that can be called using the syntax $rng?next().

The problem is that it's not possible, within the XPath/XQuery language, to implement such a function with implicit access to the object on which it is invoked. The implementation of the function does not have access to any kind of $this variable.

This issue considers how we can move forwards from supporting simple records to introduce object capabilities, in an incremental and compatible way.

Here are three steps in that direction:

1. Where a named record type is declared, also create a corresponding constructor function. So if you declare

declare item type my:loc as record(longitude as xs:double, latitude as xs:double)

you also get a constructor function allowing my:loc(180, 180), allowing both positional or keyword arguments corresponding to the field names,

2. Allow default values to be defined in the record type, which act as default values for the parameters in the constructor function.

3. Allow functions that are defined as part of a record type access to a variable $this. The constructor function provides an implicit binding of this variable to the record/map/object that is being instantiated.

4. Allow self-reference to a named record type (and its constructor function) within the record definition.

So you can now do:

declare type my:counter as record (
   value as xs:integer,
   increment := fn() as my:counter {my:counter($this?value + 1)}
)

and then

let $x := my:counter(0)
return $x?increment()?value

which returns 1.

This PR contains error fixes (typos, examples that contradicted the spec text), some (hopefully) improved language and one breaking change.

The current draft uses the type map(xs:integer, xs:string) for the column-names option to fn:csv-to-xdm and fn:csv-to-xml. This PR flips that to map(xs:string, xs:integer). It turns out that the examples were already using this, and it seems to me that having the names entry in the csv-columns-record record type be the transposed version of the column-names option that creates it, rather than be the same thing, is counterproductive.

I can think of some examples (a CSV split into several chunks, with only the first containing the headers) where being able to feed the names entry right back into another invocation of fn:csv-to-xdm would be useful. If nothing else it's confusing and not obvious, or I wouldn't have messed up the examples, and somebody would have noticed during the review process...

Enable recursive descent transformation with template rules for maps and arrays.

Fix #570

Adds the attribute capture="yes" to xsl:accumulator-rule. This has been available as a Saxon extension for some time and makes many accumulators much easier to implement.

Fix #211

What is a generator?

Generators are well known and provided out of the box in many programming languages. Per Wikipedia:

“In computer science, a generator is a routine that can be used to control the iteration behaviour of a loop. All generators are also iterators.[1] A generator is very similar to a function that returns an array, in that a generator has parameters, can be called, and generates a sequence of values. However, instead of building an array containing all the values and returning them all at once, a generator yields the values one at a time, which requires less memory and allows the caller to get started processing the first few values immediately. In short, a generator looks like a function but behaves like an iterator.”

The goal of this proposal (major use-cases)

A generator in XPath should be a tool to easily implement the solutions to the following use-cases:

1. Processing a huge collection whose members may not all be needed.
   A generator will produce only the next member of the collection and only on demand basis.

2. Handling a collection containing unknown or infinite number of members. When requested the next member of the collection the generator will always produce it, if the collection still contains any members. It is the responsibility of the caller to issue only the necessary number of requests for the really needed next members.

What is achieved in both cases:

• A (next) member is produced only on request. No time is spent on producing all members of the collection.
• A (next) member is produced only on request. No memory is consumed to store all members of the collection.

A good problem that is based on these use-cases is to generate a collection of the first N members that have some wanted properties, and are generated from other collection(s), when it is not known what the size of the original input collections would be in order for the desired number of N members to be discovered.

For example: Produce the first 1 000 000 (1M) prime numbers.

Sometimes we may not even know if N such wanted members actually exist, for example: Produce the first 2 sequences of 28 prime numbers where the primes in each of the sequences form an arithmetic progression.

The Proposal

A generator is defined as (and synonym for):

let $generator as record
                   (initialized as xs:boolean,
                    endReached as xs:boolean,
                    getCurrent as function(..) as item()*,
                    moveNext as function(..) as .. ,
                    *  ) 

A generator is an extensible record .

It has four fixed-named keys, and any other map-keys, as required to hold the internal state of that specific generator.

Here is the meaning of the four fixed/named keys:

• initialized is a boolean. When a generator $gen is initially instantiated, $gen?initialized is false(). Any call to $gen?getCurrent() raises an error. In order to get the first value of the represented collection, the caller must call $gen?moveNext()

• endReached is a boolean. If after a call to moveNext() the value of the returned generator's endReached key is true() then calling moveNext() and/or getCurrent() on this generator raises an error.

• getCurrent is a function of zero arguments. It must only be called if the values of initialized is true() and the value of endReached is false(), otherwise an error must be raised. This function produces the current member of the collection after the last call to moveNext, if this call didn't return a generator whose endReached value was true()

• moveNext is a function of zero arguments. When called on a generator whose endReached value is false() then it produces the next (state of the) generator. including a possibly true() value of endReached and if this value is still false(), then calling getCurrent() produces the value of the next member of the collection.

Examples of operations on generators

The following examples are written in pseudo-code as at the time of writing there was no available implementation of records. And also, the code for recursion in pure XPath makes any such example longer than necessary for grasping its meaning.

The Empty Generator

emptyGenerator() {
             map{
                 initialized : true(),
                 endReached: true(),
                 getCurrent: function($this as map(*)) {error()},
                 moveNext: function($this as map(*)) {error()}
                }
              }

Take the first N members of the collection

take($gen as generator, $n as xs:integer) as generator
{
  let $gen := if(not($gen?initialized)) then $gen?moveNext()
                      else $gen,
      return
         if( $gen?endReached or $n eq 0) then emptyGenerator()
            else map{
                     "initialized": true(),
                     "endReached": false(),
                     "getCurrent": $gen?getCurrent,
                     "moveNext": take($gen?moveNext(), $n -1)
                   }
}

Skip the first N members from the collection

skip($gen as generator, $n as xs:integer) as generator
{
  if($n eq 0) then $gen
     else
     {
         let $gen := if(not($gen?initialized)) then $gen?moveNext()
                       else $gen
           return
             if(not($gen?endReached) then skip($gen?moveNext(), $n -1)
               else $gen
     }             
}

Subrange of size N starting at the M-th member

subrange($gen as generator, $m as xs:integer, $n as xs:integer) as generator
{
  take(skip($gen, $m -1), $n)
}

Head of a generator

head($gen as generator)
{
  take($gen, 1)?getCurrent()
}

Tail of a generator

tail($gen as generator)
{
  skip($gen, 1)
}

At index N

at($ind as xs:integer)
{
  subrange($ind, 1)?getCurrent()
}

For Each

for-each($gen as generator, $fun as function(*))
{
   map:put($gen, "getCurrent", function() { $fun($gen?getCurrent()) }  )                              
}

For Each Pair

for-each-pair($gen1 as generator, $gen2 as generator, $fun as function(*))
{
   let $gen1 := if(not($gen1?initialized)) then $gen1?moveNext()
                  else $gen1,
       $gen2 := if(not($gen2?initialized)) then $gen2?moveNext()
                  else $gen2,
    return
      if($gen1?endReached or $gen2?endReached) then map:put($gen1, "endReached", true())
        else map:put(map:put($gen1, "getCurrent", function() { $fun($gen1?getCurrent(), $gen2?getCurrent()) } ) ,
                     "moveNext", function() { for-each-pair(skip($gen1, 1), skip($gen2, 1), $fun)}
                    )                             
}

Filter

filter($gen as generator, $pred as function(item()*) as xs:boolean)
{
     let $getNextGoodValue := function($gen as map(*), $pred as function(item()*) as xs:boolean)
         {
            let $mapResult := iterate-while(
                                            $gen,
                                            function($gen) { not($pred($gen?getCurrent($gen))) },
                                            function($gen) { $gen?moveNext($gen) }
                                            )   
            return $mapResult?getCurrent($mapResult)                     
         },
       $gen := if($gen?initialized) then $gen 
                      else $gen?moveNext($gen)
        return
          map {
               "initialized": true(),
               "endReached":  $gen?endReached,
               "getCurrent": function($this as map(*)) { $getNextGoodValue($this?inputGen, $pred) },
               "moveNext":   function($this as map(*))
                             {    let $nextGoodValue := $getNextGoodValue($this?inputGen?moveNext($this?inputGen), $pred),
                                      $nextGen := iterate-while(
                                                                $this?inputGen?moveNext($this?inputGen),
                                                                function($gen) { not($pred($gen?getCurrent($gen))) },
                                                                function($gen) { $gen?moveNext($gen) }
                                                                )
                                    return
                                      map {
                                            "initialized": $nextGen?initialized,
                                            "endReached":  $nextGen?endReached,
                                            "getCurrent" : function($x) {$nextGoodValue},
                                            "moveNext" :   $this?moveNext,
                                            "inputGen" :   $nextGen
                                           }
                             },
               "inputGen" : $gen
              }
        }

Here are some other useful functions on generators -- with just their signature and summary:

• concat($gen1 as generator , $gen2 as generator ) - produces a generator that behaves as $gen1 until $gen1.endReached becomes true(), and then behaves as $gen2

• append($gen as generator, $value as item()*) - produces a generator that behaves as $gen until $gen.endReached becomes true(), and then as a generator that has only the single value value.

• prepend($gen as generator, $value as item()*) - produces a generator whose first value is value and then behaves as $gen.

• some($gen as generator) as xs:boolean - Produces true() if $gen has at least one value, and false() otherwise.

• some($gen as generator, $pred as function(item()*) as xs:boolean) as xs:boolean - Produces true() if $gen has at least one value for which $pred($thisValue) is true(), and false() otherwise.

• ofType($gen as generator, $type as type) - Produces a new generator from $gen that contains all values from $gen of type type -- for this we need to have added to the language the type object.

• skipWhile($gen as generator, $pred as function(item()*) as xs:boolean) - Produces a new generator from $gen by skipping all starting values for which $pred($theValue) is true().

• takeWhile($gen as generator, $pred as function(item()*) as xs:boolean) - Produces a new generator from $gen which contains all starting values of $gen for which $pred($theValue) is true().

• toArray($gen as generator) - Produces an array that contains all values that are contained in $gen.

• toSequence($gen as generator) - Produces a sequence that contains all values that are contained in $gen. Values of $gen that are sequences themselves are flattened.

• toMap($gen as generator) - If the values in $gen are all key-value pairs, produces a map that contains exactly all the key-value pairs from $gen.

These and many other useful functions on generators can and should be added to every generator upon construction.

Thus, it would be good to have an explicit constructor function for a generator:

     construct-generator($record as 
                               record( initialized as xs:boolean,
                                       endReached as xs:boolean,
                                       getCurrent as function(..) as item()*,
                                       moveNext as function(..) as .. ,
                                      )
                         ) as generator

Implements the CG decision to roll back the changes that introduced two separate default namespaces for elements and types.

Fix #372

I propose that the following attributes on an xsl:function should be accessible as annotations, for example in a call on function-annotations:

• visibility
• streamability
• new-each-time
• cache

plus any extension attribute in a user-defined namespace, for example <xsl:function saxon:debug="yes"/> should have the annotation %saxon:debug("yes"). The value is always a single string, the actual attribute value as written.

Copied from https://github.com/qt4cg/qtspecs/pull/710#pullrequestreview-1630129066:

For avoidance of doubt, we should say in XQuery 4.6.2.4 (Named Function References) that the function created by a named function reference has its annotations taken from the function definition. There are other places where we are not explicit about the annotations of a function item, for example with partial function application. We should add a note that in XPath and XSLT, it is not possible to define function annotations, so this function will always return an empty result. (However, we should consider giving user-defined functions in XSLT annotations based on their attributes, e.g. visibility and streamability).

…and my complementary note in the PR thread:

I decided to merge the PR without changes, and not add the reference to XPath and XSLT, because the function may also return results for XQuery functions imported via fn:load-xquery-module.

Additional notes from today’s meeting:

• For annotations without values, we could assign true() as a default value.
• Examples could be added to fn:function-annotations to demonstrate how to check for annotations without values (or with values whose EBV is false(): 0, "", etc.).
• Dimitre suggested restructuring the spec for features that are not available in XPath.
• Maybe annotations would also be helpful in XPath.

Feel free everyone to add more comments.

Related: #623. And an editorial note:

https://qt4cg.org/specifications/xpath-functions-40/Overview.html#func-sort

I think the type for the 3rd argument of fn:sort…

$key  as (function(item()) as xs:anyAtomicType*)+  := fn:data#1

…should be changed to zero-or-more:

$key  as (function(item()) as xs:anyAtomicType*)*  := fn:data#1

While it will be rare to encounter queries with key := (), there seems to be no urgent reason to enforce at least one sort key. The change would also be in alignment with the corresponding rule (with the current signature, $key cannot be empty), which says:

The number of sort key definitions is determined by the number of function items supplied in the $key argument. If the argument is absent or empty, the default is a single sort key definition using the function data#1.

This issue develops ideas presented in issue #596, which itself is a continuation of ideas raised in issue #341, issue #350, and elsewhere. It's related to the requirements presented in issue #262 and issue #297.

Firstly, I propose a change to the data model so that annotations can be attached to any item [or perhaps any value?], not only to a function. The annotations are a map of type map{xs:QName, item()*}. Some general principles:

• Annotations on an item do not affect the result of any operation on that item unless otherwise specified.
• Operations that are described as returning a result that contains items that are present in one of the operation's operands retain the annotations of those items, unless otherwise specified. (So for example $a[C] returns a sequence of items from $a in which the annotations are preserved).
• Operations that construct "new" items (for example $a + $b) return an item with no annotations, unless otherwise specified.
• The function annotations($x) (replacing function-annotations($x)) returns the annotations of an item.
• The function annotate($x, key, value) returns a "clone" of $x with an additional annotation. (A clone of an item differs from the original only in having different annotations. All operations other than annotation-sensitive operations produce exactly the same result on the clone and the original - including tests for node identity.)
• To avoid confusion, the term "type annotation" is replaced by "type label".

Secondly, we use annotations to aid navigation of JTrees (a term I use to describe trees of arrays and maps such as might be produced by parsing JSON).

We introduce a component of the static context tracked=true|false, defaulting to false. The construct tracked{expression} evaluates an expression and its subexpressions in tracked mode. In tracked mode any operator or function that performs selection within an array or map (for example the lookup operator, the map:get and array:get function, using the map/array as a function item, or the array:head() and array:foot() and map:find() functions) annotates the items in its result with two properties: "container" whose value is the map or array from which the item was selected, and "key" which is the key or array index of the selected value within that container. The effect is that if an item was found in a JTree using a tracked expression, the annotations on the resulting item can be used in effect to navigate upwards within the tree that was searched.

Note that this is not a new idea: in effect, the result of a tracked selection is a zipper data structure, as described in https://en.wikipedia.org/wiki/Zipper_(data_structure).

A further exploitation of the idea allows us to introduce deep update of JTree structures. For example, modify(root:=$a, selection:=fn{?x?y?z,} change:=fn{.+1}) can evaluate the selection argument in tracked mode, apply the change function to the resulting items, and then navigate back using the container annotation to create modified versions of all traversed containing JTrees, eventually returning a modified version of the root tree.

@michaelhkay Again, some hints might need to be added for XPath (?).

Based on https://github.com/qt4cg/qtspecs/issues/707#issuecomment-1721596055 and the comments following in that thread:

I've been thinking recently about adding checked{} and unchecked{} modes. For example

<xsl:apply-templates select="checked{.//item}"/>

would throw an error if there are no items. The mode of execution would propagate downwards, so checked{a/b/c/d} would be able to tell you that a b was found, but it had no c children.

checked{item[22]} would have the effect of making sequences behave more like arrays, with bound checking; conversely unchecked{item?22} would return an empty sequence instead of throwing an error.

Motivation

The motivations for this are to explore the creation of sequences where the next item in the sequence is determined by evaluating a function on the current state of a system. These sequence generators have an initial starting value and state. They can also stop when some condition is met.

The motivating example here is the fn:random-number-generator function, where:

1. let $rnd := fn:random-number-generator() initializes a new random number sequence with the default seed as its state;
2. let $value := $rnd?number returns the current value of the sequence;
3. let $rnd := $rnd?next() returns the state and value of the next item in the sequence.

This has all the properties needed for a forward (left-to-right) generating sequence. To make it a generalized generator sequence, the number field should be renamed value.

NOTE: The fn:random-number-generator function defines an infinite sequence as it has no termination/end of sequence condition.

Sequence Generators as Record Types

Therefore, a forward generator sequence could look like this:

declare item-type sequence-generator as record(
    value as item(),
    next as function() as record(value, next, *)?,
    *
);

declare function generated-sequence() as sequence-generator?;

If calling ?next() on a sequence above returns the empty sequence then there are no more values in that sequence.

If the generated-sequence() returns the empty sequence then there are no items in the sequence.

NOTE: This does not currently define reversible sequence support. A reversed property could be provided that returns a sequence-generator that operates on the sequence from right to left. I haven't figured out exactly how this should look, but reversed generator sequences may be better investigated as a separate issue.

NOTE: An implementation can process this sequence iteratively if needed.

Analysis

While this allows generator sequence types to be created, they are -- like fn:random-number-generator() cumbersome to use. This does have the advantage of being backward compatible with fn:random-number-generator(), though.

The problem comes when trying to make these work like sequences. The subtype and function coercion rules should be doable. The other sequence operations like filtering are more complicated to define properly.

However, this has the same issues that allowing array() in fn:* functions has -- how do you differentiate the use cases where the user is working on a sequence of generators, or the generated sequences?

NOTE: You can't extend the functions to take a (sequence-type | item()*) parameter as a sequence-type is a subtype of item() which matches item()*. If the functions were to be extended to handle these, then fn:head(fn:random-number-generator()) would return a number instead of a map.

Sequence Generator Function

The Kotlin language has a generateSequence function that takes a next function, an optional seed value or construction function, and returns a lazy sequence over that. -- Internally, it is building a Java iterator that produces values from calling the next function. The sequence will terminate when the next value is null.

I propose that -- in addition to the sequence-generator type above -- XPath defines the following function:

declare function fn:sequence($generator as sequence-generator) as item()*;

This solves the issues in the Analysis section above, and is analogous to array:values. It is implementation defined how the sequence is constructed. -- This allows an implementer to appropriately map the generator to their internal sequence implementation in order to provide lazy evaluation and other operations.

There should also be the following helper function for random numbers:

declare function fn:random-numbers(
    $seed as xs:anyAtomicType? := ()
) as xs:double* {
    fn:random-number-generator() => fn:sequence()
};

The user-defined sequences then become e.g.:

let $generator as sequence-generator := map {
    value : 1,
    next : function () { () }
}
return fn:sequence($generator)

A fundamental – and brilliant – property of XPath is that many operations tolerate empty sequences: Instead of throwing an error, the empty result is passed on unchanged to the next operation. While this is unrewardingly confusing for binary operations (() + 1, () eq 5), it’s wonderful for pipelines:

(: paths :)
$nodes / a / b / c
(: lookups :)
$data ? 1 ? 2 ? 3
(: simple map operators :)
$data ! do(.) ! something(.)
(: arrow operator works differently, but the syntax is similar:  :)
$data => do() => something()

As far as I can judge, it would be a very simple and user-friendly addition if we extended dynamic function calls to return an empty sequence (instead of raising an error) if the base expression is an empty sequence. This way, the following expressions would all run through:

let $map := map { 'giovanni': map { 'city': 'roma' } }
return $map('andrea')('city'),
let $data := ()
return $data(1)(2)(3),
()(123),
()()

Many people use parentheses instead of the lookup operator for accessing maps & arrays, and the proposed change would make the syntax more interchangeable. I believe it would also be useful for function items in general.

Currently, the member keyword must always be placed directly after the for clause:

(: valid :)
for $a in 1 to 10
for $m member $m in $array

(: invalid :)
for $a in 1 to 10, member $m in $array

In addition, the keyword applies to all other bindings in the same for clause:

for member $m1 in $array1, $m2 in $array2

My feeling is that this syntax is a bit odd, as other keywords (allowing empty, at) only refer to the currently bound variable. Next, the member syntax would differ from the semantics of XQuery Full Text: The score keyword is placed before the variable name, and can be used more than once (or omitted):

let score $s1 := $data1, score $s2 := $data2

I think we should change this. It would also simplify the grammar:

InitialClause     ::=  ForClause | LetClause | WindowClause
ForClause         ::=  "for" ForBinding ("," ForBinding)*
ForBinding        ::=  (SimpleForBinding | ForMemberBinding) PositionalVar? "in" ExprSingle
SimpleForBinding  ::=  VarBinding AllowingEmpty?
ForMemberBinding  ::=  "member" VarBinding
AllowingEmpty     ::=  "allowing" "empty"
PositionalVar     ::=  "at" "$" VarName
VarBinding        ::=  "$" VarName TypeDeclaration?

And one more motivation for changing is that map bindings will be easier to define (see #31).

In 4.6.4 Function Coercion, a rule was added to support functions with an arity lower than the expected one:

If F has lower arity than the expected type, then F is wrapped in a new function that declares and ignores the additional argument; the following steps are then applied to this new function.

If I got it right, this is the resulting 4.0 behavior:

Spoiler: I probably got it wrong, see the next comment.

declare function local:function($a) { };
declare variable $function := function($a) { };

(: now legal :)
filter    (1984, true#0)
$function (1984, 'ignored')
fn { }    (1984, 'ignored')
map { }   (1984, 'ignored')
true#0    ('ignored')
sum(?, ())(1984, 'ignored')

(: still illegal :)
local:function(1984, 'ignored')

(: still legal: RHS items will be supplied one by one :)
map { }(1984, 'processed')     

Maybe some more examples should be added in the corresponding sections that refer to function coercion.

The new rule is powerful and allows for greater flexibility (see #516 and other issues), but the behavior may also be unexpected. We should probably document that:

• It may go unnoticed that a passed on argument will be ignored. In other words, we reduce type safety by allowing users to supply more arguments than will be processed.
• It makes a difference whether the invoked function is static or dynamic (dynamic functions will now provide less type safety than static functions).

The specification defines Variable References for accessing values bound to a variable, and we should rename the equivalent operation for accessing the context from “Context Value Expression” to “Context Value Reference” (even more so if we should decide to introduce a Context Value Declaration later on, as discussed in #755).

Related: https://github.com/qt4cg/qtspecs/pull/703#issuecomment-1719345430

Fix #129 This replaces the previous attempt from several months ago, which had too many conflicts to be salvageable.

This is a wide-ranging and pervasive change, and I would like the changes to be applied promptly and incrementally to reduce the risk of conflicts, even if further work is needed later. This first PR addresses the XQuery and XPath language specifications. Further changes (in subsequent PRs) are needed for F+O and for XSLT. There are also a couple of minor changes affecting Serialization (but none affecting the data model).

Closes #701

With #161, we plan to introduce support for variadic functions.

The scope of this issue is much smaller and can be seen as a preparatory one; it’s about allowing the first two arguments of the function optional. I’ll create a little PR for it.

With issue #129 generalising the context item to a context value, we have the opportunity to define context-based mapping and filtering operators for arrays that work in the same way as the A!B and A[B] operators for sequences.

I propose A!!B as a mapping operator for arrays. Unlike !, this does not flatten the result. The result is an array whose members correspond one-to-one with the members of A, each member of the result array being formed by evaluating B with the corresponding member of A as the context value.

For an array whose members are singletons, the expression A!!B has a similar effect to A?*!B, but (a) it is clearer, (b) it returns an array rather than a sequence, and (c) it performs no flattening. For the more general case, it can be used in place of the higher-order function call A => array:for-each(fn{B}).

I propose A?[B] as a filter operator for arrays. Unlike A[B], this is not overloaded to perform index-based selection. The result is an array containing those members of A for which B has an effective boolean value of true, when evaluated with the corresponding member of A as the context value. The expression $A?[B] is equivalent to $A => array:filter(fn{B}).

For example, $A?[exists(.)] filters the array $A to retain only those members that are non-empty.

@ndw Sorry to keep you busy. I think we should disable the enforced signing of commits. We know the persons who send PRs, and currently we only have three persons (you, Reece, me) who sign their commits.

If signing is disabled, for example, also non-admins will be able to merge those PRs.

@ndw I’ve copied your suggestion from https://github.com/qt4cg/qtspecs/issues/645#issuecomment-1657056816 to a new issue:

we can tell git to fix the line endings in comments. I'll see about getting that setup.

In addition, it would be great if line endings were not changed when editing files in place.

Due to my changes in #645, all relevant files should now use Unix-style line endings, so this would be a sane default (e.g. if the modification of files leads to a mixture of \n and \r\n).

#645 (editorial)

Completes action QT4CG-042-02.

1. Rework the query segments so that they're a simple map of key/value pairs.
2. Rename query-segments to query-parameters.

In the XPath specs, it seems that a simple modification from...

[34] | RangeExpr | ::= | AdditiveExpr ( "to" AdditiveExpr )? -- | -- | -- | --

...to...

[34] | RangeExpr | ::= | AdditiveExpr ( "to" AdditiveExpr ("step" AdditiveExpr)? )? -- | -- | -- | --

...would be nonintrusive, and bring some nice benefits customary in other PLs, allowing expressions such as 1 to 9 step 2 and 100 to -100 step -4.

Thoughts?

Minor edits here are motivated by clarity or localized consistency. In one case, the change of 0 to 0 to 0 to 9 appears to address an important typo.

I have introduced select examples, to illustrate points in the corresponding rules.

The notes I have introduced need some context. Option 9 for the primary format token is somewhat vague, and the attached note of clarification stokes the imagination. I have trimmed that note for clarity (without substantive changes) but introduced a set of notes later, to help caution developers on unexpected behaviors with option 9, and secondarily to caution processor implementers on the challenges inherent in supporting this option. If I had my druthers, I would advocate deprecating option 9. It is--to use a technical term--squishy.

Spec editors: I am happy to pull back on any of this.

The following functions are not defined in the current spec:

• fn:unparcel, fn:parcel → dropped
• fn:xdm-to-json → #576
• fn:concat() → see #701
• fn:parts → see #463
• codepoints-to-string(), etc. (sequence-values arguments)

Completes action QT4CG-042-01 on NW.

There is a corresponding PR against the test suite.

Fix #688.

This PR fleshes out the detailed semantics of local union types, enumeration types, and type aliases. It fills a number of gaps in the current specification but doesn't aim to change the overall intent.

Clarifies the rules for constructor functions, especially for list and union types, and for types defined by means of type aliases rather than in an imported schema. Fix #687.

The current specification contains a diagnostic function called fn:stack-trace. Many other languages provide a similar function: The returned output can possibly help to understand which function calls led to an error during the evaluation of a code.

Still, I have strong doubts that it is a good decision to include this function in the standard:

The specification gives you a vast amount of freedom how to implement and optimize things. As a result, it’s completely feasible and reasonable to rewrite the following code…

declare function local:double($f) {
  $f * 2
};
(1 to 6) ! local:double(.)

…to (1 to 6) ! (. * 2) at compile time. If a user adds a fn:stack-overflow call in the function body, s·he would expect to find the function invocation of the original code representation in the output. As always, there are technical solutions to achieve this (store additional information in the evaluation tree on the original query; suppress optimizations when fn:stack-trace is found), but all of them can affect the runtime behavior and lead to different evaluation trees, hiding possible bugs in the implementation (which can be a reason to call fn:stack-trace at all).

A standard should provide a minimum amount of assurance that a function behaves similarly across different implementations. At this time, I don’t believe we can’t give that guarantee.

Related: #55, #686

– As an alternative, a stack trace could optionally be created by an implementation when an error is triggered.

The coercion rules for enumeration types have not been defined (there is a TODO in the spec).

For union types (including both schema-defined and locally-defined union types), the rules appear to need some further work. Given types R1 and R2 that are defined by restriction from B1 and B2, if an atomic value V is an instance of B1 that conforms to the rules of R1, then the relabelling coercion means it V will now be acceptable where the required type is R1. But if the required type is union(R1, R2), the relabelling coercion is not invoked. This feels inconsistent, since union(R1, R2) might be expected to accept anything that R1 accepts.

Test case FunctionCall-056 (currently failing) illlustrates the problem.

This is a deficiency in the 3.1 F+O specification.

Constructor functions for user-defined types are very poorly described:

1. It's unclear how anonymous types are handled. The spec says there is a constructor function for every simple type in the static context. That would include anonymous types. But constructor functions for anonymous types, if they exist, are essentially useless, because their names are not known.

2. The semantics of constructor functions for user-defined list and union types are described very vaguely, by analogy with built-in types; and the analogy points to the section on built-in atomic types which doesn't cover union and list types.

3. For a union type U, it says that the return type of the constructor U(x) is defined as xs:anyAtomicType. Why not define it as U? Perhaps this predates the ability to use union types as return types.

From an informal discussion on Slack, I feel that clarity is needed in the Diagnostic tracing section. The problem is that fn:trace() and fn:log() introduce the terms "trace output" with no definition or explanation, and this is easily confused with the primary output defined by the function signatures, and affects how readers think about the determinism of the functions. "The serialization of the trace output..." implies that the processor will necessarily serialize something, but I doubt that can or should be presumed. More needs to be said about the responsibilities of the processor in the contract for these functions.

In my opinion, this section would benefit from a brief preamble, providing context to set the stage for the rules. Some draft text for us to discuss:

Diagnostic tracing functions provide a transfer of information, either from the processor to the dynamic context, or vice versa.

The function that transfers information from the processor to the dynamic context, fn:stack-trace(), returns a string that can be further processed and used in the XPath expression and elsewhere in a host language.

The functions that transfer information from the dynamic context to the processor, fn:trace() and fn:log(), each have two effects. The first effect, the output, pertains to the returned values, defined by the function signature and essential to the XPath expression. Such output is always deterministic. The second effect, processor behavior, concerns the way the processor handles the values bound to the parameters, supplied for diagnostic tracing. Processor behavior is always directed toward the user or environment that invoked the processor. Actions may include sending messages, serializing the values and writing them to a log file or database, or something else. Unlike the output (the first effect), the results of processor behavior are implementation-defined and nondeterministic with respect to order of the parameter values.

The draft above attempts to avoid "output" to describe the processor-side diagnostics, so as to avoid potential confusion when dealing with the return-type defined in the signature. Where "trace output" appears in each of the rules, "processor behavior" can be used instead.

Questions:

• Any objections, corrections, or suggestions?
• Any other examples of how a processor might use trace diagnostics?
• Should a paragraph be added to explain briefly how trace() and log() differ from xsl:message? (E.g., a serialized tree should not be presumed.)
• In the informal Balisage birds-of-a-feather discussion this summer, reservations were expressed by participants about the name log(). Is it possible to drop the function and simply extend the arity of trace() with a parameter $return-input as xs:boolean? := true()?

1. Put the XSLT processor version comment at the end of the file instead of the beginning. Putting it before the <!DOCTYPE html> forces browsers into quirks mode.
2. Improve the XPath Functions stylesheets so that they don't put div elements inside p elements when outputting examples.

This is just norm hacking about

Added note clarifying the relationship between context and focus, for the purpose of illustrating the relationships between focus/context in/dependent functions. Wrapped with companion <note>s in a parent <notes>.

Fix #637

Fix #665

Fix #668

Fix #669

Fix #671

The XSLT spec has rules for the streamability of all system functions and XPath language constructs. These need updating for new 4.0 constructs.

Rework PR 664 (fix for 663) on new baseline

Fix #663.

This PR applies my action items for updating the HTML XDM mapping around namespaces and local names.

Note: this currently makes dm:namespace-nodes return an empty sequence. I'm not currently sure what the best approach is here.

Most substantive change is the trimming of prose held over from before the revision of the diagrams.

By syntax analogy with switch,

choose
   test ($a < $b) return "lesser"
   test ($a > $b) return "greater"
   test ($a eq $b) return "equal"
   default return "Getting the default is hard to explain"

Something like

ChooseExpr ::= "choose" ChooseTestClause+ "default" "return" ExprSingle
ChooseTestClause ::= "test" "(" Expr ")" "return" ExprSingle

I know I can do this by stringing if-then-else together, but would greatly appreciate the cleaner and more manageable syntax for those times when many tests are inescapable.

The trouble with XPath‘s fn:fold-right.
Laziness in XPath.

This article discusses the standard XPath 3.1 function fn:fold-right, its definition in the official Spec, its lack of apparent use-cases and its utter failure to reproduce the (lazy) behavior of Haskell’s foldr , which is presumed to be the motivation behind fn:fold-right.
The 2nd part of the article introduces the implementation of short-circuiting and generators, which together unprecedentedly provide laziness in XPath. Based on these, a new XPath function: fn:fold-lazy is implemented, that utilizes laziness, similar to Haskell’s foldr. This behavior is demonstrated in specific examples

Introduction

Higher order functions were introduced into XPath starting with version 3.0 in 2014 and later in version 3.1 in 2017.
The definition of the standard function fn:fold-right closely mimics that of Haskell’s foldr, and anyone acquainted with foldr can be left with the impression that fn:fold-right would have identical behavior (and hence use-cases) as Haskell’s foldr.

Unfortunately, there is a critical difference between the definitions of these two functions. Whereas the definition of foldr explicitly defines its behavior when provided with a function, lazy in its 1st argument – from Haskell’s definition of foldr:

“… Note that since the head of the resulting expression is produced by an application of the operator to the first element of the list, given an operator lazy in its right argument, foldr can produce a terminating expression from an unbounded list.”

The XPath definition of fn:fold-right does not mention any laziness.

There is no official concept of “laziness” in XPath, thus fn:fold-right doesn’t cover some of the most important use-cases of Haskell’s foldr , which can successfully produce a result when passed an infinite (or having unlimited length) list.

This in fact makes fn:fold-right almost useless, and explains why even some of the members of the XPath 3.1 WG have stated on occasions that they do not see why the function was introduced.

fn:fold-right gone wrong – example

This Haskell code:

foldr (\x y -> (if x == 0 then 0 else x*y)) 1 (map (\x -> x - 15) [1 ..1000000])

foldr (\x y -> (if x == 0 then 0 else x*y)) 1 (map (\x -> x - 15) [1 ..10000000])

foldr (\x y -> (if x == 0 then 0 else x*y)) 1 (map (\x -> x - 15) [1 ..])

produces the product of all numbers in the following list, respectively:

[-14, -13, -12, -11, -10, -9, -8, -7, -6, -5, -4, -3, -2, -1, 0, 1, 2, 3, …, 999985]

[-14, -13, -12, -11, -10, -9, -8, -7, -6, -5, -4, -3, -2, -1, 0, 1, 2, 3, …, 9999985]

[-14, -13, -12, -11, -10, -9, -8, -7, -6, -5, -4, -3, -2, -1, 0, 1, 2, 3, …, ] -- up to infinity.

Because all these 3 lists contain a zero as their 15th item, the expected result is 0 when evaluating any of these 3 expressions – even in the last case where the provided as argument list is infinite. And this is indeed what happens:

Not only Haskell produces the correct result in all cases, but regardless of the list’s length, the result is produced instantaneously!

Now, let us evaluate this equivalent XPath expression with BaseX:

let $product := function($input as array(xs:integer)) as xs:integer
                         { 
                           array:fold-right($input, 1, function($x as xs:integer, $y as xs:integer) as  xs:integer 
                                                               {if($x eq 0) then 0 else $x * $y}) 
                         },
    $ar := array { (1 to 36) ! function($x as xs:integer) as xs:integer {$x -15}(.)}
  return
     $product($ar)

Here we are passing a list containing just 36 integers. The result is quite unexpected and spectacular:

Here is what happens:

1. Even though when processing the 15th integer in the array the result is 0, the XPath processor continues to evaluate the RHS (right-hand side) until the last member of the array (36).

2. On “its way back” the XPath processor multiplies: (36*35*34*33*32* …*6*5*4)*3, and the result of the right-most multiplication is bigger than the maximum integer (or decimal) that this XPath processor supports.

3. C r r r a s s h … As seen in the screenshot above.

The root cause for this unfortunate behavior is that the XPath processor doesn’t support short-circuiting and laziness. And thus, fn:fold-right is useless even in the normal/trivial case of a collection (array) with only 36 members. Not to speak of collections containing millions of members, or even infinite ones…

Let us see what happens when evaluating similar expressions with another XPath processor: Saxon.

Saxon seems to produce the correct result, however it takes exponentially longer times when the length of the passed array is increased, leading to this one:

It took 261 seconds for the evaluation to be done, but accessing the 15th member of the array and short-circuiting to 0 should be almost instantaneous…
So what happens in this case? The difference between BaseX and Saxon is that Saxon implements a “Big Integer” and thus can multiply almost 1 000 000 integers without getting a value that cannot be handled… But doing almost 1M multiplications of big integers obviously takes time …

What is common in these two examples? Obviously, neither BaseX nor Saxon detects and performs short-circuiting. Why is this? What is the reason for this?

I asked a developer of BaseX if I could submit a bug about this behavior. His answer was shockingly unexpected: “This is not a bug, because no requirement in the Specification has been violated”.

Thus, the main cause of the common behavior of both XPath processors to handle the evaluation of these examples, is the specification of the function, which blatantly allows such crap to happen.

Now that we see this, let us try to provide the wanted, useful behavior writing our own function.

The fix: Step 1 – fn:fold-right in pure XPath

Before going in depth with our pure XPath solution, we need as a base a pure-XPath implementation of fn:fold-right .

 let $fold-right-inner := function ($seq as item()*,
                                    $zero as item()*,
                                    $f as function(item(), item()*) as item()* ,
                                    $self as function(*)
                                   ) as item()*
{
  if(empty($seq)) then $zero
    else
      $f(head($seq), $self(tail($seq), $zero, $f, $self))
},

    $fold-right := function ($seq as item()*,
                             $zero as item()*,
                             $f as function(item(), item()*) as item()* 
                            ) as item()*
{
  $fold-right-inner($seq, $zero, $f, $fold-right-inner)
},
               
   $fAdd := function($x, $y)  {$x + $y},
   $fMult  := function($x, $y)  {$x * $y}
   
   return
     $fold-right((1 to 6) ! function($x){$x - 3}(.), 1, $fMult)

When we evaluate the above with any of the two XPath processors, the correct result is produced:

720

And we certainly do have exactly the same problems as the provided built-in fn:fold-right with a similar example:

The fix: Step 2 – $fold-right-sc detecting and performing short-circuiting

Now that we have $fold-right as a base, let us add code to it so that it will detect and perform short-circuiting. We will implement a function similar to $fold-right but having this signature:

    $fold-right-sc := function ($seq as item()*,
                                $zero as item()*,
                                $f as function(item(), item()*) as item()*,
                                $fGetPartial as function(*)
                               ) as item()*

The last of the function’s parameters $fGetPartial returns a new function that is the partial application of $f, when its 1st argument is set to the current member of the input sequence $seq. The idea is that whenever short-circuiting is possible, $fGetPartial returns not a function having one argument (arity 1), but a constant – a function with 0 arguments (arity 0).

If the arity of the so produced partial application is 0, then our code will immediately return with the value $f($currentItem).

Here is the complete code of $fold-right-sc:

 let $fold-right-sc-inner := function ($seq as item()*,
                                       $zero as item()*,
                                       $f as function(item(), item()*) as item()*,
                                       $fGetPartial as function(*),
                                       $self as function(*)
                                      ) as item()*
{
  if(empty($seq)) then $zero
    else
      if(function-arity($fGetPartial(head($seq), $zero)) eq 0)
        then $fGetPartial(head($seq), $zero) ()
        else $f(head($seq), $self(tail($seq), $zero, $f, $fGetPartial, $self))
},

    $fold-right-sc := function ($seq as item()*,
                                $zero as item()*,
                                $f as function(item(), item()*) as item()*,
                                $fGetPartial as function(*)
                               ) as item()*
{
  $fold-right-sc-inner($seq, $zero, $f, $fGetPartial, $fold-right-sc-inner)
},
               
   $fAdd := function($x, $y)  {$x + $y},
   $fMult  := function($x, $y)  {if($x eq 0) then 0 else $x * $y},
   $fMultGetPartial := function($x, $y)
   {
     if($x eq 0)
       then function() {0}
       else function($z) {$x * $z}
   }
   
   return
     $fold-right-sc((1 to 1000000) ! function($x){$x - 3}(.), 1, $fMult, $fMultGetPartial)

Do note:

1. If the current item (the head of the sequence) is 0, then $fMultGetPartial returns a function with 0 arguments (constant) that produces 0.

2. $fold-right-sc (inner) treats differently a partial application of arity 0 from a partial application with arity 1. In the former case it simply produces the expected constant value without recursing further. Here is the relevant code fragment

  if(empty($seq)) then $zero
    else
      if(function-arity($fGetPartial(head($seq), $zero)) eq 0)
        then $fGetPartial(head($seq), $zero) ()
        else $f(head($seq), $self(tail($seq), $zero, $f, $fGetPartial, $self))

And now BaseX has no problems with the evaluation, even though the input sequence is of size 1M. The complete evaluation takes just a fraction of a millisecond (0.04 ms):

With Saxon things are not so good. Even though Saxon produces the correct result, evaluating the expression with an input sequence of size 1M takes 0.5 seconds (half a second), and evaluating the expression with an input sequence of 10M takes 5 seconds (10 times as long):

What is happening?

Even though Saxon performs much faster than the previous 261 seconds, due to detecting the short-circuiting possibility and performing the short-circuit, Saxon still processes all 10M items when evaluating this subexpression (which obviously the more optimized BaseX doesn’t do in advance):

(1 to 10000000) ! function($x){$x - 3}(.)

Therefore, we have one remaining problem: How to prevent long sequences (or arrays) from being fully materialized before starting the evaluation of $fold-right-sc ?

The fix: Step 3 – replacing collections with generators

Generators are well known and provided out of the box in many programming languages. Per Wikipedia:

“In computer science, a generator is a routine that can be used to control the iteration behaviour of a loop. All generators are also iterators.[1] A generator is very similar to a function that returns an array, in that a generator has parameters, can be called, and generates a sequence of values. However, instead of building an array containing all the values and returning them all at once, a generator yields the values one at a time, which requires less memory and allows the caller to get started processing the first few values immediately. In short, a generator looks like a function but behaves like an iterator.”

A full-fledged generator (such as implemented in C#) is an instance of a Finite State Machine(FSM), and implementing it in full generality goes beyond the topic and goals of this article. Expect another article soon that will provide this.

Here we will implement a simple kind of generator, that when passed an integer index $N, produces the $Nth item of a specific sequence. Although this is probably the simplest form of a generator, it can be useful in many cases and is a good illustrative solution to our current problem. The whole approach of replacing “something” with a function that must be called to produce this “something” is known as “lifting”

First, we will add to our $fold-right just the use of generators, without the detection and performing of short-circuiting:

let $fold-right-lifted-inner := function ($seqGen as function(xs:integer) as array(*),
                                    $index as xs:integer,
                                    $zero as item()*,
                                    $f as function(item(), item()*) as item()* ,
                                    $self as function(*)
                                   ) as item()*
                                {
                                  let $nextSeqResult := $seqGen($index),
                                      $isEndOfSeq :=  $nextSeqResult(1),
                                      $seqItem := $nextSeqResult(2)
                                    return
                                      if($isEndOfSeq) then $zero
                                        else
                                          $f($seqItem, $self($seqGen, $index+1, $zero, $f, $self))
                                },

    $fold-right-lifted := function ($seqGen as function(xs:integer) as array(*),
                                    $zero as item()*,
                                    $f as function(item(), item()*) as item()* 
                                  ) as item()*
                                  {
                                    $fold-right-lifted-inner($seqGen, 1, $zero, $f, $fold-right-lifted-inner)
                                  },
                                  
   $NaN := xs:double('NaN'),
   
   $fSeq1ToN := function($ind as xs:integer, $indStart as xs:integer, $indEnd as xs:integer) as array(*)
                {
                  if($ind lt  $indStart or $ind gt $indEnd)
                    then  array{true(), $NaN}
                    else array{false(), $ind}
                },
   $fSeq-1-6 := $fSeq1ToN(?, 1, 6),
               
   $fAdd := function($x, $y)  {$x + $y},
   $fMult  := function($x, $y)  {$x * $y}
   
   return
     $fold-right-lifted($fSeq-1-6, 1, $fMult) 

Here we see an example of a simple generator – the function $fSeq1ToN.

This function returns an array with two members: a Boolean, which if true() indicates the end of the sequence, and the 2nd member is the current head of the simulated sequence.
The generator has two other parameters which are the values (inclusive) for the start-index and the end-index. Whenever the passed value of $ind is outside of this specified range, $fSeq1ToN returns a result array with its first member set to true() (the 2nd member of the result must be ignored in this case), which indicates end-of sequence.
Otherwise it returns array{false(), $ind} . It is the responsibility of the caller to stop calling the generator:

   $fSeq1ToN := function($ind as xs:integer, $indStart as xs:integer, $indEnd as xs:integer) as array(*)
                {
                  if($ind lt  $indStart or $ind gt $indEnd)
                    then  array{true(), $NaN}
                    else array{false(), $ind}
                }

Evaluating the complete XPath expression above produces the correct result both in BaseX and in Saxon: the product of the integers 1 to 6:

Now that we have successfully implemented the last missing piece of our complete solution, let us put everything together:

The fix: Step 4 – putting it all together

Finally we can replace the input sequence in $fold-right-sc with a generator:

let $fold-right-sc-lifted-inner := function ($seqGen as function(xs:integer) as array(*),
                                    $index as xs:integer,
                                    $zero as item()*,
                                    $f as function(item(), item()*) as item()* ,
                                    $fGetPartial as function(*),
                                    $self as function(*)
                                   ) as item()*
                                {
                                  let $nextSeqResult := $seqGen($index),
                                      $isEndOfSeq :=  $nextSeqResult(1),
                                      $seqItem := $nextSeqResult(2)
                                    return
                                      if($isEndOfSeq) then $zero
                                        else
                                          if(function-arity($fGetPartial($seqItem, $zero)) eq 0)
                                            then $fGetPartial($seqItem, $zero) ()
                                            else $f($seqItem, $self($seqGen, $index+1, $zero, $f, $fGetPartial, $self))
                                },

    $fold-right-sc-lifted := function ($seqGen as function(xs:integer) as array(*),
                                       $zero as item()*,
                                       $f as function(item(), item()*) as item()*,
                                       $fGetPartial as function(*) 
                                      ) as item()*
                                      {
                                         $fold-right-sc-lifted-inner($seqGen, 1, $zero, $f, $fGetPartial, $fold-right-sc-lifted-inner)
                                      },
                                  
   $NaN := xs:double('NaN'),
   
   $fSeq1ToN := function($ind as xs:integer, $indStart as xs:integer, $indEnd as xs:integer) as array(*)
                {
                  if($ind lt  $indStart or $ind gt $indEnd)
                    then  array{true(), $NaN}
                    else array{false(), $ind}
                },
   $fSeq-1-6 := $fSeq1ToN(?, 1, 6),
   $fSeq-1-1M := $fSeq1ToN(?, 1, 1000000),
   $fSeq-1-1M-minus-3 := function($n as xs:integer)
   {
     array{$fSeq-1-1M($n)(1), $fSeq-1-1M($n)(2) -3}
   },
               
   $fAdd := function($x, $y)  {$x + $y},
   $fMult  := function($x, $y)  {$x * $y},
   $fMultGetPartial := function($x, $y)
   {
     if($x eq 0)
       then function() {0}
       else function($z) {$x * $z}
   }
   
   return
     $fold-right-sc-lifted($fSeq-1-1M-minus-3, 1, $fMult, $fMultGetPartial) 

Now this expression (and even one involving a sequence of 10M items take 0 seconds to be evaluated in both BaseX and Saxon, producing the correct result 0:

Summary

This article demonstrated the problems inherent to the standard XPath fn:fold-right and correctly determined the root causes for these problems: no short-circuiting and no collection generators.

Then a step-by-step solution was built that shows how to implement lazy evaluation in XPath based on short-circuiting and collection generators. This fixed the error raised by BaseX and dramatically reduced the evaluation time of Saxon from 261 seconds to 0 seconds.

The new function produced can be called $fold-lazy and is a good candidate for inclusion in the XPath 4.0 standard functions.

A complete design and implementation of a general collection-generator will be published in a separate article.

The word "appearing" is doubled.

The semantics of the collation URI http://www.w3.org/2005/xpath-functions/collation/html-ascii-case-insensitive are described (in F&O section 5.3.4) are described by reference to the HTML5 "living spec". The cross-reference to a changing spec is inevitably fragile and I suggest we make it non-normative. I also suggest that we define the ordering implied by this collation rather than leaving it implementation defined.

A sufficient definition is: the comparison of two strings A and B under this collation delivers the same result as the comparison of ascii-lower-case(A) to ascii-lower-case(B) under the Unicode codepoint collation, where ascii-lower-case($S) function is translate($S, "ABCDEFGHIJKLMNOPQRSTUVWXYZ", "abcedfghijklmnopqrstuvwxyz").

Perhaps we should also consider defining a collation URI that is unicode-case-blind, with the same definition except that ascii-lower-case(X) is replaced by fn:lower-case(X).

Various small edits. I added two simple examples to help readers quickly understand the difference between => and =!> before going to more complex examples.

For transition purposes, it may be useful to provide or enable "polyfill" implementations of functions that are specified in QT40, but not yet available in all implementations. Currently this is not possible because the relevant namespaces are reserved.

I propose that we relax the rule on reserved function namespaces:

(a) in XQuery, if the function has an annotation along the lines `%polyfill('http://www.w3.org/2005/xpath-functions'), where the parameter indicates that the function should be injected into the specified namespace instead of the namespace of the containing module.

(b) in XSLT, if the attribute xsl:function/@override-extension-function="no" is present. A function is allowed to be in a reserved namespace if this attribute is present.

I've chosen syntax here that's already available in 3.0/3.1, to minimise the impact on existing processors. Of course, we can't retrospectively change the 3.0/3.1 specs to authorise older processors to make this work as intended. But we can suggest that they bend the rules.

In both cases, (a) if the annotation is present then the rules on reserved namespaces don't apply, and (b) the function declaration is ignored if the processor provides its own internal implementation of the function.

I'm proposing to publish polyfill implementations of many of the new functions (not the complex ones like parse-html!). But I don't intend to make this a QT4 deliverable, I'm thinking of doing it as an open source project in GitHub/Saxonica space.

The specifications refer to $seq in place of $input.

Fix #663.

A simple fix, just specify that if xsl:original is called with keywords, the keywords used are those of the overridden function.

We need to define what happens if xsl:original() is called with keyword arguments.

The answer isn't trivial, because (for 3.0 compatibility reasons) an overriding function isn't required to use the same parameter keywords as the function it overrides.

Perhaps we should recognize that there are functions that do not support argument keywords. This might be the case, for example, with Java or C# extension functions.

Changes argument name to "value", makes argument default to context item.

Changes argument name of constructor functions to value, makes the argument optional, revises the way the proto markup is used to indicate emptyOk arguments.

Fix #658

In the current XQuery draft in 5.18.3 Function Parameters, it’ stated that:

If a parameter is optional, then all subsequent parameters in the list must also be optional. In other words, the parameter list includes zero or more required parameters followed by zero or more optional parameters.

I would suggest raising XPST0017 if that’s not the case.

Fix #647

Note: Need to check that this builds successfully. In my working copy, the new text for "import schema" found its way into the "xquery-assembled.xml" file but not into the final HTML. I can't see any reason for this.

The parameter name for constructor functions is $arg: https://qt4cg.org/specifications/xpath-functions-40/Overview.html#constructor-functions

We should change it to $value, in alignment with the XQFO functions:

fn:string(value := 123),
xs:string(value := 123)

I wonder where this has been discussed before (but I can’t find it):

For simple scripts and main modules, the necessity to prefix all functions with a local prefix is cumbersome. Next, it’s counterintuitive as the prefix is not required for variable declarations:

declare function local:f() { 1 };
declare variable $x := 1;
$x + local:f()

It’s possible to use declare default function namespace '...';, but that’s doesn’t feel much easier:

declare default function namespace 'x';
declare function f() { 1 };
f()

I wonder if we can do one of the following things?

1. Allow functions without namespace (declare function x() {}, declare function Q{}x() {}).
2. Assign functions without namespace to the default function namespace.

Uses a record type (record(key, value)) for the return type of map:pair, to give more precision and to align with map:pairs() and map:of-pairs()

See https://github.com/qt4cg/qtspecs/issues/93#issuecomment-1017937220:

One solution to more powerful sorting would a variant of fn:sort that uses a comparator function. We've resisted this in the past because we can't trust a user-supplied comparator function to be well behaved (e.g. transitive). I wonder how serious an obstacle this is?

We should try to trust; there are much more hidden pitfalls in the existing language.

fn:sort could be extended with a comparator parameter with two arguments and returning xs:boolean:

(: returns John, Joe, Jim, Jack :)
sort(('Jack', 'Joe', 'Jim', 'John'), comparator := op('>'))

(: returns -1, 2, -3 :)
sort((-3, -1, 2), comparator := fn($a, $b) { abs($a) < abs($b) })

Obviously, if a comparator is supplied, other parameters (collation(s), key(s), ascending, see #623) must not be specified in parallel.

Add covers-40 attribute to the generated test set for function keywords, to avoid failures in app-CatalogCheck test Catalog014

As an enabler for #652 (and for other reasons) I propose that XQuery should have an option to suppress recognition of entity references. This is appropriate for any context in which XQuery expressions are embedded in XML (including our own test suite), where entity expansion will have already been done before the XQuery text is parsed. With this change, any XPath expression becomes a valid and equivalent XQuery expression.

This could be done using a new prolog declaration such as

declare entity-expansion on|off;

If set to "off", & is treated as a normal character in contexts such as string literals and direct element constructors, rather than signalling an entity reference. The default remains "on".

This issue is motivated by Mary Holstege's talk at Balisage 2023: Adventures in Single-Sourcing XSLT and XQuery

https://www.balisage.net/Proceedings/vol28/html/Holstege01/BalisageVol28-Holstege01.html

It is also motivated by other issues that have been raised here proposing improved capabilities for writing applications in pure XPath.

I propose that we define a syntax for creating a module containing a set of function definitions that can be used to define a function library available to both XQuery and XSLT applications, and potentially also by pure XPath applications. A file should contain all the functions in one namespace. The format should be suitable for translation into other formats such as XSLT and XQuery, which means it should be in XML (though we could also consider JSON). We should provide XSLT stylesheets that convert libraries in this format into XSLT stylesheet modules or XQuery library modules.

Function signatures should be expressed in an XML syntax similar to xsl:function in XSLT.

Function bodies should generally be written in the form of a single XPath expression, though there should be fallback mechanisms to allow XSLT and XQuery implementations to be supplied in cases where XPath lacks sufficient expressive power.

There should be mechanisms to define the more important components of the static context, such as namespace bindings and dependencies on other function libraries.

We could consider using this format to publish "polyfill" implementations of many of the new XPath 4.0 functions.

I think we've made a mistake in the choice of name for this function. Without the namespace prefix, it looks far too much like a function that computes logarithms. Also, it would be nice in the future to find some way of allowing the math functions to be called without a namespace prefix, and this choice scuppers that possibility.

I think my choice would be fn:message.

Ensures that an xsl:fallback instruction is not processed in forwards compatibility mode, so that errors in the instruction are reported rather than being silently ignored; informally encourages adoption of the same rule in 3.0 and earlier processors where possible.

Fix #649

I made the mistake of writing a test that said:

    <xsl:array version="4.0">
      <xsl:fallback select="array{1 to 10}"/>
      <xsl:sequence select="1 to 10"/>
    </xsl:array>

Now, the xsl:fallback instruction doesn't allow an @select attribute; and there's not much point in adding one, because xsl:fallback is only there for a processor implementing XSLT 1.0, 2.0, or 3.0, and such processors will ignore anything the XSLT 4.0 specification says.

However, we could supply some clarification of what such a processor is expected to do when it finds an xsl:fallback element with an unexpected @select attribute. At present, it seems that because we don't say anything else, the xsl:fallback element is itself evaluated in forwards compatibility mode, which means that the select attribute is ignored. Since the whole purpose of xsl:fallback is to provide code that an earlier XSLT processor can handle, I think it would make much more sense to say: "the effective version for an xsl:fallback element and its descendants, unless overridden with an explicit [xsl:]version attribute, is the version of the processor in use", so a 3.0 processor executing an xsl:fallback instruction in a 4.0 stylesheet reports a static error if it finds a construct like the above.

Although the 4.0 spec cannot dictate what a 3.0 processor does, we could add a non-normative note encouraging this interpretation.

Weird things can happen if the user defines a schema that imports the schema for the FN namespace and then adds members to its substutition groups or extends its complex types. We can prevent this happening by blocking substitution and extension. We should also specify that when we validate the input to fn:xml-to-json, xsi:schemaLocation must be disabled.

XQuery 3.1 clarified what is intended when an "import module" declaration provides multiple location hints - there's now a clear indication that the processor should expect to load multiple modules all with the same module namespace.

For "import schema" it's still completely vague what is intended, and there's no analogy in XSD or XSLT which only ever allow a single location URI to be supplied. Currently we say:

The URILiterals that follow the at keyword are optional location hints, and can be interpreted or disregarded in an implementation-dependent way. Multiple location hints might be used to indicate more than one possible place to look for the schema or multiple physical resources to be assembled to form the schema.

I propose changing this to:

The URILiterals that follow the at keyword are optional location hints, intended to allow a processor to locate schema documents containing definitions of the required schema components in the target namespace. Processors may interpret or disregard these hints in an implementation-dependent way. The recommended default strategy is as follows (but this may be varied through user options):

• All the location hints are dereferenced, treating them as relative URIs
• If any location hint cannot be dereferenced, or fails to resolve to a valid schema document with the required target namespace, then that location hint is disregarded (optionally with a warning); but if none of the location hints can be resolved to a valid schema document with the required target namespace, then a static error is reported.
• If multiple location hints are dereferenced, yielding multiple schema documents A, B, and C, then they should be treated as if there were a single schema document containing xs:include declarations referencing A, B, and C. This implies that the schema documents must together comprise a valid schema, for example there cannot be two different type definitions with the same name.
• Notwithstanding the previous rule, if a processor is able to establish that two or more location hints refer to identical or equivalent schema documents, then the duplicates should be ignored.

This text gives users and implementors alike a much better sense of what is intended, while still retaining flexibility for implementations to do something different.

@ndw @michaelhkay I wonder what we should do with minor edits like this (which fixes an example)? Should we merge them by ourselves, or wait for someone to approve and merge it?

If we use a consistent newline encoding in all XML documents, it will be easier to perform cleanups and to edit documents in the GitHub frontend. Most documents already use Unix-style newlines. If I am correct, only three documents need to be updated:

specifications/xslt-xquery-serialization-40/src/errors.xml
specifications/xslt-xquery-serialization-40/src/ns-xslt-xquery-serialization.xml
specifications/xslt-xquery-serialization-40/src/xslt-xquery-serialization.xml

Per meeting today, made good on my comment in #546

Old -> lambda syntax removed from various examples; minor unifications.

I believe it’s ready to merge; please jump in otherwise.

I propose that we drop some serialization errors in favour of producing a fallback representation of the supplied value.

The rationale is that (a) serialization is often used in contexts like xsl:message where the primary purpose is diagnostic, and the last thing you want when producing diagnostics is a secondary error; and (b) seeing a fallback representation of an inappropriate value often shows you much more clearly what you have done wrong than any error message can do.

Compare with the .toString() method in Java and similar languages, which always outputs something even if it's not quite what you wanted.

I'm not proposing to change the principle that the output should always be syntactically valid (e.g. well formed XML or JSON).

I think some of the specific error conditions we might drop are:

• In sequence normalization rule 7, instead of raising an error when an attribute, namespace, or function (including a map or array) is encountered, serialize that item using the adaptive output method, treat the result as a text node, and insert the text node into sequence S6.

• In the JSON output method: when a sequence of two or more items is encountered, instead of raising SERE0023, treat it as an array containing those items.

Closely related, and perhaps best considered together: should the fn:string() function accept anything as input, and never raise an error?

@ndw Should be ready to be merged

A new function fn:void was added to the spec (see #359 for details).

This issue can be used to discuss alternative names for the function, as was suggested by @dnovatchev.

Occurrences of e.g. should be replaced with alternatives, such as (for example).

See https://github.com/qt4cg/qtspecs/pull/629#issuecomment-1649952964

Functions annotations in XQuery have become a popular feature to attach vendor-specific information (for unit testing, locking, RESTXQ, etc.) to functions.

Annotation values are limited to literals, though. It would often be helpful to supply boolean values, but we don’t have literals for that in the language.

I suggest enhancing the existing grammar…

Annotation  ::=  "%" EQName ("(" Literal ("," Literal)* ")")?

…and allowing the strings false() and true() as values:

Annotation  ::=  "%" EQName ("(" AnnotationValue ("," AnnotationValue)* ")")?
AnnotationValue  :=  Literal | "false()" | "true()"

The suggestion is upward compatible if we should decide later on that we want to allow arbitrary expressions for annotation values.

Per #171 the WG decided to allow the ternary operator. I'm looking at 4.16 of the current version of XPath 4.0 and the ternary operator is presented as illustrative, but the operator does not appear to have been properly introduced and defined in the specs. The terms "ternary", "??", and "!!" appear only twice each, and in contexts that could be confused as being illustrative.

By my reading, the definition of [11] ExprSingle should be expanded to allow a new option, call it TernaryOption, and to the grammar should be new entry TernaryOption ::= Expr "??" ExprSingle "!!" ExprSingle. Does such a definition allow any ambiguous constructs?

I propose that 4.16 be subdivided into two subsections, the first one briefly introducing the ternary operator and the second handling if/then statements. Slight aside: the latter should also include a note pointing out how to avoid a then branch (and an example), a courtesy to developers who are overly accustomed to the unbraced approach as to not give the braced approach much thought.

This PR addresses part (not all) of issue 451.

It recognises that an application may use more than one schema; for example in a pipeline using multiple stylesheets, it must be possible for the first stylesheet to produce valid output that is valid input to the second, without requiring that the two stylesheet have absolutely identical schema imports. It recognises that there are cases (for example involving substitution groups) where two schemas X and Y may both include the same type T, but produce different results when an element is validated against T. So it defines a concept of schema compatibility and defines its limitations, especially on the semantics of item types such as element(*,T) and schema-element(E). The rules for schema compatibility between different modules of a query and between different packages in a stylesheet are tightened up and brought into line with each other.

See Matt’s comment in https://github.com/qt4cg/qtspecs/pull/533#issuecomment-1647945368

Note, this batch of edits includes a shift from "built-in" to "system" when describing functions, per edits made in XDM.

Observed by @line-o: https://xmlcom.slack.com/archives/C01GVC3JLHE/p1689946671105499

I did my best to define rules for a counterpart of the fn:encode-for-uri function, including various edge cases.

I’m convinced that the function has been requested often enough to justify its inclusion in the spec. I’m also aware that users may have different expectations regarding the details of the conversion rules. On the other hand, this discussion can be observed for other languages as well, and that’s mostly due to the… heterogeneous history of URIs, not the actual implementations. For example, URLDecode.decode in Java converts the plus character to a space, and JavaScript’s decodeURI adopts it unchanged. I decided to convert it as well, as fn:encode-for-uri encodes the plus sign to %2B.

@ndw My rules have largely been inspired by your decoding rules for fn:parse-uri. I hope these rules can be dropped and replaced with a reference to this new function (analogous to fn:build-uri, which references fn:encode-for-uri).

Minor edits to ch. 3 of XPath spec. Note, this PR includes an adjustment to the XDM spec, in the form of a cross-reference, because the terms "string value" and "typed value" are heavily used in XDM but defined only in XPath.

• New function fn:log
• Rules of fn:trace revised

I've noticed that a few tests have appeared in QT4tests distinct-values() that assume the order of results is "order of first appearance" (search for assert-deep-eq). We should either change the tests, or change the spec to require order of first appearance.

Since no implementors have objected to these tests, it seems likely that implementations are delivering results in "order of first appearance", and if that is the case, then I think it would be a convenience to users to guarantee this in the spec.

To allow for parallel implementations, we could say that the order is undefined if ordering mode is unordered.

Attempted revision in light of #624. Perhaps not everything is exactly right, but it should be a step in the right direction.

This brief edit addresses issue #556, which I've chosen to handle apophatically.

All minor edits. Some edits are motivated by an attempt to address broken parallelism or some other form of localized inconsistency. Does not include questions about function descriptions, raised at #624.

In the XPath specs, 2.2, function definitions, the reader is informed that every (statically known) function definition takes one of three mutually exclusive categories: application, system, or external.

[Definition: Application functions are function definitions written in a host language such as XQuery or XSLT whose semantics are defined in this family of specifications. Their behavior (including the rules determining the static and dynamic context) follows the rules for such functions in the relevant host language specification.]

The first sentence appears to point to user-written functions in the host languages, and the second sentence appears to point to functions defined by the host language specifications. I assume that this category is meant to include both, and I propose the language be tightened up to make that clear. If my assumption is incorrect, then some other type of revision is needed.

Later on, when the term “built-in function” is introduced, it is not as clearly stated as it should be how that term maps onto the three-way division. It seems that “built-in function” encompasses all system functions and only those application functions that are defined by the specifications, and not user-written functions. Whatever the case, and whatever the best path of revision, this paragraph would be most effective if moved up with the tripartite category discussion.

The term "external function" is introduced here in the static context, with language that is highly suggestive of a definition. But the definition proper is reserved for the dynamic context, and in shorter, different prose that I think loses some of the considerations provided in the static context. I propose that the two passages be consolidated and located in the static context, where the two other types of functions are defined, with an xref in the dynamic context to the meaning of external function.

Down in the dynamic context:

The dynamically known function definitions may include external functions.

This sentence is a bit puzzling, because of course they are allowed to include all three types of functions -- none are forbidden, since, after all, the dynamically known functions are a superset of statically known ones. Perhaps the point is to drawn the reader's attention to those functions that are known dynamically but not statically? Perhaps this revision?: "Many of the function definitions known dynamically but not statically will be external functions, but they may include user-written application functions written in a host language, known only dynamically, e.g., through fn:transform."

Cumulatively the above points could go beyond mere minor touch-up, so comments are welcome before I attempt any edits.

Enhances fn:sort to allow multiple major-to-minor sort keys each of which can independently specify a collation and an ascending/descending option.

Also includes infrastructure changes to allow occurrence indicators on function arguments or results that reference a named record type.

Similar changes will be needed for array:sort; to reduce the risk of rework I propose to make those changes after this PR has been reviewed and accepted.

Fix #93

Note, the example was invalid because of a failure to access the included xsd file.

As noted in Slack, chapter 4 of the XDM specs is out of place. In this PR I have moved the very general material in chapter 4 to the preamble of chapter 6 (now 5), which provided the opportunity to orient the reader to the structure of that chapter.

Cross-references to chapter 4 have been search for and dealt with. The CSS deletion comes from my observation that there is no such class infoset-mapping in the resultant HTML file; section 4's infoset-mapping is an id.

Per #616 names of nodes have been set lowercase. This PR does not address the good suggestion that xrefs and styling be selectively applied. That is reserved for a future pass.

If we keep fn:html-parse and if we add fn:csv-parse, we should also add fn:html-doc and fn:csv-doc.

See also Issue #397 and Issue #322, which this proposal may partially supersede.

I propose that when declaring a named record type in the static context, this should automatically create a constructor function definition for records of that type.

So in XQuery if you write

declare item type my:location as record(longitude: xs:double, latitude: xs:double);

then the static context acquires both a type (let $loc as my:location := ....) and a function which you can call with either positional or keyword arguments (let $loc := my:position(-2.03, 50.95) or := my:position(longitude := -2.03, latitude := 50.95).

The semantics of the function are roughly map{longitude: -2.03, latitude: 50.95} treat as my:position, except that the function arguments are first coerced to the required types (in this case the decimals are coerced to double).

If the record type is extensible, the constructor function does not provide any capability to set values for extension fields. I'm not sure yet whether it will be possible to distinguish fields set to an empty sequence from fields that are absent.

This is consistent with user-defined atomic types where you automatically get a constructor function.

Similarly for named union types. If you declare

declare item type my:binary as union(xs:hexBinary, xs:base64Binary)

then you should automatically get an arity-1 constructor function my:binary($value) with the same semantics as if my:binary were an XSD-defined union type (that is, the same semantics as cast $value as union(xs:hexBinary, xs:base64Binary).

At the risk of being branded a pedant....

Currently the XDM specs have a predilection for Attribute Node, Element Node, Comment Node, Document Node, Text Node, Processing Instruction Node, Namespace Node. But in a healthy minority of cases in the XDM specs, they are rendered lowercase.

To complicate matters further, the other specs, which rely upon the XDM specs for the definition of the terms, universally prefer the lowercase form, and they use the terms preponderantly more. A healthy sample of four of the terms across five of the specifications ("elective" excludes instances where that particular case is required, e.g., capitalization in headers):

Term | Spec | Capitalized | Electively Capitalized | Electively Noncapitalized -- | -- | -- | -- | -- Attribute node | XDM | 71 | 41 | 18 Element node | XDM | 79 | 71 | 23 Comment node | XDM | 44 | 39 | 0 Document node | XDM | 55 | 48 | 10 Attribute node | Serialization | 0 | 0 | 15 Element node | Serialization | 0 | 0 | 39 Comment node | Serialization | 0 | 0 | 2 Document node | Serialization | 0 | 0 | 15 Attribute node | XQuery | 1 | 0 | 82 Element node | XQuery | 0 | 0 | 141 Comment node | XQuery | 0 | 0 | 9 Document node | XQuery | 2 | 0 | 50 Attribute node | XFO | 1 | 0 | 28 Element node | XFO | 1 | 0 | 67 Comment node | XFO | 1 | 0 | 4 Document node | XFO | 1 | 0 | 57 Attribute node | XSLT | 8 | 0 | 100 Element node | XSLT | 16 | 0 | 77 Comment node | XSLT | 1 | 0 | 6 Document node | XSLT | 12 | 0 | 144

Options:

1. Do nothing.
2. Change all specs to capitalize these terms universally.
3. Change only XDM to capitalize these terms universally.
4. Determine a principle that would differentiate the contexts in which the term should be capitalized or not within XDM only.
5. Change all specs to uncapitalize these terms universally.

My personal preference is no. 5. I do not see any passages in XDM where the reader would be confused by the term being uncapitalized (the minority cases being witnesses). Lowercase would bring XDM into conformity with the other specifications, and with how the term is commonly used outside the specs. Further, the XDM specs do not capitalize other terms it specially defines, e.g., accessor. And in all the specs, including XDM, the clear preference is to use the lowercase for the names in the raw (i.e., without "node"): element, attribute, namespace, etc.

I wanted to bring this to the group before doing any edits. I am happy to do the work, but do not want to do it if it is unwelcome, or if there is a clear preference for another option.

I decided to create a PR for the initial proposal of this function, as I came across at least two other use cases for it since the issue was created.

I believe that the group by clause is the best choice for more complex operations, such as advanced comparisons or creating histograms.

It seems silly to allow "union" as a synonym for "|" in some places and not others. For example it's not allowed in a "case" clause of a typeswitch, nor in a catch clause of try/catch.

We've introduced the ability to write child::(a|b) as a synonym for child::a | child::b, but we don't allow child::(a union b) as a synonym for child::a union child::b.

1. map:of renamed to map:of-pairs (as a hint that the input must match a specific format)
2. array:of renamed to array:of-members (as a hint…)
3. add map:pair for creating a single pair
4. add array:split for decomposing arrays to singleton arrays

Bugging you again, @ndw …

My feedback has been triggered by #607. I’ve moved various code examples into eg blocks. In principle, the result is promising, but it produces cases such as the following one (see https://qt4cg.org/pr/607/xpath-functions-40/Overview.html):

Maybe the result is better to read if single-line and multi-line monospace is formatted identically (i.e., without lines at the top and bottom, and with a transparent background)?

Next, it would be fine if we could get rid of the scrollbars. Often, it’s not apparent from the rendering that the presentation includes results at all:

We could possibly fix it by wrapping even more examples manually. That can be challenging, though, for example if long strings are used (such as "http://www.w3.org/2013/collation/UCA?lang=en;alternate=blanked;strength=primary").

Finally, the formatting on mobile devices (here: an Android tablet with a Chrome renderer) differs from the desktop view: code blocks are rendered much smaller than the rest. – That’s just to mention it; I’d probably be annoyed if I was tasked with fixing that.

This PR contains lots of fixes in the XQFO examples and improves the formatting of examples with nested arguments and multiline expressions.

It’s probably helpful to merge it as soon as possible to avoid conflicts with other PRs.

Fix #23

Fix #21

Fix #39

Fix #602

Strict static typing, as originally defined for XQuery 1.0, was not a success, because it prohibits too many constructs that are perfectly reasonable to write. However, the alternative, pure dynamic typing, prevents a processor reporting many obvious errors at compile time. A compromise is optimistic static typing, where the processor is allowed to report a type error statically in the cases where it can be shown that evaluation of an expression is bound to fail at run-time.

Optimistic static typing has proved a reasonably successful compromise, but there are a number of cases where things that are obviously user mistakes cannot be reported as static errors. I propose that a processor should be allowed (not required) to treat some of these conditions as static errors.

The first of these conditions is exemplified by passing an argument whose static type is xs:integer* to a function where the declared parameter type is xs:string*. Under optimistic static typing this cannot be reported as a static error, because it is not bound to fail; if the actual value at run-time turns out to be an empty sequence, the call will succeed. So the proposal is that where the inferred supplied type and the required sequence types are both emptiable (that is, occurrence indicator is "?" or "*"), but their respective item types are disjoint, the processor should be allowed to report a static error.

The second condition is what I call a void path expression. Specifically, if we know statically that the result of $A/B, or $A!B, or $A?B will be an empty sequence for any possible value of $A (given its inferred type), this almost certainly means the user has made a mistake, and we should be allowed to report a static error. This extends to the unary or implicit forms of these operators, based on the inferred type of the context item. This is most likely to occur with schema-aware code, where it should be possible to report a path such as A/B/C/D as incorrect if the schema does not allow such a path. But it also arises for example for $A?B if the inferred type of $A is a non-extensible record type and B is not one of its known fields; and it arises for inappropriate combinations of axes such as @code/text().

Note that I'm not proposing (as XQuery 1.0 static typing did) that any expression whose result is bound to be empty is a static error; the rule is confined to a few specific operators.

Perhaps, for backwards compatibility and interoperability, we should require processors to provide an option to switch this kind of static error detection off. (For example, XSLT 1.0 code sometimes deliberately uses /.. to represent an empty sequence, and this construct would be flagged under these rules.)

Feedback we got (translated):

There are the every and some keywords, and there are the new functions fn:some and fn:all. They appear to be more or less similar, so it would be consistent if fn:all was renamed to fn:every.

If fn:all is called this way because there’s also fn:all-different and fn:all-equal, we should also have fn:some-different and fn:some-equal.