@qt4cg statuses in 2020

The TypeswitchExpr symbol has changed as follows:

-TypeswitchExpr ::= "typeswitch" "(" Expr ")" CaseClause+ "default" ("$" VarName)? "return" ExprSingle
+TypeswitchExpr ::= "typeswitch" "(" Expr ")" CaseClause+ "default" ("$" VarName)? "$" VarName "return" ExprSingle

The CaseClause symbol has changed as follows:

-CaseClause ::= "case" ("$" VarName "as")? SequenceTypeUnion "return" ExprSingle
+CaseClause ::= "case" ("$" VarName "as")? "$" VarName "as" SequenceTypeUnion "return" ExprSingle

That is, both of these have a repeated "$" VarName ... section from the previous optional section.

Is this an effect of removing the references to the older specifications from the grammar?

The SingleType definition has changed as follows:

-SingleType ::= SimpleTypeName "?"?
+SingleType ::= (SimpleTypeName | LocalUnionType) "?"? 	

However, SimpleTypeName has also changed:

-SimpleTypeName ::= TypeName
+SimpleTypeName ::= TypeName | LocalUnionType 	

Therefore, the change to SingleType is not needed.

Comparing the WithExpr grammar, XPath has:

[10] NamespaceDeclaration ::= QName  "="  StringLiteral

whereas XQuery has:

[44] NamespaceDeclaration ::= QName  "="  URILiteral

Therefore, it looks like the intention is to use URILiteral in the XPath EBNF grammar for NamespaceDeclaration as well.

This uses schema-for-xslt30.xsd as the basis for adding the new elements and attributes in the current XSLT 4.0 draft.

Given a function that returns a sequence, array, or map of a fixed length or structure, it would be useful to extract those values in a destructuring declaration like can be done in other languages (such as JavaScript, Kotlin, C++, and Python). For example:

let $(sin, cos) := sincos(math:pi()) (: sequence :)
let $[x, y, z] := camera-angle() (: array :)
let ${r, i} := complex(1, 2) (: map :)

These would be equivalent to:

let $ret := sincos(math:pi()), $sin := $ret[1], $cos := $ret[2] (: sequence :)
let $ret := camera-angle(), $x := $ret?(1), $y := $ret?(2), $z := $ret?(3) (: array :)
let $ret := complex(1 ,2), $r := $ret?r, $i := $ret?i (: map :)

It should be possible to define the type of a component and/or the whole construct:

let $(sin as xs:float, cos) as xs:float* := sincos(math:pi()) (: sequence :)

For maps, it would also be useful to rename the components, such as:

let ${re := r, im as xs:double := i} := complex(1, 2) (: map :)

It should also be possible to capture any left-over items in the sequence/array/map, for example:

let $(headings, rows) := load-csv("test.csv")

A destructuring declaration should be usable anywhere a variable binding can be defined.

It should not be an error to use the same variable name twice. This supports conventions such as using _ for unused values. For example:

let $[_, y, _] := camera-angle()

Requirements/Use Cases

1. It should be possible for a library or application written in XQuery to define, access, and use custom annotations without relying on vendor extensions.
2. It should be possible for a processor or editor/IDE to check and verify user-defined annotation usage (can an annotation be used multiple times; what argument values/types are valid) to provide better validation for custom annotations.
3. It should be possible to access the details of the annotations on a function (name, argument values) so tools like xqDoc can read and format annotations without using vendor extensions.

Annotation Declarations

1. An annotation declaration is a function declared using the %annotation annotation.
2. Annotation parameters must be a SequenceType that has an ItemType which is either a) a subtype of union(xs:string, xs:double, xs:float, xs:decimal)*, or b) item() (to denote any permitted literal value). Note: This is because annotation values are restricted to literal values. Note: It is therefore possible to use an EnumerationType to define a set of allowed string values (e.g. "yes" and "no").
3. If a function annotation name matches an annotation declaration in the statically-known functions using the function name resolver, it is verified using the same rules for static function calls; if there are no matching annotation declarations, then no error is issued, although an implementation may choose to issue a warning.

Annotation Tests

1. A new AnnotationTest (annotation(...)) SequenceType is provided with the same structure and semantics as a FunctionTest.
2. AnnotationTest parameters must be a SequenceType that has an ItemType which is either a) a subtype of union(xs:string, xs:double, xs:float, xs:decimal)*, or b) item() (to denote any permitted literal value). Note: This is because annotation values are restricted to literal values. Note: It is therefore possible to use an EnumerationType to define a set of allowed string values (e.g. "yes" and "no").

fn:annotations

fn:annotations($f as function(*)) as annotation(*)*

Returns all the annotations on the function.

The annotation signature will match the signature of the annotation, so %a(1, 2) will have the annotation(xs:integer, xs:integer) annotation type.

If a function has multiple annotations with the same signature (e.g. %values(1) %count(2)) then two annotations will be returned with the same signature, referring to the different annotations.

fn:annotation

fn:annotation($f as function(*), $name as xs:QName) as annotation(*)*

Returns all the annotations on the function with the name $name.

The annotation signature will match the signature of the annotation, so %a(1, 2) will have the annotation(xs:integer, xs:integer) annotation type.

If a function has multiple annotations with the same name and signature (e.g. %values(1) %values(2)) then two annotations will be returned with the same signature, referring to the different annotations.

fn:annotation-name

fn:annotation-name($annotation as annotation(*)) as xs:QName

Returns the name of the annotation.

fn:annotation-arguments

fn:annotation-arguments($annotation as annotation(*)) as array(*)

Returns the arguments passed to the associated annotation.

1. Searching the F & O document for the union ( | ), intersect and except operators finds 0 occurrences. Neither can , be found, and of course, it is unsearchable.

2. In the XPath spec these are described in section "4.7.3 Combining Node Sequences". The word node-set is not mentioned at all and this is misleading, because the union ( | ), intersect and except operators are in fact set-operators, or at least set-generating operators.

Thanks, Dimitre

It is high time that we come up with a set type in XPath. We actually have to deal all the time with sets (not just node-sets, but sets of any-type values), and it is painful to read in the spec how two maps are compared for equality when explaing fn:deep-equal():

"If $i1 and $i2 are both ·maps·, the result is true if and only if all the following conditions apply:

Both maps have the same number of entries.

For every entry in the first map, there is an entry in the second map that:

has the ·same key· (note that the collation is not used when comparing keys), and

has the same associated value (compared using the fn:deep-equal function, under the collation supplied in the original call to fn:deep-equal)."

When if we had the set type the above would simply say:

"If $i1 and $i2 are both ·maps·, the result is true if and only if the sets of their keys are equal, and the corresponding values for each key in the two maps are deep-equal."

I propose that starting with XPath 4.0 we introduce the set type and define set equality, the union ( | ), intersection (intersect) and set difference (except) not only for node-sets but for sets of any-typed values.

Then we can have a function: to-set($collection as item()*) as set, which would produce a set (of the distinct values) of any collection-typed argument supplied to it: sequence(its distinct values) , map (a set of its entries), array (a set of its members).

This makes fn:distinct-values() almost unnecessary.

We will no longer have to explain in a "Remarks" section that the result of a function is "unordered" or that its order is "implementation-defined" -- just by making this function return a set.

How can almost all major programming languages (not even speaking of SQL), such as C#, Python and Java have a set data type / interface, but even in XPath version 4 we still have to describe it in a free language narrative?

Thanks, Dimitre

To-dos:

• Add number-formatter to the json method in the serialization spec.

Sometimes people put very large numbers in a JSON file, if they are parsed as double, the numbers are corrupted. Or they become confused when 1000000 from the input becomes 1e6 in the output. Finally parsing a double is slower than parsing an integer.

There could be an additional option for parse-json/json-doc number-type with possible values:

• double: Parse all numbers as xs:double
• decimal: Parse all numbers as xs:decimal
• string: Return numbers as xs:string (so 1e6 stays "1e6" and 1000000 stays "1000000")
• auto: Parse numbers containing e as xs:double, numbers containing . as xs:decimal, and numbers containing neither as xs:integer

Within the scope of a catch clause, some variables are implicitly declared. However, it’s more than tedious to forward all this information to another function:

try {
  'oh' + 'dear'
} catch * {
  handle-error($err:code, $err:description, $err:line-number, $err:column-number, ...)
}

What about binding all data to an additional $err:map variable?

map {
  'code': $err:code,
  'value': $err:value,
  'description': $err:description,
  'line-number': $err:line-number,
  'column-number': $err:column-number,
  'additional': $err:additional
}

The code could then be shortened to:

try {
  'oh' + 'yes'
} catch * {
  handle-error($err:map)
}

Edit: Current proposal (https://github.com/qt4cg/qtspecs/issues/31#issuecomment-1613342841):

for key $key in ...
return …

for value $value in ...
return …

for key $key value $value in ...
return …

With the addition of the for member syntax for arrays, it is possible to use a ForExpr/FLWORExpr to enumerate the contents of sequences and arrays, but not maps. In order to be consistent and symmetric across these types, the for member syntax should be extended to support maps by enumerating the key/value entries of the map.

Given a map of type map(K, V) the member RecordTest would be record(key as K, value as V). Given a map of type map(*), the member RecordTest would be record(key, value).

This would allow a user to write expressions like:

for member $entry in $map
return element { $entry?key } { $entry?value }

NOTE: With the addition of the array:values and map:entries functions in issue #29, it is possible to avoid the need of the for member syntax for arrays and maps, but may be worth keeping for people who prefer the wordy style of the XPath/XQuery syntax.

In order to keep the XPath and XQuery languages parseable without the need of a pre-processing step, the operators should be integrated into the grammar, such as in:

[29] UnionExpr ::= IntersectExceptExpr ( ("union" |  "|" | "∪") IntersectExceptExpr )*

To make them more easily discoverable to someone reading the specification, the behaviour of the Unicode aliases should be included the description in the text related to that section. For example:

The union, |, and ∪ operators are equivalent. They take two node sequences as operands and return a sequence containing all the nodes that occur in either of the operands.

NOTE: With the introduction of the LocalUnionType, a pre-processing step would need to map ∪ to | in order to avoid ∪(xs:string, xs:integer) being valid. -- This would then mean that things like the CatchErrorList would then be able to use ∪, as in try {} catch err:FOER0001 ∪ err:FOER0002 {}.

EDIT: Revised and aligned with the recently added and proposed map and array functions (#314, #357):

The suggested functions are based on the observations that:

• users who prefer readable function names tend to reject ?* as handy shortcut;
• functions are often more flexible and better composable than FLWOR enhancements (but of course both could be added).

array:values

Summary

Returns all members of an array as a sequence. Equivalent to $array?*, but better composable and easier to read (especially for newcomers), and known from other programming languages.

Signature

array:values($array as array(*)) as item()*

Example

(: Query :)
let $array := [ (), 2, [ 3, 4 ] ]
return array:values($array)

(: Result :)
2
[ 3, 4 ]

map:values

Summary

Returns all values of a map as a sequence. Complementary to map:keys, and equivalent to $map?*, but better composable and easier to read (especially for newcomers), and known from other programming languages.

Signature

map:values($map as map(*)) as item()*

Example

(: Query :)
let $map := map { 'a': (), 'b': 2, 'c': [ 3, 4 ] }
return map:values($map)

(: Result :)
2
[ 3, 4 ]

map:entries

See #357 on composing and decomposing Key-Value Records.

Summary

Returns each entry of a map as a singleton map. Equivalent to map:for-each($map, map:entry#2).

Signature

map:entries($map as map(*)) as map(*)*

Example

(: Query :)
let $map := map { 'a': (), 'b': 2, 'c': [ 3, 4 ] }
for $entry in map:entries($map)
return element { map:keys($entry) } { string-join(map:values($entry)) }

(: Result :)
<a/>
<b>2</b>
<c>34</c>

array:members

See #314.

This implements the proposal in issue #22 to support multiple clauses in XPath ForExpr and LetExpr, with both the grammar and specification text changes.

This fixes issue #24.

In another issue ( #25 ) it was already discussed that the proposed "sequence-variadic" functions have definite problems.

This issue is about the incorrect value in the MaxK column of the "sequence-variadic" (last) row of the table titled "Number of Arguments allowed in a Function Call", which is in the section "4.4.1 Static Functions".

As published, the value of the 7th (last, for the MaxK column) item in this row is 0.

In fact all of the required positional arguments can be entered as keyword arguments in a function call (there is no reason why this cannot/shouldn't be done).

Thus, the value for MaxK (the maximum number of keyword arguments) must be R, (the current value of 0 is wrong).

Proposed corrective action:

1. Replace "sequence-variadic" with "array-variadic", which avoids the problems of "sequence-variadic" (as discussed in #25 )
2. In the row for "array-variadic" of the table, specify R as the value for the column MaxK

The current specification for sequence variadic functions (at: https://qt4cg.org/branch/master/xquery-40/xpath-40-diff.html#id-static-functions) has the following issues:

1. An empty sequence cannot be specified into the sequence of variadic arguments
2. Even the last (sequence) argument that is supposed to contain the variadic arguments "disappears" when no variadic argument is specified on the function call

This results in difficulties to describe, explain and implement such sequence-variadic functions, and in inability to provide any argument value (like the empty sequence) within the last, sequence argument.

Proposed improvement, which doesn't have the above problems:

Use array-variadic functions instead of sequence-variadic:

%variadic("array") indicates that the function is array-variadic. An array-variadic function declares one or more parameters, of which the last must be an array. If the declaration includes N parameters, then a call on the function may supply N-1 or more arguments. However, the effective call always consists of N arguments, the first N-1 of which are the values for the positional parameters of the function, and the Nth argument is an array that contains all provided variadic arguments.

Why an array-variadic function doesn't have the problems 1 and 2 above?

1. An empty sequence can be specified in the last argument, for example: [1, (), 2]. For example: myFun($posArg1, $posArg2, ..., $posArgN_minus_1, [1, (), 2])
2. When no variadic arguments are supplied, the last argument of the effective function call is still supplied -- it is the empty array. For example: myFun($posArg1, $posArg2, ..., $posArgN_minus_1, [])

The references to ArgumentPlaceHolder should have a lower case h to match the EBNF symbol name ArgumentPlaceholder in the grammar.

Kind regards, Reece

In the XPath and XQuery drafts the element and attribute tests have been extended to support any NameTest, meaning they now support all wildcard forms, which is great.

It is possible to write a path expression that takes a union of different paths -- such as html//(ol|ul) -- however it is not possible to define a precise type that accepts that path expression in variables, parameters, or return types, so if a user does specify a type some type information is lost during the static analysis phase.

As such, I propose renaming CatchErrorList to NameTestUnion and making ElementTest and AttributeTest accept a NameTestUnion:

[86] CatchClause ::= "catch" NameTestUnion EnclosedExpr
[87] NameTestUnion ::= NameTest ("|" NameTest)*
[212] AttributeTest ::= "attribute" "(" (NameTestUnion ("," TypeName)?)? ")"
[215] ElementTest ::= "element" "(" (NameTestUnion ("," TypeName "?"?)?)? ")"

Kind regards, Reece

Currently, it is possible to have multiple SimpleLetBindings in a SimpleLetClause, but not have multiple SimpleLetClauses in a LetExpr. The same applies for ForExpr.

I propose that this should be possible. In other words, make the following changes to the XPath 4.0 grammar:

[12] ForExpr ::= SimpleForClause SimpleForClause* "return" ExprSingle
[15] LetExpr ::= SimpleLetClause SimpleLetClause* "return" ExprSingle

This way, a user can write the following in XPath 4.0:

let $x := 1.0
let $y := 2.0
return $x + $y

in addition to the following in XPath 3.1 and earlier:

let $x := 1.0, $y := 2.0
return $x + $y

Note that this does not add any new capability to XPath (in particular, for and let clauses cannot be mixed like in XQuery) as a user can define multiple let or for clauses, it just provides them with an alternative way of expressing that like they can do in XQuery (where it is common to use a series of let clauses instead of let bindings for multiple local variables).

Kind regards, Reece

In section A.3 (Reserved Function Names) of the XPath 4.0 and XQuery 4.0 draft specifications, the tuple and union keywords have been added.

I don't think these (or more accurately, the keywords for the new item type constructs) are needed, as they cannot currently be used in places where functions can. Only KindTests can appear as path expressions, and the only other reserved names should be for other types of expression (if, switch, typeswitch, etc.).

From a similar reasoning, the following keywords should be able to be removed from the list as they are also only ItemTypes/SequenceTypes:

1. array
2. empty-sequence
3. map

Kind regards, Reece

The diff versions of the specs highlight what has changed in the wording of the specifications, which is useful, but don't highlight the differences in the EBNF grammar. This makes it harder for implements and tool vendors to work out what has changed in the grammar to then know what changes need to be made to the lexer and parser in order to support the new constructs.

By analogy with Schema annotations, i'd like to see an annotation-prefixes attribute on xsl:stylesheet/transform containing a space-separated list of NCName namespace prefixes that are associated with annotations; the XSLT processor would discard these elements (including children) during compilation. Here's a rough go at some text.

The purpose is to be able to include annotations at any level where elements are allowed - for example, inside an xl:variable or template or function body. Annotations might include XTest unit tests, Schmatron rules, human-readable documentation, CSS styles, or more, and could be used by other operations than the XSLT transformation: for example, by processing the XSLT source itself with XSLT.

It should be possible for the same element to be both an extension element and an annotation, but the behaviour is implementation-dependent in this case (for example, an API might allow an extension to access content or convert the annotation elements to something else on compilation).

XSLT instructions occurring inside annotation elements are ignored along with other content, except for xsl:fallback instructions (and their contents) if the prefix was also declared as an extension prefix and no matching extension was found. Similarly, extension attributes are discarded. The fallback behaviour might be used to support an XSLT-based implementation, for example by reading the XSLT source and processing embedded Schematron tests.

Attributes of annotation elements are not considered to be attribute value templates and content is not considered to be text value templates - that is, { and } are not special. However, an annotation element or attribute backed by an extension could perform such processing if an implementation supported it. The behaviour of shadow annotation attributes and xsl:use-when is implementation defined, but expressions contained in them must be processed as elsewhere.

Section §2.8.1 in the XDM specification states "The space of all possible function signatures forms a hierarchy of function types. "

This is clearly incorrect. It's a directed acyclic graph. (It's not even a lattice, because there is no minimum or maximum - there is no function signature that is a supertype of all others, nor is there one that is a subtype of all others). (I think there is one lattice for each possible arity.)

The readme needs an update

Link to the 4.0 html spec files rather than the 3.1 spec

More 🤦

🤦

I think this PR fixes the build errors. (CircleCI is having issues at the moment, and I know that master is currently broken, so I'm just going to merge this and hope!)

Spec identifiers are case sensitive.

• Delete rafts of build artifacts (all the html and build directoies)
• Remove a few more specs we're unlikely to edit
• Make sure the builds work even if the html and build directories are initially missing

filter has signature:

fn:filter($seq as item()*, $f as function(item()) as xs:boolean) as item()*

It could easily be changed to:

fn:filter($seq as item()*, $f as function(item()) as xs:boolean?) as item()*

with () meaning false().

Then it could be used with a map to pick elements from the seq. E.g.

filter(1 to 10, map {3: true(), 4: true()}) 

~> (3, 4)

filter(("a", "b", "b", "c", "d"), map {"b": true(), "d": true()}) 

~> ("b", "b", "d")

Same for array:filter

This PR fixes the publication date, cleans up a few other details, and makes the generated index more robust. It also publishes all of the new specs.

Just more infrastructure work.

1. Remove old specs
2. Refactor a few ant tasks so that they will work in gradle
3. Fix a few markup errors
4. Add a top-level gradle build that publishes the specs

More work will be needed to automate publication, but this is a start.

When $y is positive and $x is negative, the notes define this to be equal to π - atan($y div $x) when it should be atan($y div $x) + π.

When $y is negative and $x is positive, the notes don't define the value. It is the same as the first case, when $y is positive and $x is positive, so the first case should be "If $y is positive or negative, and ...".

When $y is negative and $x is negative, the notes don't define the value. The value is equivalent to atan($y div $x) - π.

The ThinArrowTarget and FatArrowTarget EBNF symbols do not follow the convention in the XPath and XQuery specifications of describing the semantics of the symbols, but instead describe how they are written.

I propose the following names:

1. CurryingArrowTarget instead of FatArrowTarget -- This is because the LHS is being curried into the function, making the function call it applies to take one less parameter. Other possible names would be SimpleArrowTarget or UnaryArrowTarget.
2. MappingArrowTarget instead of ThinArrowTarget -- This is because both variants make use of the simple map operator.

A minor request:

I find I often want to re-use the same behaviours across modes, or use related modes across similar content.

In the xsl:template and xsl:apply-templates, we can use a space separated list of mode names: it would be convenient if I could do the same thing in xsl:mode instructions.

A simple proposal, for a change:

A new function would be handsome to insert separators in a sequence. Equivalent functions exist in other languages; in Haskell and Dart, they are called intersperse:

Summary

Inserts the defined separator between the items of a sequence and returns the resulting sequence.

Signature

fn:intersperse($values as item()*, $separator as item()*) as item()*

Example

let $div :=
  <div>
    <span>one</span>
    <span>two</span>
    <span>three</span>
  </div>
return fn:intersperse($div/span, ',')

It would be good to have functions to convert from Q{uri}local format to xs:QName (perhaps with the option of supplying a prefix), and the reverse, generating Q{uri}local from an xs:QName.

I don't think we can extend the xs:QName constructor function to do this, because that has to remain consistent with XSD and (sadly) we can't change the lexical space in XSD.

So I would suggest a single-argument form of fn:QName that accepts either local or Q{}local or Q{uri}local. But this doesn't allow a prefix to be added. An alternative would be fn:EQName#1 accepting local or Q{}local or Q{uri}local, with an optional second argument to supply a prefix.

For the reverse, perhaps fn:EQName-from-QName#1? But the terminology here is a bit loose: in the grammar, EQName covers various formats of which URIQualifiedName is one. But fn:URIQualifiedName-from-QName is a bit of a mouthful.