See #2408

Fix #2527 Fix #2528

Looking at the issues raised by this action I decided to make a couple of changes that are not purely editorial.

• Confirming that an xsl:param in the used package cannot be overridden by an xsl:param or xsl:variable in the using package, because it is private (the 3.0 spec contained conflicting statements here).
• Allowing a non-static xsl:param in the used package to be initialised using a non-static expression in the using package (thus allowing parameter values to be passed through transitively).

Fulfilling the action. Editorial only.

Fulfilling action QT4CG-155-02

The current XQFO draft makes normative reference to RFC 7159 for the definition of the JSON grammar. This spec has been obsoleted by RFC 8259. The CG may want to consider updating this reference.

The new version substantively differs in mandating exclusively mandating UTF-8 for interchange: passages assuming the acceptability of UTF-16 and UTF-32 would need to be changed. The definition of fn:json-doc already goes beyond RFC 7159 by allowing the parser to process any additional charset it wants, so I don't think this really affects anything.

The definition of fn:parse-json says:

The input MUST consist of an optional byte order mark (which is ignored) followed by a string [...]

The wording here is unclear; is the BOM required or optional? Previous CG discussion (https://github.com/qt4cg/qtspecs/issues/1704) indicates the intent to be latter, in which case the definition should be rewritten.

I propose enhancing the item type syntax to allow facets (as in XSD) to be defined for atomic types.

For example

declare item type cal:month as xs:integer with (minInclusive=1, maxInclusive=12);

declare variable $quote-character as xs:string with (minLength=1, maxLength=1) external;

The semantics are exactly as if a restricted type with the same facets were imported from a schema.

At least initially:

• facets are allowed only on atomic types and not on xs:untypedAtomic
• the value of the facet must be a Constant
• we only allow facets that restrict the value space (therefore not whitespace)
• we allow the pattern facet only on xs:string and xs:anyURI so it becomes a restriction on the value space
• we don't allow the enumeration facet (though we could consider using it to replace our current enum type)
• we only allow one value for each facet (so no multiple patterns)
• we drop some of the dubious combinations such as defining length facets on xs:QName.

In the subsumption rules, the atomic type A with (FACETS) is a subtype of A.

The first feedback on JNodes shows that people find the analogy between XML and maps/arrays really attractive. At the same time, it’s perhaps the most challenging part, as the subtle differences of the two data types need to be understood.

For example, our constructors behave differently: Whereas the node constructor includes the full node in the result…

let $xnode := document { <n>123</n> }
return document { $xnode/n },

(: result :)
document { <n>123</n> }

…the map constructor includes only the ·jvalue· and ignores the ·jkey· and, such that the following query fails:

let $jnode := { 'n': 123 }
return { $jnode/n }

Maybe we can tweak map constructors by generating single-entry maps from non-root JNodes?

let $jnode := { 'n': 123 }
return { $jnode/n }

This would enable us to do many other things, such as filtering:

let $jnode := {
  'lu': 'lunes',
  'ma': 'martes',
  'mi': 'miércoles',
  'ju': 'jueves'
}
return { $jnode/(lu, mi) }

(: result :)
{ 'lu': 'lunes', 'mi': 'miércoles' }

In addition, if required, we could introduce an explicit function to convert a JNode to a single-entry map (basically, a shortcut for map:entry(jkey(), jvalue()).

4.3 Lookup Expressions says:

If $V is a JNode then it is coerced to the required type (map(*)|array(*)).

We should change the type to (map(*)|array(*))*. Otherwise, the following expression raises an error:

let $jnode := { 'a': ([ 0 ], [ 0 ]) }/a
return $jnode?1

Origin: https://github.com/qt4cg/qtspecs/pull/2506#discussion_r2900780344

We're currently using @v4 which (apparently) uses a deprecated version of Node.js. Before June, we should upgrade to @v6, but I think I won't do that while I'm on vacation.

This PR started as a fairly local change to resolve issue #2520 (year zero in build-dateTime and parts-of-dateTime) but it gradually became more ambitious.

The data model now states that the definitions of atomic types are taken from XSD 1.1. This means, for example, that xs:date includes year zero, and xs:double includes negative zero. It is still permitted to use an XSD 1.0 processor for schema processing and validation, but other operations (including casting from string) must follow the XSD 1.1 rules.

In the course of this some rules (including the rules on limits) have migrated from the XQuery and XSLT specs into the XDM specification.

Fix #2520

Fragment identifiers are now allowed in fn:doc calls, but they are still illegal in other functions (like fn:unparsed-text), for no apparent reason. We should generally allow them.

While reviewing qt4cg/qt4tests#340, I noticed again that the error codes for fn:unparsed-text are a bit of a mess, and difficult to adhere to when using low-level I/O:

• A dynamic error is raised [err:FOUT1170] if the $source argument contains a fragment identifier, or if it cannot be resolved to an absolute URI (for example, because the base-URI property in the static context is absent), or if it cannot be used to retrieve the string representation of a resource.
• A dynamic error is raised [err:FOUT1190] if the value of the encoding option is not a valid encoding name, if the processor does not support the specified encoding, if the string representation of the retrieved resource contains octets that cannot be decoded into Unicode characters using the specified encoding, or if any resulting character is not a permitted character.
• A dynamic error is raised [err:FOUT1200] if the encoding option is absent and the processor cannot infer the encoding using external information and the actual encoding is not UTF-8.

Things don’t get better with the introduction of the fallback option. I would prefer to see them merged to single error code, preferably FOUT1170, and to simplify the description to…

A dynamic error is raised [err:FOUT1170] if the the string representation of a resource cannot be retrieved.

…or to at least drop FOUT1200 and merge it with FOUT1190.

The spec needs to mention the year zero problem.

I think we should make build-dateTime and parts-of-dateTime consistent with string-to-date and date-to-string conversion, which gives the implementation three options:

(a) reject all dates earlier than +1.

(b) follow XSD 1.0: year 0 is not allowed, year -1 is the year before +1

(c) follow XSD 1.1: year 0 is allowed, year -1 is the year before 0, which is the year before +1.

Fix #2000

Building on issue #1979, I suggest that we define a limited set of cases where processors are required to report a static type error.

Certain constructs have a readily determined static type, including

• variable references if the type is declared
• static function calls
• literals
• most binary operators

The XSLT streaming spec gives a simple set of rules at https://qt4cg.org/specifications/xslt-streaming-40/Overview.html#static-type-of-xpath-expressions - we could adapt these rules easily enough. Those rules were written before choice types were available and could now be simplified.

Similarly, any operand context that applies function coercion has a required type which we already define.

We could define that it is a static error if the static type of an expression has no overlap with the required type, and cannot be coerced to the required type.

As suggested in #1979, we could then define that $x?y is a static error if the static type of $x is a record type and y is a field that is not defined in the record type.

We could also absorb some of the current definition of "implausibilities", for example by making @x/@y a static type error.

There's a backwards compatibility implication: we'll be detecting errors in code that weren't reported statically before, but would only be reported if actually executed.

This isn't a proposal to reintroduce "strict" or "pessimistic" static typing as defined for XQuery 1.0. It only reports errors for constructs that are provably wrong, whereas 1.0 static typing required the code to be provably correct. Reporting "provably wrong" constructs statically has always been permitted (and has always been implemented in Saxon); this merely makes it mandatory.

Stylesheet change to add the 'fn:' prefix to names of built-in-record types in function signatures

Fix #2316

Fix #2433

Fix #2475

1. Rename a number of productions, for example ElementTest becomes ElementNodeType
2. Drop the syntax jnode(X, *), it is now written jnode(X)

I didn't implement the suggestion of extending jnode(X) to allow alternative key values.

Fix #2403

Note this is XPath 4.0 only.

Closes #2497

Just make the fallback option a boolean.

I propose this as an alternative to #2467

Use of the magic date 01 January 2000 triggers the stylesheet xmlspec-2016 to insert the current date in its place.

Fix #2353

Fix #2498

Fix #2500

Fix #2503

In view of creating some points for discussion on generators, I propose a minimalist alternative to Dimitre's current generator proposal PR#2247.

This follows on from https://github.com/qt4cg/qtspecs/issues/2263#issuecomment-3961934656 (by @michaelhkay)

We also need to say more about the constructor functions associated with built-in record types.

For example, consider fn:dateTime-record. Does the constructor function for this record type impose any constraints beyond those in the type signature? Does it ensure that month is in the range 1-12? Does it ensure that if month is 4, then day is in the range 1 to 30? Does it ensure that if hours and seconds are present, then minutes is also present?

And editorially, where do we document this?

At the moment F&O § 9.3 Record fn:dateTime-record reads as though non-optional range constraints are being specified for most of the fields.

If they're not optional, when are implementations expected to confirm that values conform to their respective field constraints? When accessed via fn:build-dateTime the spec says that:

A dynamic error is raised [err:FODT0006] if any of the components in $value is outside the permitted range (for example if the value of the minutes component is less than zero or greater than 59), or if the combination of fields is not a valid date/time (for example, if month is 4 and day is 31).

Is an implementation also expected to check the field-specific ranges when a value is retrieved using, for example, ?day or map:get($my-dateTime-record, 'day')?

Similarly, is an implementation expected to validate inputs when the record is initially constructed (via fn:parts-of-dateTime or the implicit constructor)? Or, also, each time they're modified, for example, using map:put?

For further context, when a user defines a custom record type in XSLT (xsl:record-type), it doesn't appear possible to specify a range constraint.

Continuing on from https://github.com/qt4cg/qtspecs/issues/2406

In F&0 § 23.2 it says that:

When casting from a string to a duration or time or dateTime, it is now specified that when there are more digits in the fractional seconds than the implementation is able to retain, excess digits are truncated. Rounding upwards (which could affect the number of minutes or hours in the value) is not permitted.

For consistency, shouldn't that rule apply generically, when reducing the precision of a dateTime or duration? To truncate rather than round.

Fix #2499

Some test cases assume that the value of a JNode base expression of a dynamic function call is unwrapped. We should clarify in the spec whether we want to do that.

See https://github.com/qt4cg/qt4tests/issues/336

We appear to have inadvertently changed the default priority of patterns such as .[. instance of array(*)] from +1 to +0.5.

The change appears to have been conscious because test match-131 has been updated to reflect the change. However, no backwards incompatibility is recorded in the spec, and I can't see the justification for introducing one. I suspect that when we (probably me!) made the change, we were under the impression that it did not affect copnstructs permitted in XSLT 3.0.

Slightly more than editorial. The rule…

If J is a JNode, the result is in the form jnode(T), where T is the result of applying the type-of function to the ·content· property of J.

…should be adapted to reflect the new JNode type syntax jnode(K, T).

Two JNode diagrams need to be updated after my changes in #2490: selector must be renamed to jkey, content/position/parent to jvalue/jposition/jparent (or we skip the leading j’s).

@ndw I can update the svg and n3 manually, unless you have an automated way to do this…

One of my favorite topics (sorry for being persistent): I would like to update EBV slightly to reflect the semantics and usage patterns of current versions of the language.

Currently, an EBV can be computed if $data is an XML or JNode, so you can do…

• $node[.]
• boolean($map-or-array/.)
• if($xnode1 and $xnode2) then ...
• if($map | $array) then ...

…but you cannot do…

• $map-or-array[.]
• boolean($map-or-array)
• if($map1 and $map2) then ...

I think it would be much more intuitive to directly allow boolean($map-or-array) and to return true if the map or array exists.

Similarly, I cannot see boolean($qname) causing any trouble.

Fix #2494

The whitespace issue turned out to be in the CSS not related to fos.scm directly. I fixed it, but it's not clear what changed. Keep your eyes open for code elements that look like they have extra or missing whitespace!

With this PR, we get:

With a recent change, probably in fos.scm, additional whitespace has slipped into the function signatures:

Old:

New:

In general, signatures have become much longer than in previous days, but I cannot think of an obvious solution to render them more elegantly:

Closes #2052

Fix #2484

Closes #2443.

I have also renamed ·parent· and ·position· (maybe we get rid of the position? related: #2073), and the PR includes minor editorial cleanups.

@michaelhkay I guess you thought about this one:

I could be clearer if we renamed fn:parts-of-dateTime to fn:parse-dateTime. From a technical point of view, the difference is certainly that dateTime items have already been parsed. From a user perspective, I assume it is nontrivial to understand the difference between analyzing/parsing xs:date('2001-01-01) and xs:anyURI('http://url.com') items.

As an alternative, we could also rename fn:parse-uri to fn:parts-of-uri. In addition, we should either rename dateTime-record to dateTime-structure-record or uri-structure-record to uri-record.

A change to the test-generation stylesheet to correctly handle narrative results where there is an explicit test assertion.

Corrections to many of the narrative examples to make them pass the tests.

Stylesheet changes to give consistent rendition of term definitions across the family of specifications

The dashboard contains one entry for PR, each linking to each one of the specifications, whether the spec is affected or not.

The XSLT Streaming specification is not included in this list.

The adaptive serialization of anonymous functions becomes implementation-dependent.

(Some implementation might be able to do better than the current spec suggests, e.g. by displaying the source code of the function).

Fix #2388

In the subtyping rules, the type node() should be equivalent to a choice type of the seven specific node kinds.

More generally, any abstract type should be equivalent to a choice of its subtypes. For example, item() should be equivalent to gnode() | anyAtomicType | function(*), and gnode() should be equivalent to jnode() | node().

I came to the conclusion that the existing spec was technically sound, but that some editorial clarification was appropriate.

Fix #1479

Discussion in #2467 suggests that there are circumstances where you'd need to do a binary load and then decode the string with fallback.

If we think this is worth doing, do we just add it to the argument list, or do we break down and switch to bin:decode-string($value,$options)?

Fix #1528

Define a restricted list of keywords that are reserved in computed node constructors

Fix #2465

Fix #2472

Fix #2476

Removes the requirement for a conformant streaming XSLT processor to report an error when streaming is possible according to the processor's own rules, but not according to the rules in the specification.

Rebuilding the XSLT Streaming specification does not change the specification date in the header of the document.

The XSLT Streaming specification defines conditions under which a stylesheet is deemed to be "guaranteed streamable": see §3.1 Streaming Guarantees, and requires a streaming processor under these conditions to process the source document in streaming mode.

The specification also states that if the stylesheet is not "guaranteed streamable", but streaming is requested, then the processor must report a static error.

The latter requirement is very burdensome because it requires the streamability analysis to be done without any tree rewrites of the kind that would turn a non-streamable construct into a streamable one. This means, for example, that type inferencing must be done before any rewriting. Saxon attempts to meet the requirement, but there are almost certainly areas where it fails to do so. Saxon is also non-conformant in that the option to check for guaranteed (i.e. interoperable) streamability is off by default. The only benefit of the requirement for users is that they know that if one conformant streaming processor can handle the stylesheet, then every conformant streaming processor will be able to do so. But even if multiple conformant streaming processors were available, this benefit would be minor in proportion to the burden on implementors.

I propose dropping this requirement from the spec. It will still be true that a conformant streaming processor must always use streaming if the stylesheet adheres to the rules in the spec; it will no longer be the case that a conformant streaming processor is required to raise an error if it doesn't.

It’s always rewarding when syntax needs only to be learnt once, and I think the current JNodeType and NodeKindTest rules could be further aligned:

• With the JNode type syntax, a wildcard can be used for the sequence type: jnode(a, *). We should either allow this for element (attribute, …) tests – for example, element(name, *) – or we should discard jnode(name, *) – jnode(name) is equivalent.
• Similar as for element tests, we could allow unions in JNode types: jnode(a|b|'c d'):

JNodeType      ::=  'jnode' '(' ( JKeyTestUnion ( ',' JValueType )? )? ')'
JKeyTestUnion  ::=  JKeyTest ( '|' JKeyTest )*
JKeyTest       ::=  JRootSelector | NCName | Constant
JValueType     ::=  '*' | SequenceType

Proposed renamings in the grammar:

• NodeKindTest → XNodeType
• DocumentTest → DocumentType, etc.
• AnyItemTest → AnyItemType (analogous to AnyMapType etc.)
• AnyNodeKindTest → AnyNodeType
• NamespaceNodeTest → NamespaceType (in analogy with the rules for documents, which does not contain Node)
• Basically, all RegularItemType and XNodeType RHS rules should probably end with Type.

If we prefer the Test naming scheme, we should rename JNodeType to JNodeTest, etc.

Fix #272

Would anyone object if we moved EXPath/file to expath-file and EXPath/binary to expath-binary?

Having the specifications at two different levels in the hierarchy complicates any process that wants to process them as a whole.

The phrase "potential visibility" appears twice in the XSLT 3.0 specification. It's not at all clear what it is supposed to mean.

Refactored things a bit so that the code for records is shared between XSLT and F&O.

Fix #1310

This proposal meets the requirement expressed in the cited issue, but goes some way beyond it.

First, it proposes a new function fn:regex which effectively compiles a regular expression and returns a number of function items using that regular expression, so you can do for example

let $regex := regex("[A-Z][0-9]")
return //e[$regex?matches(@att)]

to give imlementations a better chance of optimising repeated use of the same regex.

Secondly, it proposes a new sub-function regex()?matching-segments() which returns similar information to analyze-string(), but as a map rather than as XML.

The proposal needs a bit more editorial work (more notes and examples) but I'm interested at this stage to get feedback on whether the general direction is sound.

Fix #2466

Drops the appellation "XML Path Language" which no-one ever uses.

Fix #2448

Close #2460

This PR harmonizes the functions fn:unparsed-text, fn:unparsed-text-lines, file:read-text, and file:read-text-lines with respect to handling non-permitted characters. Each function has an options parameter and that parameter may contain a fallback function to remap non-permitted characters.

This leaves unresolved the question of what to do about permitted characters not allowed in XML, but that’s orthogonal. Strings containing such characters might arise from any of these functions, but equally, might arise from other operations.

The specification of format-number() says:

If there are several such values that are numerically equal to the mantissa (bearing in mind that if the mantissa is an xs:double or xs:float, the comparison will be done by converting the decimal value back to an xs:double or xs:float), the one that is chosen should be one with the smallest possible number of digits not counting leading or trailing zeroes (whether significant or insignificant).

The parenthetical "bearing in mind" note needs updating, because comparison of a decimal to a double is no longer done by converting the decimal back to a double.

Background: XSLT test case format-number-044a, which I have extricated from the composite test format-number-044, formats the x:double obtained as 1E100 div 3. This is giving me a result with sixteen 3s on Java, fifteen 3s on C#. I am trying to work out which is correct, or at any rate which one should be produced according to the above rules.

The error description for FODC0006 can be raised by fn:parse-xml and fn:parse-xml-fragment. It is currently described as

err:FODC0006, String passed to fn:parse-xml is not a well-formed XML document.
Raised by [fn:parse-xml](https://qt4cg.org/specifications/xpath-functions-40/Overview.html#func-parse-xml)
if the supplied string is not a well-formed and namespace-well-formed XML document;
or if DTD validation is requested and the document is not valid against its DTD.

I propose to alter the description to

err:FODC0006, String cannot be parsed as XML.
Raised by [fn:parse-xml](https://qt4cg.org/specifications/xpath-functions-40/Overview.html#func-parse-xml) or [fn:parse-xml-fragment](https://qt4cg.org/specifications/xpath-functions-40/Overview.html#func-parse-xml-fragment)
if the supplied string is not a well-formed and namespace-well-formed XML document;
or if DTD validation is requested and the document is not valid against its DTD;
or if it was passed to parse-xml-fragment and is not a well-formed external general parsed entity, if it contains entity references other than references to predefined entities, or if a document that incorporates this well-formed parsed entity would not be namespace-well-formed.

The description of fn:path() says Returns a path expression that can be used to select the supplied node relative to the root of its containing document, but gives no guidance as to how exactly to use such a path expression to select a node; one can use eval(), or could write an invisible XML grammar, but that'd like saying “if and do-while can be used to write a program to solve chess problems”.

Easiest fix - remove the text saying people can use the result of path() to select nodes, since they can’t.

Slightly harder - add a path-to-node($root, $path) as node()? function.

I’d prefer the function—unless there is one and i missed it, and then i’d prefer the obvious editorial change :-) —but reducing the reader’s expectations would be OK too.

With keyword arguments being available, it would make sense to have a map:apply($function, $map), in which keys in the map are mapped to argument names of the function.

Admittedly since fn:apply() takes an array and is not called array:apply(), maybe an optional 3rd argument to fn:apply() would work better, with the same rules as for static function application.

I would like to question the decision to allow function calls on sequences.

While I favored the change in the past, repeated user feedback indicates that the new behavior is confusing and complicates debugging. First, it legalizes cryptic code like ()(). Second, seemingly simple function calls like $add(1, 2) return an empty sequence if $add turns out to be empty.

I believe we should make type safety and readability a priority over convenience. If a short syntax is needed, one can still write $add ! .(1, 2).

Related: #2219 (can be closed if we revert dynamic function calls).

Support for unparsed entities and notations is something of a minority interest, but recent correspondence with a Saxon user reminds me that there are applications that depend on them quite heavily. There is limited support for them in the data model and in XSLT, but none in XPath or XQuery.

There is no logical reason for the two functions unparsed-entity-uri() and unparsed-entity-public-id() to be XSLT-only, other than to save XQuery implementors the trouble of implementing them.

At the same time, the functions are incomplete and inadequate. For example, there is no way of obtaining a complete list of declared entities, and there is no way of getting information about the notations that they refer to.

Many XML parsers do not expose information about entities and notations, so anything we define should be capable of returning a result that indicates "information not available for this document (or for this implementation)".

I propose a new function unparsed-entities($doc) which returns a map along the lines:

{ "entities": {
     "e-name-1": {
         "system-id": ....
         "public-id": ....
         "notation": ....
     }, ....
  },
 "notations":{
    "n-name-1": "system-id", ...
  }
} 

and which is allowed to return an empty sequence if information about unparsed entities is not available (making it possible to provide a trivial fallback implementation).

file:read-text() should be aligned with fn:unparsed-text() to remove the restriction regarding valid XML characters.

Incidentally, bin:decode-string() has never had such a restriction even in the EXPath 1.0 version.

Are we applying XML 1.0 rules or XML 1.1 rules? Does the user get to decide? How?

I can currently write $map/jnode("key") or $map/get("key"), and both have the same meaning.

Both imply the child axis: I can also write $map/child::jnode("key") or $map/child::get("key")

There are differences:

• jnode also allows a content type to be specified
• get allows the key to be an arbitrary expression, while jnode requires a Constant
• jnode allows an NCName-valued key to be written without quotes
• get allows multiple keys to be selected
• jnode allows all keys to be selected
• get also works with XNodes

I propose that we unify these constructs.

To do this we distinguish jnode as an item type from jnode as a selector.

When used as a selector, the first argument of jnode should accept a KeySpecifier rather than a Constant. KeySpecifier is the subset of Expression that we allow after the lookup operator ?. (We should extend it to generalise Literal to Constant) A KeySpecifier allows an arbitrary expression in parentheses; we can debate what focus should be used to evaluate it.

This means, for example, that $array/get($i) becomes $array/jnode($i)

We could apply the same treatment to element() and attribute(), allowing $doc//element($N).

The second optional argument of jnode(), element(), and attribute() is unaffected.

Changes the main function catalog to make improved use of the fos:result element. Specifically:

• Uses fos:error-result for error examples
• Makes greater use of explicit results rather than narrative results where possible
• Uses fos:result narrative="true" to get auto-checking of examples having no definitive result, and improved rendition.

Following a schema change that allows the function catalog to contain code examples annotated <fos:result narrative="true">, this PR makes stylesheet changes allowing such examples to be rendered.

• In generating the specification documents, narrative results are simply rendered as prose, without containing <code> tags
• In generating the QT4 tests, the generated test case ensures that the example can be successfully compiled and run, but the test always succeeds so long as the result is not a static or dynamic error.

The current rules of the file:copy function say:

Copies a file or a directory given a source and a target path/URI. The following rules apply if $source points to a file:

1. if $target does not exist, it will be created.

I think we should refine this rule and prevent this function from creating arbitrary directory structures if simple files are copied.

I think we could easily tweak the grammar by changing…

Constant     ::=  StringLiteral | ("-"? NumericLiteral) | QNameLiteral | ("true" "(" ")") | ("false" "(" ")")
Literal      ::=  NumericLiteral | StringLiteral | QNameLiteral

…to:

Constant     ::=  Literal | ("true" "(" ")") | ("false" "(" ")")
Literal      ::=  ("-"? NumericLiteral) | StringLiteral | QNameLiteral

As a result, a negative number would be returned by the parser as a literal instead of a unary expression (weird constructs like - - -2344 will still be possible), and the key specifier of a lookup expression could be a negative number.

This issue identifies a bug in XSLT 3.0 (retained in XSLT 4.0).

There is a special rule in XSLT 3.0 designed to ensure that the pattern match="a" will match an a element even if it is parentless. Without the special rule, it would not do so, because "a" expands to child::a which would otherwise only match an element that is a child of something.

The rule is written (in §5.5.3):

If any PathExprP in the Pattern is a RelativePathExprP, then the first StepExprP PS of this RelativePathExprP is adjusted to allow it to match a parentless element... If PS uses the child axis (explicitly or implicitly), and if the NodeTest in PS is not document-node() (optionally with arguments), then the axis in step PS is replaced by child-or-top, which is defined as follows. If the context node is a parentless element, comment, processing-instruction, or text node then the child-or-top axis selects the context node; otherwise it selects the children of the context node. It is a forwards axis whose principal node kind is element.

Now consider the pattern match="*/(b union c)". Clearly b is a PathExprP and a RelativePathExprP, so the adjustment applies to its first StepExprP, namely to b. So the pattern should expand to match="*/(child-or-top::b union child-or-top::c)", and a literal reading of the semantics of patterns, in conjunction with the semantics of path expressions, means that this should match a parentless b or c element, whereas the clear intention (and the Saxon implementation) is that it matches a b or c element only if it has an element node parent.

Similarly, consider the pattern match="b[b or c]". Again, the rules suggest that this should be adjusted to match="child-or-top::b[child-or-top::b or child-or-top::c]" which would technically match a parentless b element having no b or c child. This is clearly not intended

Saxon in fact is not applying any syntactic adjustment to the pattern at all. Rather, it is evaluating the pattern steps from right to left, and if a sub pattern has no preceding "/" or "//" operator then it is assuming there are no constraints on the element's ancestry.

I can think of a couple of ways the bug might be fixed.

Firstly, we could try to define more precisely the StepExprP subexpressions that are subjected to this adjustment. It's basically any StepExprP that is not the right-hand operand of "/" or "//", and that is not an operand of a union, intersect, or except expression that is the right hand operand of "/" or "//", and is not contained within a predicate.

Alternatively, we could try to amend the "equivalent expression" rule. We say that a pattern P matches a node N if N has an ancestor-or-self node $A such that the path expression $A//(P) selects N. We could amend this rule to say that if N is parentless, then this rule is evaluated "as if" N had an imaginary parent.

Neither solution is very elegant.

Fix #2450

Fix #2444

The effect of the change is that simple patterns like match="*" and match="order" will only match element nodes, they will no longer match JNodes.

This is motivated firstly by implementation experience: the current rules cause a performance regression compared with XSLT 3.0 because it's harder to do precise static type inferencing, and some streaming use cases are no longer streamable for the same reason.

However, I think there is also a usability benefit. These simple match patterns are instinctively understood by the entire XSLT user population, and extending their meaning so they match things unexpectedly could be a debugging nightmare. Consider the case of a mature stylesheet with hundreds of template rules designed to process XML elements, which is then extended with a new module to handle a JSON representation of the same data; it's very unlikely the user will actually want or intend the same template rules to process both, and if this is what they want, it's clearer to make this explicit by using a union pattern.

In the description of JNode types in 3.2.9, there is no explicit statement of what a type like jnode(*, map()) means. In particular, it isn't clear (except perhaps from studying examples) whether it matches a JNode whose selector property is absent (i.e. the root of a JTree). The examples suggest that it does.

For pattern matching in XSLT, it would be useful to have a pattern that ONLY matches the root of a JTree. Perhaps the syntax jnode(/, map()) might serve this purpose.

Fix #2445

The title of the XPath specification is

XML Path Language (XPath) 4.0

No-one actually calls it "XML Path Language" -- and it's now a path language for JSON as well.

I propose we change the title to

XPath 4.0

XQuery 3.1 allowed element foo { "bar" }.

And then we discovered syntax ambiguities, so we allowed element "foo" {"bar"}

And then we introduced QName literals, allowing element #foo {"bar"}

We then dropped string literals from the grammar, but there are incorrect examples that use it:

XQuery 4.0 allows the node name to be written in quotation marks (for example, element "book" {}

in 4.12.3

Fix #2427

Following discussion in #1948, test element-to-map-017 has been changed so it is treated as if the xsi:type attribute were not present. However, I can't see anything in the current spec to justify this behaviour.

I propose changing the spec to say that all attributes in the xsi namespace should be ignored (except to the extent that when the input is schema-validated, they may have affected the choice of a type annotation, which itself may affect the outcome). This means (a) the attribute itself is not included in the result of the conversion, and (b) if all the attributes of an element are in this namespace, the element is treated as having no attributes.

A pattern such as match="order" currently matches both an element named "order" and a JNode whose selector is "order".

This makes it much more difficult to do type inferencing on the body of the template rule, and it greatly complicates streamability analysis. While usability rightly takes priority over implementation concerns, I don't think this design has usability benefits either. I think in practice users will know whether their template rules are intended to match XNodes or JNodes, and using the same pattern syntax for both is more confusing than helpful. We already offer the syntax `match="jnode(order)" to match JNodes, and I think that is clearer.

Note that the semantics of match="order" already depart from the semantics of the equivalent XPath expression child::order because the pattern will match a parentless element. So we only have to adapt the current magic rule:

If PS uses the child axis (explicitly or implicitly), and if the NodeTest in PS is not document-node() (optionally with arguments), then the axis in step PS is replaced by child-or-top, which is defined as follows. If the context node is a parentless element, comment, processing instruction, or text node then the child-or-top axis selects the context node; otherwise it selects the children of the context node. It is a forwards axis whose principal node kind is element.

so that the child-or-top axis only selects XNodes.

I am playing around with the new JNode functions, and my impression is that the resulting code does not look very catchy. It seems way too technical to me. For creating a map with element names and contents, we can write:

map:build($xnodes, name#1, data#1)

Similar code for JNodes would be:

map:build($jnodes, jnode-selector#1, jnode-content#1)

I wonder whether we really need to introduce so many completely new terms for rather straightforward concepts, instead of borrowing existing terminology. What about renaming the terms “Selector” to “JKey” and “content” to “JValue”? This would pretty much resemble the map terminology (even if we also use it for arrays), and it might help to understand that these terms are specific to JNodes.

map:build($jnodes, jkey#1, jvalue#1)

This PR is #2404 with the addition of the compiled fos.scm.

Close #2404 Close #2403 Close #2177

The CG agreed to merge this PR at meeting 151

Fix #2434 Fix #2437

1. The axis step child::gnode() should be allowed.
2. The axis step self::array(*) is used in an example but is invalid syntax and makes no sense
3. Functions with an argument of type gnode() that accept the context item as a default should allow the context item to be any gnode, not just an XNode.

Fix #2432

Fix #2435

This is MK's PR with the compiled form of the schema added.

The CG agreed to merge this PR at meeting 151.

Close #2404 Close #2403 Close #2177

With the current grammar…

AxisStep         ::=  (AbbreviatedStep | FullStep) Predicate*
AbbreviatedStep  ::=  ".." | ("@" NodeTest) | SimpleNodeTest
FullStep         ::=  Axis NodeTest
NodeTest         ::=  UnionNodeTest | SimpleNodeTest
SimpleNodeTest   ::=  TypeTest | Selector
TypeTest         ::=  NodeKindTest | JNodeType

…type tests in axis steps are limited to node() and its subtypes as well as jnode(). The test cases include tests for additional types like gnode() or array(*).

Maybe TypeTest should be replaced by the RegularItemType:

RegularItemType  ::=  AnyItemTest | NodeKindTest | GNodeType | JNodeType | MapType | ArrayType | RecordType | EnumerationType

The current spec defines the following grammar for the jnode type:

JNodeType  ::=  "jnode" "(" (("*" | NCName | Constant) ("," ("*" | SequenceType))?)? ")"
Constant   ::=   StringLiteral | ("-"? NumericLiteral) | QNameLiteral | ("true" "(" ")") | ("false" "(" ")")

As far as I can judge, none of the current test cases seems to use this syntax. Instead, the tests I found expect a single sequence type argument, for example fn-jtree-006:

<test-case name="fn-jtree-006">
  <description> JNode applied to an array - type of result</description>
  <created by="Michael Kay" on="2025-06-16"/>
  <test>fn:jtree([1,2,3]) instance of jnode(array(xs:integer))</test>
  <result>
    <assert-true/>
  </result>
</test-case>

Is it the test suite or the spec that needs to be updated?

If this is still subject to discussion, my preference would be to disallow constants in the jnode syntax:

• It is not clear to me how fn:jtree({ 'a': 1, 'b': 2 }) instance of jnode(a) can be interpreted.
• Instance checks for linear hierarchies are generelly more intuitive.
• With the presence of get(), the jnode constants should be redundant.
• We can only use atomic items in the node tests for which literals exists.

In EXPath Binary Module 4.0 some example functions have been redefined in a different namespace prefix (which was bin:) to avoid suggesting they were part of the supported library.

However in 2.2 Example – reading and writing variable length ASN.1 integers, the definitions of asn:int-octets() and asn:encode-ASN-integer() still contains references to the original prefix definition bin:int-octets() which should be asn:int-octets()

I believe that the recently added examples for fn:has-children need to be fixed (or it’s my brain that needs to be updated):

[1,2,3] => has-children()
[] => has-children()

The function signature expects gnode()? as input type, so I would expect both queries to return an error unless the arguments are not explicitly wrapped into a JNode.

The rules for fn:jtree say:

If two maps or arrays M1 and M2 have the same function identity, as determined by the function-identity function, then jtree(M1) is jtree(M2) MUST return true: that is, the same JNode must be delivered for both.

Note: It is to some extent implementation-defined whether two maps or arrays have the same function identity. Processors SHOULD ensure as a minimum that when a variable $m is bound to a map or array, calling jtree($m) more than once (with the same variable reference) will deliver the same JNode each time.

Shouldn’t SHOULD be MUST? The argument will always have the same function identity if jtree($m) is called more than once. Maybe the note can also be dropped, as fn:jtree is not about the identity of maps and arrays, but the identity of JNodes.

The specification says in [22.1 Constructor functions for XML Schema built-in atomic types](https://qt4cg.org/specifications/xpath-functions-40/Overview.html#constructor-functions-for-xsd-types)…

If the value passed to a constructor is not in the lexical space of the datatype to be constructed, and cannot be converted to a value in the value space of the datatype under the rules in this specification, then an dynamic error is raised [err:FORG0001].

…but it is not clear which rules in the specification are meant.

Specifically, I think we should clarify whether query like the following one are supposed to return an error or a duration:

xs:anyURI('P2000Y') => xs:yearMonthDuration()

On all pages: Improve display of Headlines with slightly smaller font-size.

On rule detail pages: The Name of the Grammar is now visible in the back button of the ribbon instead of the H1 which only displays the rule name.

On the right hand side:

• Fix complement character class display.
• Sequence and choice items now are indented if they do not fit on one line
• occurrence indicators always stick to the item they belong to
• use spans to group items and mark any character that is displayed
• dashes in character ranges and the pipes in choices are now also recognized as part of EBNF syntax
• literals are never wrapped into the next line

Updates the schema for XSLT 4.0:

• Adds canonical to xsl:output and xsl:result-document
• Adds xsl:package-location to content model of xsl:use-package.

I'm suggesting two minor changes following my reading of the most recent draft specs.

This doesn't actually abolish the feature, it just de-emphasises it. AFAIK, no-one actually uses it.

There was pushback on issue #573 which proposed a set of functions for constructing nodes, on the grounds that for XQuery users, this was unnecessary duplication.

A possible alternative is to add a subset of the XQuery syntax for node construction to XPath: specifically, computed node constructors, which are relatively free of hassles such as dependence on the namespace context, boundary space rules, etc.

Specifically we could add computed constructors:

ComputedConstructor::=CompDocConstructor
|  CompElemConstructor
|  CompAttrConstructor
|  CompNamespaceConstructor
|  CompTextConstructor
|  CompCommentConstructor
|  CompPIConstructorCompDocConstructor::="document" EnclosedExpr
 CompElemConstructor::="element"  CompNodeName EnclosedContentExpr
 CompAttrConstructor::="attribute"  CompNodeName EnclosedExpr
 CompNamespaceConstructor::="namespace"  CompNodeNCName EnclosedExpr
 CompTextConstructor::="text"  EnclosedExpr
 CompCommentConstructor::="comment"  EnclosedExpr
 CompPIConstructor::="processing-instruction"  CompNodeNCName EnclosedExpr

with the restriction that CompNodeName / CompNodeNCName are either expressions in curly braces, or use the new XQuery 4.0 form with a leading "#".

It is of course trivial to define a function library on top of this if someone wants the extra flexibility:

let $new-element := fn($name, $content) { element {$name} {$content} }

etc

(Incidentally, EnclosedContentExpr serves no useful purpose as it's identical to EnclosedExpr.)

We could common up the rules for "constructing simple content" and "constructing complex content" at the same time, putting them in XPath where both XQuery and XSLT can refer to them. I believe they are identical except for (a) error codes, (b) with duplicate attribute names, XQuery throws an error while XSLT takes the last.

Fixes nearly everything in #2408

Since we link to pull requests in the spec and in test cases, I wonder whether it would be possible to publish a permanent diff showing the effect of each PR?

Essentially, the idea would be to take the HTML diff as we currently publish it, and reduce it to those sections of the specs that actually contain changes.

I would find this very useful, for example, when a PR has been accepted but is still marked with "Tests needed" - it's not easy at present to see retrospectively what tests might be required. It would also be useful, of course, when implementing the PR. But I think that all readers of the specs might find this beneficial.

Building on the awesome work from @ndw I just tweaked the grammar explorer layout a little more

• layout works better on bigger and smaller screens
• consistent navigation between all screens with back button alwasy on the navigation at the top
• consistent sizes, paddings, colors set by CSS variables
• output as html5 which fixes small issues with whitespace in inline elements
• additional, minor layout improvmements

Before

After

Fix #2421

XSLT Section 3.11 describes embedded stylesheet modules - a stylesheet rooted at an element node which is not the outermost element of a document. There are no real conformance requirements associated with this feature and it isn't widely used. I proposed we drop the section, while retaining the statement in §3.5 that a stylesheet module can be "all or part" of an XML document.

Simplified stylesheets have changed so that the implicit template rule now does match="." rather than match="/".

This creates a theoretical incompatibility when

(a) the stylesheet is invoked supplying a node other than a document node as the input. It will now execute the (only) template rule, previously it would execute the built-in template for the node kind

(b) the simplified stylesheet module is included/imported into another stylesheet. This is a highly unlikely scenario, but it is tested by test case include-0601.

The incompatibility should be documented.

h/t @line-o

Plus a few other tweaks.

Fix #2292

Fix #2399

Fix #2399

Fix #2406

There seem to be two consistent errors in the diff markup that appears in PRs on the dashboard:

1. When an inline <code> element is modified, the diff version shows the old code as deleted (red background), but does not show the new code.

For example:

2. When a grammar entry is modified, the text gets duplicated.

For example:

This PR drops the concept of extensible record types, replacing it with a rule that coercion to a record type drops any map entries that are not defined by the record type. In effect this means that a record type used when declaring a function parameter is implicitly extensible.

The benefits of the proposal are:

• It simplifies the spec, especially rules on type subsumption and on generation of implicit constructor functions
• It avoids the need to declare pairs of record types, one extensible and one not.
• It avoids all the awkward decisions about whether record types used in core functions should be extensible or not.

The rules for type patterns in XSLT are changed to invoke coercion.

Fix #1484 Fix #2365

Fix #2395 Fix #2396

Fix #2397

Fix #2398

Fix #2407

This issue summarizes the unresolved comments from #2195:

• [x] Oxford/serial comma should be used consistently at numerous places. Candidates:
  • sine, cosine and tangent
  • durations, dates and times
  • year, month, day, hour, minute, second and timezone
  • The month, day, hour and minute components
  • the tokens w, W and Ww
  • rules for overflow, underflow and approximation
• [ ] The comma may need to be removed at other places:
  • This function is context-independent, and focus-independent.
  • A dynamic error is raised [err:FODC0002] if a relative URI reference is supplied, and the base-URI property in the static context is absent.
• [x] fn:distinct-values: $coll → $collation
• [x] “If … is an empty sequence” vs. “If … is the empty sequence” (which one do we prefer?)
  • MHK: I can't say I have a strong preference, but pedantically, the is probably more accurate.
• [x] Section 2.1.3 Values of the XPath 4.0 spec includes a change note highlighting that the terms XNode and JNode have been introduced but the section text only mentions XNode; there's no reference to JNode in that section other than in the change note itself.
• [ ] In the XPath 4.0 spec, the publoc URL appears to be incorrect (/spec/header/publoc/loc)
• [x] A number of examples in the function catalog should be annotated with spec="XQuery" so that the corresponding test cases are marked as inapplicable to XPath. Specifically:

  fo-test-fn-count-001
  fo-test-fn-deep-equal-005
  fo-test-fn-every-010
  fo-test-fn-function-annotations-002
  fo-test-fn-function-annotations-003
  fo-test-fn-hash-009
  fo-test-fn-hash-010
  fo-test-fn-serialize-004
  fo-test-fn-sort-with-005

• [x] The serialization spec, in the section on the serialization parameter document, makes it rather hard to discover that the namespace prefix output is bound to the URI http://www.w3.org/2010/xslt-xquery-serialization
• [x] F+O The definition of fn:compare refers to a function fn:months-from-dateTime, this should be fn:month-from-dateTime
• [x] F+O The description of unparsed-text refers to the available text resources component of the dynamic context which has been dropped.
• [x] The serialization spec for Adaptive serialization of function items gives an example **fn:exists#1** is serialized as **function fn:exists#1** but the word function does not actually appear in the result.
• [x] fn:parse-html needs to define an error code for use when $encoding is an unknown or invalid encoding.
• [x] Consistent rendition for term definitions. Most of the specs output definitions as [Definition: here is the definition]. F+O uses [Definition] here is the definition. XSLT puts the keyword "Definition" in small caps.
• [x] The function catalog contains entries for functions such as op:gMonthDay-equal that are no longer referenced.
• [x] Serialization, HTML5, Processing Instructions: Dashes in the name must be escaped as well. Example: <?a---b c---d?> should be serialized as <!--?a- - -b c- - -d?-->..
• [x] UTF-16le → UTF-16LE, UTF-16be → UTF-16BE (caused by #2239)
• [ ] fn:unparsed-text: Reference bin:infer-encoding, drop redundant rules
• [x] op:divide-dayTimeDuration-by-dayTimeDuration: An example could be simplified by using seconds(1)

As we use the fn alias in (almost) all XQFO signatures, we could also return it by the fn:type-of function.

The precision returned for dates, times, and durations is implementation-defined, and it has changed between Saxon releases. This leads one of our users to point out that there is no easy way to request a reduced precision (e.g. milliseconds) in order to ensure interoperability. The simplest approach we can offer seems to be current-dateTime() => format-dateTime("....") => xs:dateTime() which is pretty cumbersome and inefficient.

Rather than providing specific functions for rounding dates, times, and durations, the most versatile solution to this might be to provide functions that reduce a dateTime or duration to a record containing the numeric values of the components, allowing these to be manipulated as numbers, with a further function to reconstruct the dateTime or duration from the record: rather like the parse-uri()/build-uri() pair. Something like:

parts(current-dateTime()) ! map:put(., 'seconds', round(?seconds, 3)) ! build-dateTime()

It’s not the specification XML, it’s the pre-fixed-up HTML as XML. h/t to @martian-a for noticing first!

Schema enhancements to the function catalog for

Issue #2403 - allow non-testable results for examples to be labelled narrative="true" Issue #2177 - allow a fos:see-also element to make links to related functions

Note this PR is purely an enabler, it does not include changes to the function catalog to exploit this features, nor stylesheet enhancements to render them.

In PR #2401 Norm introduced a temporary fix needed because the function catalog in the EXPath file spec doesn't conform to the fos.xsd schema.

I think we can fix this without a schema or stylesheet change by using existing mechanisms illustrated by this example from fn:collation:

             <fos:test>
               <fos:expression>collation({ 'lang': 'de', 'strength': 'primary' })</fos:expression>
               <fos:result>"http://www.w3.org/2013/collation/UCA?lang=de;strength=primary"</fos:result>
               <fos:test-assertion>
                  <result xmlns="http://www.w3.org/2010/09/qt-fots-catalog">
                     <any-of>
                        <assert-string-value>http://www.w3.org/2013/collation/UCA?lang=de;strength=primary</assert-string-value>
                        <assert-string-value>http://www.w3.org/2013/collation/UCA?strength=primary;lang=de;</assert-string-value>
                     </any-of>
                  </result>
               </fos:test-assertion>
               <fos:postamble>The order of query parameters may vary.</fos:postamble>
            </fos:test>

The fos:result (or fos:error-result) element must always be present, and will always be rendered in the spec as the expected result. If the example will not always deliver this result, then <fos:test-assertion> can appear to give the result as it will appear in the generated test case.

But I suggest we add another attribute <fos:result narrative="true"/> to indicate that the result is given as explanatory prose, not as a testable XPath expression. This would allow another fn:collation example

         <fos:example>
            <p>The expression <code>collation({ 'lang': default-language() })</code>
               returns a collation suitable for the default language in the
               dynamic context.</p>
         </fos:example>

to be rewritten as

         <fos:example>
            <fos:test>
                <fos:expression>collation({ 'lang': default-language() })</fos:expression>
                <fos:result narrative="true">A collation suitable for the default language in the
               dynamic context.</fos:result>
                <fos:test-assertion>
                   <result xmlns="http://www.w3.org/2010/09/qt-fots-catalog"><assert>true()</assert></result>
                </fos:test-assertion>
           </fos:test>
         </fos:example>

which will (a) make it easier to fit the results into the tabular presentation of examples, and (b) cause a test case to be generated which will ensure that the example is syntactically valid.

(Or we could leave out <fos:test-assertion> in this example. If the supplied fos:result has narrative="true" and there is no test assertion, the generated test case can assume <assert>true()</assert>)

We're never going to merge this, it's just a CI test.

The content model of fos:test requires an fos:result or fos:error-result. For the EXPath File module, we have to work out what those should be or change the markup or change the schema.

In the short term, I’ve made bogus fos:results of FIXME:

RFC 8785 says:

Note: Since Not a Number (NaN) and Infinity are not permitted in JSON, occurrences of NaN or Infinity MUST cause a compliant JCS implementation to terminate with an appropriate error.

We have just decided to treat these cases more liberally, but I think we should continue to raise an error if canonical serialization is requested.

In our serialization spec, we also say:

Implementations may serialize an xs:double value using any lexical representation of a JSON number defined in [RFC 7159], but it is recommended to use the same representation as when the canonical parameter is true.

We may need to exclude the edge cases from this recommendation.

If we keep the recommendation, we may need to fix the test case Serialization-json-11, which expects -0 instead of 0 (what is returned for RFC8785).

The description for the new fn:highest function does not align with the description for the fn:lowest function; contrary to what I'd expect. In particular, the rules section makes no mention of the $key argument; so I suspect this is not up to date?

The new XSLT 4.0 functions current-merge-key-array and regex-groups are missing from the "Functions Defined in XSLT" section.

Also I believe the function unparsed-text-available should be added saying "Originally XSLT 2.0; then XPath 3.0 and later".

Please add changes entries to say "New in 4.0" for the functions: function-identity and jnode-content.

Also I assume the first change entry for function-annotations should actually say "New in 4.0" (rather than be a duplicate of the second change entry).

Please add a changes entry in the spec for the new XSLT 4.0 regex-groups function.

Standardizes on upper case for encoding names, and mentions that comparisons are case-blind.

Fix #2361

Adopted from #2351:

We have recently dropped map:pairs and map:of-pairs. With array:members, the members of arrays are returned as single-entry maps, which may confuse users. Thus, for the sake of reducing redundant functionality, do we want to keep array:members and array:of-members, or rather promote the use of for member $m and array:split/array:join instead?

If we keep the functions, we should add a dedicated record type for record(value).

Fix #2384

Fix #2378

Don’t panic, i am not suggesting a large change :)

But i would like to suggest a small change to the semantics of method calls, with the goal of third parties being able to build something much larger.

Today, we have,

A method call combines accessing a map M to look up an entry whose value is a function item F, and calling the function item F supplying the map M as the implicit value of the first argument.

I’d like to add,

If there is no such function, but there is in the map a key fn:fallback whose value is a function, then that function is called with the map, the function name, and the arity of the desired function as arguments.

In this way one could write a function that looked for an "isa" entry in the map whose value was a sequence of "class" maps, and find the function.

It’s limited in that there is no possibility of polymorphic functions, but we do not have those elsewhere in the language.

One practical benefit is that you can have a map with all your functions in it, and “instance maps’ then do not need to have, say, 40 entries for all the methods that can be called. In an application in which a map gets updated a million times (I do have one of those), adding 40 extra entries to copy is a significant burden, even though of course it’s encapsulated in a single function.

The adaptive serialization method was introduced “for the purposes of debugging query results”. For our processor, it has turned out pretty soon that it does not satisfy the requirements of our users, which is why we have introduced a custom debugging method.

I wonder what others think: Shouldn’t we relax several of the rules and let the implementation decide what to output? We haven’t defined either how the output of fn:trace needs to look like.

Some examples:

• The output of doubles often causes confusion. If parsed JSON is output, small integers will be output in exponential notation. For example, parse-json('{ "A" : 20 }') needs to be output as { "A": 2.0e1 }.
• xs:date("2001-01-01") is output as xs:date("2001-01-01"), while xs:token('x') is output as "x".
• fn() { 1 } is output as (anonymous-function)#0, whereas an implementation could prefer to use the output of fn:function-identity (see #2388), or output the original query string (if available), reproduce a string representation of the function body, etc.

I will be glad to create a PR.

The serialization specs defines rules for creating a string representation for function items. Now that we have `fn:function-identity', we should replace the rules and use this string instead.

Addresses part of issue #641

In the JSON serialization method, NaN is output as null, and infinity is output as ±1e9999.

The parse-json function adds recommendations on how to achieve round-tripping of these values.

Changes the stylesheet for generating keyword and function signature tests so that the namespace prefix "output" is explicitly declared in the test environment. This prefix is used in one of the tests and it needs to be declared if the test is to work in XPath.

Probably XQuery too. I have no idea why.

The type signature of the xsd-validator function suggests that it can be used to validate attribute nodes (as well as documents and elements), and the prose description concurs with this. However the function summary says "can be invoked to validate a document or element node against this schema."

Test case xsd-validator-092 expects an attribute node to be rejected.

Also the Notes in the specification say

The validation process is explained in more detail in the XQuery ([[XQuery 4.0: An XML Query Language]] section [4.25 Validate Expressions] and XSLT ([[XSL Transformations (XSLT) Version 4.0]] section [25.4 Validation]

but the detailed description has since been moved to F&O 17.2.4.

Note that XSLT has always allowed validation of free-standing attribute nodes, but the validate expression in XQuery allows only document and element nodes.

Per #2315:

1. Added ‘at-risk’ changes to the fn:insert-separator, array:members, and array:of-members
2. Added ‘at-risk’ changes to the XPath/XQuery section on map and array filtering
3. Added a note about what ‘at risk’ means to the status sections

These should have no effect without additional commits, but they have to be merged into main in order to have a visible effect on my subsequent PR.

In XQuery 3.1 there are two XDM types to represent binary values xs:base64binary and xs:hexBinary. The current draft adds binary literals as an additional option.

Thus, it is possible to base64 encode any value by casting a xs:base64binary to a xs:string.

xs:base64Binary("+w==") => xs:string()

There is no standard way to serialize those binary values to the URL safe variant of that encoding described in section 5 of RFC 4648.

The simplest workaround is replacing the unsafe characters of the alphabet (+ and /) and dropping the padding at the end with

xs:base64Binary("+w==") => translate("+/=", "-_")

This of course will only work for relatively small binary values. In order for processors to offer a performant and efficient way I see several options.

1. adding new type xs:base64BinaryUrlSafe whose string representation uses the adapted alphabet with - and _ and does not add padding at the end
2. a new function in fn namespace fn:encode-base64-url-safe($data as (xs:string | xs:base64Binary | xs:hexBinary)) as xs:string
3. a new function in bin namespace bin:encode-base64-url-safe($data as (xs:string | xs:base64Binary | xs:hexBinary)) as xs:string
4. add an output option that will serialize all binary values to base64 url-safe when cast to strings

Addendum

I am also wondering why binary values cannot be created from numeric literals. Especially now that we have the binary notation for integer literals and the xs:integer type is unbounded this would be a perfectly fine literal notation to create binary values from. At least as suitable as string literals that are currently allowed.

xs:hexBinary(0xfb) and xs:base64Binary(0b11111111)

In response to: QT4CG-147-02: NW to chase up DN and LQ about follow-up to the generator discussion

Use Case: News Feeds Aggregation Using Generators

Contents

Use Case: News Feeds Aggregation Using Generators

• Actors
• Goals
• Functional Requirements
• Constraints / Assumptions / Preconditions
• Proposed High-Level Solution
• Known Approaches that are Problematic
• Benefits of the Generators Approach
• End-to-End Flow
  • Brief Description of the Core Processes in the Pipeline
  • Notes on the Process Pipeline
• Why This Fits the Generator Datatype Extremely Well
• Alternative Flows
  • Alternative Flow-1: A Feed Temporarily Stops Producing New Items
  • Alternative Flow-2: Partial Consumption of the Pipeline
  • Alternative Flow-3: Editor Inserts or Reorders Items 11
• Exception Flows
  • Exception Flow-1: Feed Unreachable or Network Failure
  • Exception Flow-2: Malformed Feed Data
  • Exception Flow-3: Resource Exhaustion Risk
• Postconditions
• References

The Problem

Modern RSS/JSON aggregators must process hundreds of continuously updating feeds without excessive memory usage or latency, while supporting filtering, merging, and prioritization in real time.

Actors

• End-User
• Editor
• Administrator
• System components (internal processes acting as secondary actors)
• External services (RSS providers, APIs, social signals)

Goals

• End-User
  “As a user, I want to get the latest, up-to-the-minute news from many important sources. I want each brief news item to be presented with a link to more detailed information from the original source.”

• Editor
  “As an editor, I want to be alerted to any change in the aggregated news-stream, as it happens continuously, and to have powerful ways of inserting, reordering, appending, prepending or deleting one or more news-items.”

• Administrator
  “As an administrator, I want to start, stop, or restart the system, manage the configured feeds, and monitor operational health and error conditions.”

Functional Requirements

• Consume RSS / Atom / JSON-LD feeds incrementally
• Filter items by topic or sensitivity
• Merge multiple feeds chronologically
• Produce continuously updated summaries

Constraints / Assumptions / Preconditions

Assumptions

• Feeds may be large or unbounded
• Items arrive over time

Constraint

• Memory usage must remain bounded

Preconditions

• At least one news feed is configured
• Feeds are RSS or JSON-LD and timestamped
• Items within a feed are presented in reverse-chronological order
• Each item contains a content-link or optionally - inline content
• Items may belong to multiple categories

Proposed High-Level Solution

Each feed is modeled as a generator producing yield values lazily.
The ordered set of values produced by successive, demand-driven calls to move-next() is called the yield of the generator.

A generator’s yield may be finite or infinite, and may be empty for a given generator instance without implying exhaustion of the underlying data source.

Known Approaches That Are Problematic

These approaches require full materialization in memory:

• Eager sequences (XPath)
• DOM-style loading
• Materialized feeds

Benefits of the Generators Approach

• Bounded memory usage
• Low latency
• Composability
• Deterministic control of evaluation

End-to-End Flow

+-------------------------------+
| 1. Feed Fetching              |
| Input:  external providers    |
| Output: G_rawItems            |
+---------------+---------------+
                |
+---------------v---------------+
| 2. Normalization              |
| Input:  G_rawItems            |
| Output: G_normalizedItems     |
+---------------+---------------+
                |
+---------------v---------------+
| 3. Filtering                  |  <-- unwanted content removed
| Input:  G_normalizedItems     |
| Output: G_filteredItems       |
+---------------+---------------+
                |
+---------------v---------------+
| 4. Topic Classification       |
| Input:  G_filteredItems       |
| Output: G_classifiedItems     |
+---------------+---------------+
                |
+---------------v---------------+
| 5. Clustering                 |
| Input:  G_classifiedItems     |
| Output: G_clusteredItems      |
+---------------+---------------+
                |
+---------------v---------------+
| 6. Ranking                    |
| Input:  G_clusteredItems      |
| Output: G_rankedItems         |
+---------------+---------------+
                |
+---------------v---------------+
| 7. Summary Page Generation    |
| Input:  G_rankedItems         |
| Output: G_summaryPageItems,   |
|         HTML                  |
+---------------+---------------+
                |
+---------------v---------------+
| 8. Detail Page Generation     |
| Input:  G_summaryPageItems    |
| Output: HTML Detail Pages     |
+-------------------------------+

Remarks

1. The participating generator instances are named using the convention G_{name}.
2. Every stage except the final one produces a new generator.
3. Every stage except the very first uses a generator as its input.
4. Arrow semantics: the output generator of one stage is the input for the next stage.

Brief Description of the Core Processes in the Pipeline

Process 1 — Feed Fetching & Acquisition

Goal:
Continuously pull RSS / Atom / JSON-LD feeds from CNN, Fox, NBC, BBC, etc.

Includes:

• Periodic polling (e.g., every 5 minutes)
• Detection of new items (GUID, URL hash, published timestamps)
• N-way merging to ensure the resulting yield is sorted in reverse-chronological order
• Basic sanity validation (e.g., XML schema validity)

Output:
A generator whose yield values are raw feed items (XML / JSON documents) → input to Process 2.

Process 2 — Parsing & Normalization

Goal:
Convert heterogeneous raw feed items into a uniform internal format.

Normalized fields include:

• Title
• Description / Summary
• Full text (if available)
• URL
• Publication time (converted to UTC)
• Source
• Images, categories, tags
• Named entities (optional NLP-based enrichment)

Output:
A generator yielding clean, normalized NewsItem documents → input to Process 3.

Process 3 — Content Filtering & Exclusion Rules

Goal:
Remove unwanted items early using configurable rule sets.

Examples:

• Blocked topics: politics, celebrity gossip, violence, etc.
• Blocked entities: Donald Trump, Joe Biden, Kanye West, etc.
• Blocked publishers (optional)
• Expiration rules:
  • Tech news stale after 48 hours
  • Breaking news stale after 6 hours

Techniques:

• Keyword filtering
• Named Entity Recognition (NER)
• Sensitive-topic classifiers (ML-based)
• Freshness scoring

Output:
A generator yielding allowed, filtered NewsItem documents → input to Process 4.
Rejected items are stored separately for auditing.

Process 4 — Topic Classification

Goal:
Assign each item to one or more topics.

Example topics:

• Politics
• World
• Tech
• Health
• Sports
• Business
• Disasters / Urgent events
• Crime / Safety
• Entertainment

Approaches:

• Fine-tuned BERT classifier (preferred)
• TF-IDF + SVM (simpler)
• Feed-provided category tags (fallback)

Output:
A generator yielding categorized NewsItem documents → input to Process 5.

Process 5 — Similarity Analysis & Clustering

Goal:
Group news items from different sources describing the same event.

Techniques:

• Semantic vector embeddings (e.g., SBERT, Ada embeddings)
• Cosine similarity
• Hierarchical clustering or DBSCAN

Produces:

• Clusters of highly similar articles
• A primary (best) representative per cluster

Output:
A generator yielding clusters of related articles → input to Process 6.

Note:
To better match streaming behavior, clustering may operate within bounded windows (e.g., sliding windows) while still consuming the input generator.

Process 6 — Ranking, Urgency, and Freshness Scoring

Goal:
Prioritize which news appears on the Summary Page.

Computed scores:

• Freshness score (more recent → higher)
• Urgency score (disasters, crises, violence)
• Coverage score (number of sources reporting)
• Engagement score (optional: social signals)

Weighted formula:

FinalScore = a*Urgency + b*Freshness + c*Coverage + d*EditorRules

Items with the highest scores per topic are selected.

This stage does not require a full total ordering; instead a partial ordering (e.g., top-K per topic) preserves bounded memory.

Editor-driven operations (insert, remove, reorder) are modeled as generator transformations applied downstream of ranking.

Output:
A generator yielding ranked clusters → input to Process 7.

Process 7 — Summary Page Generation

This stage consumes the input generator and produces finite views intended for presentation.

Goal:
Build a continuously updated Summary Page (“Front Page”) containing:

• Top events per topic
• Short summaries
• Links to primary articles
• “Read similar news” (cluster siblings)
• Source icons
• Timestamp of most recent update

The page auto-refreshes and always reflects the newest items.

Process 8 — Detailed Pages & Cross-Links

This stage consumes its input generator and produces finite presentation views.

For each cluster:

• Canonical article (primary representative)
• Related articles across sources
• Timeline of developments
• Additional metadata (images, entities, tags)

Cross-links include:

• “More like this…”
• “Earlier developments…”
• “Follow-up stories…”

Notes on the Process Pipeline

• Feed Fetching typically wraps one or more data providers
  → produces G_rawItems lazily (RSS, JSON APIs, DB cursors, web services)
• Every stage is expressible as:
  • for-each, filter, append, prepend, insert-at, remove-where, concat, or fold, etc., producing a new generator derived from the previous one
• No stage requires full materialization unless explicitly demanded
  (e.g., to-array, bounded sort, pagination)
• Infinite generators are valid until stage 6; stages 7–8 typically consume finite prefixes (take(n))

Why This Fits the Generator Datatype Extremely Well

• The pipeline is a composition of generator transformers
• Each box maps almost 1-to-1 to generator operations
• External data providers integrate naturally at Stage 1
• Sorting can be introduced in different ways:
  • External merge-sort over generators
  • Bounded-window ranking
  • Top-K lazy ranking – e.g. using heaps.

Alternative Flows

Alternative Flow 1 — Feed Temporarily Stops Producing New Items

Condition:
A feed is reachable but has no new items since the last polling cycle.

Flow:

1. The feed generator advances (move-next()).
2. The data provider returns no new items.
3. The feed-generator instance yields no items during this interval.
4. Downstream generators remain operational.
5. If all feeds are empty, no new items are added downstream.

Result:
The pipeline continues uninterrupted; no special handling is required.

Alternative Flow 2 — Partial Consumption of the Pipeline

Condition:
Only a finite prefix of the stream is required (e.g., top N items).

Flow:

1. Downstream consumers apply take(N).
2. Upstream generators are evaluated only as needed.
3. Remaining potential yield values are never materialized.

Result:
Latency and memory usage remain bounded. The pipeline supports early termination naturally.

Alternative Flow 3 — Editor Inserts or Reorders Items

Condition:
An editor manually modifies the aggregated stream.

Flow:

1. Editor operations are applied as generator transformations
   (append, prepend, insert-at, remove-at, remove-where).
2. A new generator with the modified yield is produced.
3. Downstream stages consume it transparently.

Result:
Editorial control integrates seamlessly without breaking the pipeline.

Exception Flows

Exception Flow 1 — Feed Unreachable or Network Failure

Condition:
A feed cannot be reached during polling.

Flow:

1. The data provider reports an error or timeout.
2. The next instance of the feed generator yields no items during this polling interval.
3. The error is logged for monitoring.
4. A retry policy (e.g., exponential backoff) is applied.

Result:
The system continues operating with remaining feeds.

Exception Flow 2 — Malformed Feed Data

Condition:
A feed item is malformed (invalid XML/JSON or schema validation problems, e.g. missing required fields).

Flow:

1. The normalization stage detects the issue.
2. The item is discarded or quarantined.
3. Processing continues with subsequent items.

Result:
Malformed data does not propagate downstream.

Exception Flow 3 — Resource Exhaustion Risk

Condition:
A downstream operation risks exceeding memory limits.

Flow:

1. Bounded strategies (windowing, top-K selection) are applied.
2. Full materialization is avoided.
3. If needed, the operation degrades gracefully (e.g., reduced clustering depth).

Result:
System stability is preserved under load.

Postconditions

Upon successful execution:

Functional Outcomes

• End users see an up-to-date Summary Page.
• Each summary item links to a Detailed Page.
• Editors can intervene using generator operations.
• Administrators retain full system control.

Technical Guarantees

• Memory usage remains bounded.
• Latency is minimized through lazy evaluation.
• Full materialization occurs only when explicitly requested.

System State

• All generators remain composable.
• Generator composition remains valid after alternative and exceptional flows.
• Empty generators correctly represent exhaustion.
• Infinite yields are supported up to stages that require finiteness.

References

1. RSS 2.0 Specification
   https://www.rssboard.org/rss-specification

2. Atom Publishing Protocol (RFC 5023)
   https://www.rfc-editor.org/rfc/rfc5023

3. JSON-LD Specification
   https://json-ld.org/spec/

4. TF-IDF, “Understanding TF-IDF (Term Frequency-Inverse Document Frequency)”, https://www.geeksforgeeks.org/machine-learning/understanding-tf-idf-term-frequency-inverse-document-frequency/

5. TF-IDF + SVM, “Strengthening Fake News Detection: Leveraging SVM and Sophisticated Text Vectorization Techniques. Defying BERT?”, https://arxiv.org/html/2411.12703v1

6. Sentence-BERT (SBERT)
   Reimers, N. & Gurevych, I., 2019
   https://arxiv.org/abs/1908.10084

7. Fine-tuned BERT, “Fine-tuning a BERT model”, https://www.tensorflow.org/tfmodels/nlp/fine_tune_bert

8. Ada Embeddings (OpenAI)
   Radford et al., 2021
   https://arxiv.org/abs/2103.00020

9. Cosine Similarity
   https://en.wikipedia.org/wiki/Cosine_similarity

10. Hierarchical Clustering
    https://en.wikipedia.org/wiki/Hierarchical_clustering

11. DBSCAN
    Ester et al., 1996
    https://www.aaai.org/Papers/KDD/1996/KDD96-037.pdf

This PR adds an exported SCM version of the fos.xsd schema with an embedded license. Updates to the build script use it to validate all function-catalog.xml files.

Pro: validation

Con: any change to the fos.xsd also has to be accompanied by an update to the exported schema, which probably, only Mike or I can do.

The spec says (in both 3.1 and 4.0):

The inline elements are those included in the %inline category of any of the HTML 4.01 DTDs or those elements defined to be phrasing elements in HTML5

This could be read as defining one set of inline elements for version="4.01" and a different set of inline elements for version="5.0", or it could be read as indicating that an element is an inline element if it satisfies either of these two conditions.

F+O: editorial corrections to issues identified in #2195 (mainly corrections to examples), plus completion of missing change metadata.

Fix #2337

Fixes a number of problems from issue #2195.

Stylesheet change to allow empty NT elements, bringing them into line with other referencing elements such as termref and xnt. The markup <nt def="AxisStep"/> is treated as equivalent to <nt def="AxisStep">AxisStep</nt>. This removes a common source of error which tends to result in missing text in the spec rather than in any kind of build error.

Fix #2359

Fix #2344

Fix #2369

Currently function-catalog.xml is invalid against the schema fos.xsd. This PR updates the schema to allow "character-maps" in the enumeration of allowed context dependencies.

(Note, this should probably cause the build to fail. The problem was only spotted when using Oxygen to query the function catalog.)

F+O section 11, Processing Binary Values, is currently empty

Has something gone wrong, or should we delete the section?

Most of the changes here are to bring the change log entries up to date. Also:

Fix #2356 Fix #2366 Fix #2367

In the XSLT 4.0 spec, 3.6 Stylesheet Element says:

The optional main-module attribute is purely documentary. By including this attribute in every stylesheet module of a package, an XSLT editing tool may be enabled to locate the top-level module of the relevant package [...]

But what does "top-level module" mean? Should this say "principal stylesheet module" instead? I can see that top-level package is defined, but not "top-level module", so I'm confused.

The new serialization parameter json-lines is documented at 26.2 Serialization parameters. But please add a "changes" entry for the new attribute json-lines in the sections 25.1 Creating Secondary Results and 26.1 The xsl:output declaration in the XSLT 4.0 spec. This is currently missing.

(Note that there is already a changes entry in the Serialization spec at 3 Serialization Parameters.)

It is often useful in a function signature for an argument type to be an extensible record type (so additional fields are allowed, which the function can ignore, saving the need to check for their presence) while the return type is non-extensible (giving better static type checking for lookup expressions, for example).

Currently this requires two separate named record types to be declared, differing only in that one of them is extensible and the other not. This duplication is clearly undesirable.

One solution to this might be to have a single non-extensible definition of the name of the record type, with some way of indicating at the point where the record type is used that extensions are allowed.

For example (probably not viable syntax as written):

fn:element-to-map-plan(
       $input as element()*   
) as fn:element-to-map-conversion-plan

fn:element-to-map(
       $node as element(),
       $plan as extensible fn:element-to-map-conversion-plan
} as map(*)

Perhaps the syntax extensible(fn:element-to-map-conversion-plan) would work.

Closes #2088

Closes #2349

Closes #2355

The serializer spec says…

Serializer are required to support values of UTF-8 and UTF-16

…whereas the XQFO spec mentions utf-8 as default value for fn:serialize. Similarly, only the UTF lower-case variants are listed for fn:unparsed-text, and there may be other places.

I think we should mention the upper-case variants everywhere, and add notes that upper/case is ignored when processing the encoding string.

Is there a particular reason why the absolute slash / is defined as complicated as…

self::gnode()/(fn:root(.) treat as (document-node()|jnode())/PP

…and wouldn’t it be helpful to simplify it get rid of the treat as expression?

self::gnode()/fn:root(.)/PP

In many cases, the document node does not exist or is not really needed, and it would allow users to use the slash for nodes that would otherwise needs to wrapped into document nodes, for example:

let $as := analyze-string('abc', 'b')
return $as/fn:match[/fn:non-match]

Section 4.7.1 discusses absolute path expressions.

The first part of the section concerns leading "/", and includes the note:

If the context value includes a map or array, it is not converted implicitly to a JNode; rather, a type error occurs.

The second part concerns leading "//", and includes the statement:

Any map or array that is present in the context value is first coerced to a JNode by applying the [fn:jtree] function.

It might be inferred that "/" doesn't do this conversion, but "//" does. However, this certainly isn't stated explicitly, and there would be no logical reason for treating the two cases differently.

We should either do the conversion for both cases, or for neither.

I'm inclined to do it for neither. Partly because an implicit conversion wouldn't do any upwards navigation to a different "root" node, as users might expect; partly because doing the conversion reduces the type information available to the compiler.

Fix #2357

We use element(*) and element() interchangeably in function signatures. I propose we standardise on the simpler form, element().

Ditto attribute().

A user experimenting with xsl:for-each-group/@split-when with my 4->3 source-code transformer, inferred that the variable $group was available within the sequence constructor of the grouping instruction.

(Unfortunately due to an error in my transformer code $group was within scope in the sequence constructor, though with the wrong value ;-) - this has since been corrected.)

A close and detailed reading of the spec shows that $group and $next are implied as only in scope for the evaluation of the @split-when expression. Might I suggest that there is a small note emphasising this is the case? Similar clarification may be worthwhile for @merge-when too.