@qt4cg statuses

I have identified a bunch of small issues with the XSD for XSLT. I would be happy to fix these and create a pull request, but I wanted to document them first in case there's any discussion required.

1. @main-module is missing from the XSD
2. xsl:item-type is missing from the XSD
3. @constructor is missing from the XSD
4. @merge-when is missing from the XSD
5. @component - the XSD is missing the values "item-type" and "record-type"
6. For xsl:field, the XSD allows a @required attribute but the spec doesn't allow it (already mentioned as an "Editorial Note").
7. xsl:record-type/@visibility - the XSD allows too many values, it should only be public/private
8. xsl:copy-of should not allow mixed content but the XSD allows it. (This appears to be a carryover from previous versions of XSLT.)
9. xsl:fork should not allow mixed content, should it?
10. The XSD doesn't allow "##any" for the [xsl:]xpath-default-namespace attribute
11. The XSD allows @json-node-output-method to have the values "json" and "adaptive" but they are not allowed by the spec
12. xsl:merge-source is missing the assertion that @type and @validation mutually exclusive (present on other elements)
13. The XSD allows an xsl:fallback child for xsl:mode, but the spec does not, should that be eliminated?
14. The base type of xsl:map and xsl:map-entry be sequence-constructor-or-select instead of sequence-constructor-and-select.
15. The type of xsl:when should be based on xsl:sequence-constructor-or-select instead of xsl:sequence-constructor.
16. Since version is required on xsl:package, an assertion should be added enforcing that it exists (similar to other required attributes)
17. xsl:record is in the XSD but not the spec? I supsect this part of the spec is in flux so would ignore this one for now?
18. @default-validation should not have a default value in the XSD because XSD validation implies adding that attribute if it is not present. If it were added with the default value during XSD validation it could potentially override a non-default value in a parent element. This may be pedantic because it is unlikely someone would validate the XSLT with the XSD and then run the transformation with the PSVI of the XSLT, but I think it is worth adjusting.
19. Similarly, the @override attribute on xsl:function should not have a default value because adding it during XSD validation could result in a value that conflicts with @override-extension-function, which would be a static error.
20. The @schema-role has a default value of "yes" in the XSD but that should not be there.
21. Missing default values. I'm of two minds about default values in XSD, but they are useful for documentation purposes and do exist in the current XSD for some attributes. However, there are a lot of missing ones. Presumably if the Element Syntax Summary in the spec lists a default value, it should be in the XSD?
22. xsl:apply-templates/@select has the default value "child::node()" in the XSD which I don't think is technically wrong but is not officially a default in the spec. I assume it should be removed to match the spec.
23. Similarly, xsl:character-map/@use-character-maps has a default of "" in the schema but not in the spec. I assume it should be removed to match the spec.
24. The documentation for the type "nametest" says "a list of nametests". really it's just a single nametest.
25. The documentation for the type "nametests" should say "EQName" instead of "QName".
26. The documentation for the type EQNames lists two uses of the type, but there are others. Maybe the "where used" info should be left out of the documentation for ease of maintenance.
27. The documentation for the type EQName-in-namespace should say "EQName" instead of "EQNames".
28. The documentation of fixed-namespaces-type says "A sequence of tokens, each of which may be one of #default, an NCName, ..." but that should be "#standard" not "#default".
29. The @normalization-form attribute could be more specifically defined. The XSD is sufficient for validation but not great for documentation.
30. The XSD is mostly in alphabetical order but "document" comes before "decimal-format".

The fn:validator function envisages that it should be possible to validate a document supplied as an XDM document or element node, and that the result of the function should include error information with associated line numbers.

But line numbers on nodes are not a recognized property in the data model, and furthermore, there is no way to request the line number of a node.

I suggest that we

(a) In XDM, recognize location as an optional property of a node. Location is of the form record(entity-uri, line, column).

(b) In the doc() and document() and parse-xml() functions, provide an option to request maintenance of location information in the constructed tree.

(c) Provide a function fn:location(node()) to return the information if it is available.

Given that we allow backslash-escaped whitespace characters in the indent-unit and line-ending serialization parameters, it seems odd not to allow them in item-separator. But doing so would be an incompatible change.

In most of the contexts where serialization parameters are specified, XML character references can be used to set their values. The main exception is in the $options parameter of the fn:serialize() function in pure XPath, where char('\t') can be used. Perhaps we should drop the ability to use backslash-escapes in indent-unit and line-ending?

Fix #2830

Essentially editorial, though it makes some technical decisions on the interaction of parameters such as line-ending and item-separator which may be non-obvious.

Closes #2263

Various of the comments in the issue were outdated. Worth discussing:

• Do we want record types for parameter types (specifically, fn:sort-by and array:sort-by)?
• Do we want to keep the rule alive that “all built-in record types live in the fn namespace”? Otherwise, we could change fn:array-sort-key-record to array:sort-key-record.
• I have skipped bin:infer-encoding as I didn’t know how to extend the record infrastructure to the EXPath modules.

Closes #2808

Closes #2809

I don't think the new spec is entirely clear about where (in the "phases of serialization") substitution of NL by CR or CRLF takes place. It's not mentioned at all in the "phases of serialization", and for the XML output method (which is the only one I've studied so far) it is covered only by a (non-normative) Note in §5.1.7. The note makes clear that the injected CR characters are not subject to character mapping, but it's not explicit that the don't get escaped as 
 or equivalent. The phrase "the characters that the serializer would otherwise emit" suggests that end-of-line substitution happens immediately after indentation and before encoding (because that's the last stage at which the serializer is emitting characters rather than octets). But this begs the question of where substitution of a literal CR in content by 
 takes place. For the XML output method the rule is given in the long introductory section 5 ("A consequence of this rule is that certain characters must be output as character references") but it doesn't say when this happens: I think we need to clarify that it is part of phase 3(e).

Suggestion:

(a) clarify that substitution of CR by 
 is part of phase 3(e)

(b) introduce a new phase between (4) and (5) to handle substitution of line endings.

I haven't looked at how methods other than XML are affected.

Closes #2828

I added a section for symbolic links as they are referenced throughout the prose.

This is the placeholder for another editorial cleanup of the file module.

Closes #2772

A JNode can have any XDM value as its jvalue() property, but fn:jtree only accepts a map or array.

There is a use case for constructing JNodes that wrap other kinds of value - both array:of-members() and xsl:array-member accept JNodes as input.

And it's a bit silly if the only way to do it is jtree([VAL])/*

Closes #2823

fn:scan-left and fn:scan-right were dropped in #2631 (two years in the spec, no test cases). I was torn, as in principal I was convinced about the usefulnes. Many other languages have it, too (Kotlin, F#, Scala, Rust, of course Haskell, etc).

This is an attempt to bring it back. The motivation, in short:

1. fn:fold-left returns only the final value of an accumulation. It is not always trivial to rewrite code to accumulate the intermediate results.
2. Folds can be optimized to exit early (see the notes we recently added), but fn:scan can do so by design.

I will try to include examples in the PR that show the potential of the function.

And I think we don’t need a fn:scan-right function (it cannot be evaluated lazily: its first result is the fold over the entire input).

Fix #2821

We're getting a failure in XSLT test output-038 as a result of changing the default XHTML version to 5.0. The element

<one:h3>xxxx</one:h3>

is being serialized as

<one:h3 xmlns:one="http://ns.example.com" xmlns="">xxxx</one:h3>

The expected results do not include the namespace undeclaration xmlns="".

The reason we have added the namespace undeclaration is that XHTML5 serialization introduces prefix normalization, including the rule

Any namespace node for any of those three namespaces that was previously present on any element node in the [input tree] is also removed, unless the prefix that that namespace node declared is used as the prefix on the name of an attribute on that element or an ancestor of that element.

The element, prior to serialization, had a namespace node (""="http://www.w3.org/1999/xhtml"), which we have removed under this rule, and removing the namespace node has resulted in the serializer outputting a namespace undeclaration.

Possibly the phrase "or an ancestor of that element" is intended to cover this, but it's ambiguous (does it mean "or on the name of an attribute of an ancestor of that element?"). Logically, the purpose of the rule is to prevent unwanted namespace declarations.

Note: the serialization spec in this area is essentially unchanged between 3.1 and 4.0. The only change is that XHTML5 is now the default.