Closes #2945

The XQFO spec contains a little section on the FunctX library, which should be revised one way or the other:

https://qt4cg.org/specifications/xpath-functions-40/Overview.html#functx-library

…

Specification of fn:update would be easier if it could call on a function with the semantics of xsl:copy. An equivalent to xsl:copy would be useful in its own right in XQuery, simplifying recursive descent using typeswitch.

In principle it's not difficult: xsl:copy has two parameters, the node to be copied and the sequence constructor for its content. Plus a couple of options for things like copy-namespaces and validation.

The main challenge (and opportunity!) is that the semantics depend on the rules for constructing complex and simple elements, which are defined separately in XSLT and XQuery. The rules are extremely similar in XSLT and XQuery, with some very minor variations (e.g. for how duplicate attribute names are handled in the content). Unifying them and bringing the rules into F+O would reduce maintenance long-term but is a significant undertaking in the short term. But it probably needs to be done anyway if fn:update is going to be defined rigorously. The description of the process in the current draft of fn:update is

C is a copy of N whose attributes and children are replaced by the nodes obtained for them. C has no parent. An item that replaces the value of an element replaces its children with a single text node and retains its attributes; for an attribute, text node, comment, or processing instruction, it replaces the string value. A value supplied for a comment or a processing instruction must satisfy the constraints that the data model places on the content of such nodes.

which leaves far too many questions unanswered (for example, what is the base URI of the new node? What if there are attributes and/or namespace nodes in the content? What is the type annotation of the result?).

There is understandable resistance to defining fn:update directly by reference to the semantics of the xsl:copy instruction, despite the fact that xsl:copy does exactly what is needed.

⊑ is being used where we mean ⊆

Closes #2938

Due to the lack of local DeltaXML licenses, I have created a poor man’s XQuery diff script. It’s not that far from the professional rendering:

Local rendering

DeltaXML rendering

The script (which uses a lot of 4.0 magic) works with recent versions of BaseX, but if the proc:system call is replaced, it should be runnable with any other current XQuery processor.

I’ll merge it in a minute (@ndw hope that’s ok); feedback is welcome.

Closes #2936

EDIT: I have updated the title of this issue (see the discussion below).

---

If I get the current spec right, the builtin function…

fn:number($value  as xs:anyAtomicType?  := .) as xs:double

…behaves different than:

declare function local:number($value as xs:anyAtomicType? := .) as xs:double { number($v) };

The divergence seems unfortunate to me. Shouldn’t we evaluate a built-in function’s defaults as if the function were declared in the caller’s module?

Next, I wondered whether we can change the existing default parameter values from . to fn:current(). There’s a rule that says…

While an expression that defines the default value of an optional function parameter is evaluated, it is set to the context value of the construct that invokes the function.

…but XQFO rules say:

• Specifically, the default value expression is evaluated using the static and dynamic context of the function caller (or of a named function reference). For example, if the default value is given as ., then it evaluates to the context value from the dynamic context of the function caller
• In XPath and XQuery, the current value is the context value with which evaluation of the expression or query was initiated. The only construct that changes it is an expression that defines the default value of an optional parameter of a function declaration

So I think the aim should be to let both . and current() behave the same way in user-defined and builtin functions, so that…

• . always points to the global context value, and
• current() points to the caller’s context value.

Somewhat weird indeed, as current() is used at other places now (in XQuery) to refer to the global context value – but that’s already the case now.

Follow-on from PR raised against the test suite:

https://github.com/qt4cg/qt4tests/pull/465

which re-opens an old debate at

https://www.w3.org/Bugs/Public/show_bug.cgi?id=26865

with new information: the canonical lexical representation of xs:decimal appears to be different in XSD 1.1 from XSD 1.0, in that it specifies that an "integer" (which I think probably means a value in the value space of xs:decimal that is also in the value space of xs:integer now has a canonical lexical representation that includes no decimal point.

This would imply that the tests in question return false, because the target type of the cast has a pattern facet that requires a decimal point. The changed result applies whenever XSD 1.1 definitions are in use, which is always the case for 4.0.

Consequences for the spec:

• In F+O Casting, we should remind readers of the rule in XDM that the datatype definitions are to be taken from XSD 1.1, not XSD 1.0, and add a note in the Incompatibilities section mentioning the impact.
• In XDM, where we give examples of things affected by the change to XSD 1.1, we should include this change.
• F+O §23.3.5 says "the pattern is tested against the canonical representation of the value, as defined for the source type". I think there's an ambiguity here as regards "the value" and "the source type", because 23.3.5 (Casting within a branch of the type hierarchy) is invoked from phase 3 of 23.3.6 (Casting across the type hierarchy), and it's not clear whether the value and source type referred to are the inputs to the cast operation as a whole, or the inputs to phase 3 of the casting operation. I don't think the ambiguity affects this particular case.

Triggered by https://github.com/qt4cg/qt4tests/pull/464:

The question was: Should fn:name#0 without a context value raise an error when the function item is created, or only when it is called? XQuery 3.1 did not say; the old tests assumed “when called”. Since #894, 4.0 says “when created”.

This is stricter than the rest of the spec: for the same case, fn:id(?) only “should” fail early, and other early errors in partial function application are only “may”. We could make all rules stricter, but this would require…

• checking the context each time a function item is created, not only when it is called;
• checking the type of the context value for each context-dependent function (e.g. 1 ! name#0);
• evaluating parameter defaults when the function item is created;
• a special rule for fn:load-xquery-module, which creates function items for all arities of a module's functions.

I think we should liberalize the rule and say: the error is always raised when the function item is called, but a processor may raise it earlier, when the item is created.

Proposal: opt-in commercial arithmetic for XPath, XQuery and XSLT 4.0

Summary

Add two components to the static context, each set per module and with a default that keeps today's behaviour:

1. Default rounding mode, used by fn:round when $mode is absent. Default half-to-ceiling, as today.
2. Arithmetic mode: double (default, as today) or decimal, as Michael Kay considered in #986. With decimal, a module calculates in decimal floating-point throughout: untyped values, fn:number, numeric literals and mixed decimal/double arithmetic give xs:decimal. Binary floating-point remains only for values that have no decimal equivalent (NaN, INF) and for functions defined on xs:double, such as math:sqrt.

A stylesheet or query for commercial calculations, such as the validation of electronic invoices, could then declare once that it rounds commercially and calculates in decimal, instead of repeating casts and rounding modes in every expression. Nothing changes for modules that do not opt in.

Motivation

Commercial calculations have legal requirements that differ from the defaults of XPath:

• Commercial rounding. VAT in Germany must be rounded half away from zero (§ 14 Abs. 4 UStG with Abschn. 14.5 Abs. 20 UStAE; DIN 1333), and the French e-invoicing standard AFNOR XP Z12-012 (§ 4.4.6) requires the same, so that "rounding two strictly opposite numbers gives strictly opposite rounded numbers". EN 16931 recommends it as well. fn:round(-2.5) returns −2, but −3 is required.
• Decimal arithmetic. Amounts are decimal numbers. Without a schema, which is the normal case in Schematron validation, every amount is xs:untypedAtomic, and arithmetic converts it to xs:double: with <a>0.1</a> and <b>0.2</b>, a + b is 0.30000000000000004 and a + b = 0.3 is false; round(<p>1.005</p>, 2) returns 1, not 1.01.

Rule authors can avoid both problems today, with xs:decimal() casts on every operand and the $mode argument on every fn:round call. In practice they miss some. The official EN 16931 validation artefacts (release 1.3.16), maintained by CEN and used across the EU, are an example. Rule BR-S-08 in UBL tolerates a deviation of the VAT category taxable amount below 1.00 and tests

xs:decimal(cbc:TaxableAmount + 1) > sum(…xs:decimal(cbc:LineExtensionAmount)…)

The cast comes after the addition, so the addition is done in xs:double. For an invoice whose lines add up to 197.37 and whose declared taxable amount is 196.37, exactly 1.00 too low, 196.37 + 1 gives 197.37000000000000455 in binary, which is greater than 197.37, and the invalid invoice is accepted. The same amount 1.00 too high (198.37) is rejected, and so is the same invoice in CII syntax, whose rule compares exactly. About half of all two-decimal amounts behave like 196.37 in one of the two directions. The CII rules BR-AE-08, BR-E-08, BR-G-08, BR-IC-08 and BR-Z-08 compute ../ram:BasisAmount - 1 on the untyped amount in the same way. The verdict of the reference validation depends on binary arithmetic, not on the rule. Test invoices and a reproducible comparison.

Rules like these are written by domain experts, not by XPath specialists. EN 16931 alone has several hundred of them, and national extensions (XRechnung, Factur-X/ZUGFeRD, Peppol) and company rules add more. A single declaration per rule set is realistic; a cast on every operand is not.

Relation to earlier work in this group

• #1187 and #1274 added the rounding modes to fn:round, following a request about financial rounding (Saxon issue 6408). This proposal only adds a way to change the default of $mode for a module.
• In #986, Michael Kay considered "an 'arithmetic mode' in the dynamic context, set to either 'double' or 'decimal'", and later proposed to "change conversion from untypedAtomic to numeric to depend on the lexical form of the value, as it does for numeric literals". #2218 adopted a related rule for general comparisons: an untyped value compared with a number is cast to the type of the numeric operand, with xs:double as fallback. This proposal takes the arithmetic mode further, to decimal arithmetic throughout a module, as an opt-in, so that existing code is not affected.

Proposal

1. Default rounding mode

A new static context component, default rounding mode, whose value is one of the modes of fn:round. Its default is half-to-ceiling.

fn:round#1 and fn:round#2, and fn:round#3 with an empty $mode, use the default rounding mode of the static context of the call. Dynamic calls such as round#1 bind it at the point where the function item is created, as for the default collation.

2. Arithmetic mode

A new static context component, arithmetic mode, with the values double (the default) and decimal. The mode double is today's behaviour. The mode decimal avoids binary floating-point wherever a decimal value can represent the number, which gives exact results for every calculation that XPath can express in decimals:

• Untyped values. Wherever an xs:untypedAtomic value is converted to a number without an explicit required type of xs:double or xs:float, a lexical form that is valid as a numeric literal (after whitespace normalization) is cast to xs:integer or xs:decimal, including scientific notation such as 1.5E3. Other values, such as NaN and INF, are cast to xs:double as today, and invalid input raises the same errors. This applies to arithmetic operators, fn:sum, fn:avg, fn:min, fn:max, and the coercion of arguments whose required type is xs:numeric (such as fn:round and fn:abs).
• fn:number returns an xs:decimal for such a lexical form, and NaN as today for anything else.
• Numeric literals. A DoubleLiteral such as 1.5e0 denotes an xs:decimal.
• Mixed arithmetic. An operation with an xs:decimal and an xs:double or xs:float operand converts the binary operand to xs:decimal and returns an xs:decimal, the reverse of today's promotion, unless that operand is NaN or infinite.

Binary floating-point then remains only where decimal cannot represent the value (NaN, positive and negative infinity), in functions defined on xs:double such as math:sqrt, math:log or math:pow, whose results are mostly irrational, and in operations whose operands are all explicitly typed as xs:double or xs:float. General comparisons keep the rules of #2218.

Syntax

• XSLT: standard attributes [xsl:]default-rounding-mode and [xsl:]arithmetic-mode, allowed on any element and scoped like [xsl:]default-collation:

  <xsl:stylesheet version="4.0" default-rounding-mode="half-away-from-zero"
                  arithmetic-mode="decimal" …>
  

• XQuery: prolog declarations such as declare default rounding-mode "half-away-from-zero"; and declare arithmetic-mode decimal;

• XPath hosted by other languages: static context properties set by the host language or API. Schematron could pass them through from a rule set to the generated XSLT; that is a matter for ISO Schematron, not for this group.

Why the static context

• Compatibility. The defaults are today's behaviour. Only modules that declare the new settings change, so no existing stylesheet or query breaks.
• Visibility. The declaration is part of the module. A reader sees how it calculates, and it gives the same results on every conforming processor, unlike a processor configuration.
• Scope. A validation service runs rule sets and unrelated stylesheets in the same process. A module-level setting affects only the rule set that asks for it.
• Precedent. XPath 1.0 compatibility mode is a static context component that already changes the semantics of arithmetic and comparisons for a whole module.

Costs of the decimal mode

Decimal floating-point is more accurate for every value that has a decimal representation, which is what commercial data consists of. Its main cost is performance: decimal arithmetic needs more memory and CPU time than hardware binary floating-point. It also changes some results, which is why the mode is opt-in:

• Division by zero raises FOAR0001 instead of returning INF or NaN.
• Nonterminating division, such as 1 div 3, is rounded to an implementation-defined precision, as for xs:decimal today, where xs:double gives about 17 significant digits. fn:divide-decimals (#1261) gives explicit control.
• Extreme magnitudes such as 1e308 are exact in decimal but need correspondingly large values; an implementation may convert exponents beyond a limit to xs:double.

Open questions

• How is an xs:double operand converted in mixed arithmetic: to its exact binary value (0.1000000000000000055511151231257827…), or to the shortest decimal that identifies it (0.1)? The second matches what users see and what the value usually came from.
• Should the decimal mode also apply to the arithmetic of XPath 1.0 compatibility mode?

Implementation experience

Saxon-HE-enhanced-accuracy, a fork of Saxon-HE 13.0, implements both settings as fixed defaults in a few hundred lines: half away from zero, and decimal arithmetic for untyped values, fn:number, scientific notation and XPath 1.0 compatibility mode. It keeps mixed arithmetic with explicitly typed xs:double values binary. Saxon's existing rounding modes and the #2218 comparison code carried most of the work. 32 example calculations are run as tests against both the stock and the modified processor, and the official EN 16931 validation artefacts are run unchanged on both. The code is available under the Mozilla Public License 2.0, as Saxon-HE, and could serve as a starting point for an opt-in implementation.

This PR provides a more conservative alternative to the rather over-ambitious proposal made in PR #2815.

We already have two kinds of record type in our spec, this proposal attempts to differentiate them more clearly.

The terms "nominative record type" and "structural record type" are introduced: the term "named record type" was confusing because it could apply either to "declare type record" or "declare record", which have rather different semantics. The rules for type matching and subtyping in both cases are clarified. The special characteristics of "recursive record types" are now ascribed to "nominative record types" whether or not they happen to be recursive.

The PR paves the way for introducing nominative record types derived by extension and restriction in a separate proposal.

Fix #2908

I propose that it should be possible (perhaps by means of an annotation) to identify one of the methods on a record type as being a validation method. The validation method will be automatically invoked whenever an instance of the record type is constructed or modified (using "but with"), and will trigger a dynamic error if it returns false.

Alternatively we could allow a predicate on the record definition:

declare record (x as integer, y as integer, z as integer) where empty(duplicate-values((?x, ?y, ?z)))

Closes #2919 …and no further XQFO functions from our side.

…and some more, which I hope will improve usability:

• Layout selection: each element’s layout is chosen from the plan entry, then the * fallback, or otherwise inferred from the value’s shape.
• One table: each layout has one row that says which values it accepts and how the element is rebuilt.
• One child or several: the plan decides whether an array under a key means one child or repeated children.
• Stricter arrays: attributes come first, and forms that element-to-map never produces are rejected.
• Marker clash: a key starting with @ is read as an attribute, even with a custom attribute-marker.
• () is "", so JSON null from parse-json works.
• Note: how to convert JSON objects with several keys, and JSON arrays, by wrapping them.
• Error list: simplified in function-catalog.xml.

Only 2, 3 existing test cases will need to be changed.

Closes #2930

Proposed changes, to improve roundtripping:

1. One child or several: under a child key, create one child if the value has the shape of the child’s planned layout, and one child per member if it’s an array of such values. Only list treats an array as the content of a single child; list-plus does not.
2. Fallback: if a value doesn't have the shape of its planned layout, use the plan’s * layout. If that doesn’t fit either, or there is none, raise an error.
3. Shapes: add a table of what each layout’s value looks like, as the basis for 1 and 2.
4. Limit: add a note that a fallback result which happens to have the shape of the planned layout is converted back with the planned layout, so the element comes back different.

…and some cleanups.

Closes #2928

file:read-text should be further aligned with fn:unparsed-text:

• file:read-text, file:read-text-lines: Change default encoding from utf-8 to ()
• file:read-text: Add option normalize-newlines.

Discussion of JNode subtypes and naming conventions

Currently we have the jnode() type, jkey and jvalue() functions. It is quite likely that we will need a few more functions and a number of jnode() subtypes (useful when using path selectors and templates).

I think, we will need the following subtypes of jnode(): jmap(), jarray, jsequence(), jatom(), jempy-node().

Perhaps, a better naming option: map-jnode(), array-jnode(), sequence-jnode(), atom-jnode(), empty-jnode().

Does it make sense to move them all to a separate namespace with a default j prefix?

Using a separate namespace, the subtyping relationships are as follows:

• j:empty-node() ≡ j:node(*, empty-sequence() )
• j:empty-node(N) ≡ j:node(N, empty-sequence() )
• j:item-node() ≡ j:node(*, item() )
• j:item-node(N) ≡ j:node(N, item() )
• j:sequence() ⊆ j:node()
  • note that an item can be an instance of both j:sequence() and j:empty-node()
• j:sequence(N) ⊆ j:node(N)
• j:map() ≡ j:node(*, map(*) )
• j:map(N) ≡ j:node(N, map(*) )
• j:array() ≡ j:node(*, array(*) )
• j:array(N) ≡ j:node(N, array(*) )
• j:atom() ≡ j:node() \ ( j:sequence() | j:map() | j:array() )
  • its instances are all and only instances of j:node() that are not instances of ( j:sequence() | j:map() | j:array() )
  • note that it includes instances of j:empty-node() that are not instances of j:sequence() (should that be changed?)
• j:atom(N) ≡ j:node(N) \ ( j:sequence() | j:map() | j:array() )

In the JNode mapping scheme I propose, nodes of the j:sequence() type are located only on the seq:: axis, and this axis can contain only nodes of the j:sequence() type.

Update: j:item-node() added.