Since we link to pull requests in the spec and in test cases, I wonder whether it would be possible to publish a permanent diff showing the effect of each PR?

Essentially, the idea would be to take the HTML diff as we currently publish it, and reduce it to those sections of the specs that actually contain changes.

I would find this very useful, for example, when a PR has been accepted but is still marked with "Tests needed" - it's not easy at present to see retrospectively what tests might be required. It would also be useful, of course, when implementing the PR. But I think that all readers of the specs might find this beneficial.

Building on the awesome work from @ndw I just tweaked the grammar explorer layout a little more

• layout works better on bigger and smaller screens
• consistent navigation between all screens with back button alwasy on the navigation at the top
• consistent sizes, paddings, colors set by CSS variables
• output as html5 which fixes small issues with whitespace in inline elements
• additional, minor layout improvmements

Before

After

Fix #2421

XSLT Section 3.11 describes embedded stylesheet modules - a stylesheet rooted at an element node which is not the outermost element of a document. There are no real conformance requirements associated with this feature and it isn't widely used. I proposed we drop the section, while retaining the statement in §3.5 that a stylesheet module can be "all or part" of an XML document.

Simplified stylesheets have changed so that the implicit template rule now does match="." rather than match="/".

This creates a theoretical incompatibility when

(a) the stylesheet is invoked supplying a node other than a document node as the input. It will now execute the (only) template rule, previously it would execute the built-in template for the node kind

(b) the simplified stylesheet module is included/imported into another stylesheet. This is a highly unlikely scenario, but it is tested by test case include-0601.

The incompatibility should be documented.

h/t @line-o

Plus a few other tweaks.

Fix #2292

Fix #2399

Fix #2399

Fix #2406

There seem to be two consistent errors in the diff markup that appears in PRs on the dashboard:

1. When an inline <code> element is modified, the diff version shows the old code as deleted (red background), but does not show the new code.

For example:

2. When a grammar entry is modified, the text gets duplicated.

For example:

This PR drops the concept of extensible record types, replacing it with a rule that coercion to a record type drops any map entries that are not defined by the record type. In effect this means that a record type used when declaring a function parameter is implicitly extensible.

The benefits of the proposal are:

• It simplifies the spec, especially rules on type subsumption and on generation of implicit constructor functions
• It avoids the need to declare pairs of record types, one extensible and one not.
• It avoids all the awkward decisions about whether record types used in core functions should be extensible or not.

The rules for type patterns in XSLT are changed to invoke coercion.

Fix #1484 Fix #2365

Fix #2395 Fix #2396

Fix #2397

Fix #2398

Fix #2407

This issue summarizes the unresolved comments from #2195:

• [x] Oxford/serial comma should be used consistently at numerous places. Candidates:
  • sine, cosine and tangent
  • durations, dates and times
  • year, month, day, hour, minute, second and timezone
  • The month, day, hour and minute components
  • the tokens w, W and Ww
  • rules for overflow, underflow and approximation
• [ ] The comma may need to be removed at other places:
  • This function is context-independent, and focus-independent.
  • A dynamic error is raised [err:FODC0002] if a relative URI reference is supplied, and the base-URI property in the static context is absent.
• [x] fn:distinct-values: $coll → $collation
• [x] “If … is an empty sequence” vs. “If … is the empty sequence” (which one do we prefer?)
  • MHK: I can't say I have a strong preference, but pedantically, the is probably more accurate.
• [ ] Section 2.1.3 Values of the XPath 4.0 spec includes a change note highlighting that the terms XNode and JNode have been introduced but the section text only mentions XNode; there's no reference to JNode in that section other than in the change note itself.
• [ ] In the XPath 4.0 spec, the publoc URL appears to be incorrect (/spec/header/publoc/loc)
• [x] A number of examples in the function catalog should be annotated with spec="XQuery" so that the corresponding test cases are marked as inapplicable to XPath. Specifically:

  fo-test-fn-count-001
  fo-test-fn-deep-equal-005
  fo-test-fn-every-010
  fo-test-fn-function-annotations-002
  fo-test-fn-function-annotations-003
  fo-test-fn-hash-009
  fo-test-fn-hash-010
  fo-test-fn-serialize-004
  fo-test-fn-sort-with-005

• [ ] The serialization spec, in the section on the serialization parameter document, makes it rather hard to discover that the namespace prefix output is bound to the URI http://www.w3.org/2010/xslt-xquery-serialization
• [x] F+O The definition of fn:compare refers to a function fn:months-from-dateTime, this should be fn:month-from-dateTime
• [ ] F+O The description of unparsed-text refers to the available text resources component of the dynamic context which has been dropped.
• [ ] The serialization spec for Adaptive serialization of function items gives an example **fn:exists#1** is serialized as **function fn:exists#1** but the word function does not actually appear in the result.
• [ ] fn:parse-html needs to define an error code for use when $encoding is an unknown or invalid encoding.
• [ ] Consistent rendition for term definitions. Most of the specs output definitions as [Definition: here is the definition]. F+O uses [Definition] here is the definition. XSLT puts the keyword "Definition" in small caps.
• [x] The function catalog contains entries for functions such as op:gMonthDay-equal that are no longer referenced.
• [ ] Serialization, HTML5, Processing Instructions: Dashes in the name must be escaped as well. Example: <?a---b c---d?> should be serialized as <!--?a- - -b c- - -d?-->..
• [ ] UTF-16le → UTF-16LE, UTF-16be → UTF-16BE (caused by #2239)
• [ ] fn:unparsed-text: Reference bin:infer-encoding, drop redundant rules
• [x] op:divide-dayTimeDuration-by-dayTimeDuration: An example could be simplified by using seconds(1)

As we use the fn alias in (almost) all XQFO signatures, we could also return it by the fn:type-of function.

The precision returned for dates, times, and durations is implementation-defined, and it has changed between Saxon releases. This leads one of our users to point out that there is no easy way to request a reduced precision (e.g. milliseconds) in order to ensure interoperability. The simplest approach we can offer seems to be current-dateTime() => format-dateTime("....") => xs:dateTime() which is pretty cumbersome and inefficient.

Rather than providing specific functions for rounding dates, times, and durations, the most versatile solution to this might be to provide functions that reduce a dateTime or duration to a record containing the numeric values of the components, allowing these to be manipulated as numbers, with a further function to reconstruct the dateTime or duration from the record: rather like the parse-uri()/build-uri() pair. Something like:

parts(current-dateTime()) ! map:put(., 'seconds', round(?seconds, 3)) ! build-dateTime()

It’s not the specification XML, it’s the pre-fixed-up HTML as XML. h/t to @martian-a for noticing first!

Schema enhancements to the function catalog for

Issue #2403 - allow non-testable results for examples to be labelled narrative="true" Issue #2177 - allow a fos:see-also element to make links to related functions

Note this PR is purely an enabler, it does not include changes to the function catalog to exploit this features, nor stylesheet enhancements to render them.

In PR #2401 Norm introduced a temporary fix needed because the function catalog in the EXPath file spec doesn't conform to the fos.xsd schema.

I think we can fix this without a schema or stylesheet change by using existing mechanisms illustrated by this example from fn:collation:

             <fos:test>
               <fos:expression>collation({ 'lang': 'de', 'strength': 'primary' })</fos:expression>
               <fos:result>"http://www.w3.org/2013/collation/UCA?lang=de;strength=primary"</fos:result>
               <fos:test-assertion>
                  <result xmlns="http://www.w3.org/2010/09/qt-fots-catalog">
                     <any-of>
                        <assert-string-value>http://www.w3.org/2013/collation/UCA?lang=de;strength=primary</assert-string-value>
                        <assert-string-value>http://www.w3.org/2013/collation/UCA?strength=primary;lang=de;</assert-string-value>
                     </any-of>
                  </result>
               </fos:test-assertion>
               <fos:postamble>The order of query parameters may vary.</fos:postamble>
            </fos:test>

The fos:result (or fos:error-result) element must always be present, and will always be rendered in the spec as the expected result. If the example will not always deliver this result, then <fos:test-assertion> can appear to give the result as it will appear in the generated test case.

But I suggest we add another attribute <fos:result narrative="true"/> to indicate that the result is given as explanatory prose, not as a testable XPath expression. This would allow another fn:collation example

         <fos:example>
            <p>The expression <code>collation({ 'lang': default-language() })</code>
               returns a collation suitable for the default language in the
               dynamic context.</p>
         </fos:example>

to be rewritten as

         <fos:example>
            <fos:test>
                <fos:expression>collation({ 'lang': default-language() })</fos:expression>
                <fos:result narrative="true">A collation suitable for the default language in the
               dynamic context.</fos:result>
                <fos:test-assertion>
                   <result xmlns="http://www.w3.org/2010/09/qt-fots-catalog"><assert>true()</assert></result>
                </fos:test-assertion>
           </fos:test>
         </fos:example>

which will (a) make it easier to fit the results into the tabular presentation of examples, and (b) cause a test case to be generated which will ensure that the example is syntactically valid.

(Or we could leave out <fos:test-assertion> in this example. If the supplied fos:result has narrative="true" and there is no test assertion, the generated test case can assume <assert>true()</assert>)

We're never going to merge this, it's just a CI test.

The content model of fos:test requires an fos:result or fos:error-result. For the EXPath File module, we have to work out what those should be or change the markup or change the schema.

In the short term, I’ve made bogus fos:results of FIXME:

RFC 8785 says:

Note: Since Not a Number (NaN) and Infinity are not permitted in JSON, occurrences of NaN or Infinity MUST cause a compliant JCS implementation to terminate with an appropriate error.

We have just decided to treat these cases more liberally, but I think we should continue to raise an error if canonical serialization is requested.

In our serialization spec, we also say:

Implementations may serialize an xs:double value using any lexical representation of a JSON number defined in [RFC 7159], but it is recommended to use the same representation as when the canonical parameter is true.

We may need to exclude the edge cases from this recommendation.

If we keep the recommendation, we may need to fix the test case Serialization-json-11, which expects -0 instead of 0 (what is returned for RFC8785).

The description for the new fn:highest function does not align with the description for the fn:lowest function; contrary to what I'd expect. In particular, the rules section makes no mention of the $key argument; so I suspect this is not up to date?

The new XSLT 4.0 functions current-merge-key-array and regex-groups are missing from the "Functions Defined in XSLT" section.

Also I believe the function unparsed-text-available should be added saying "Originally XSLT 2.0; then XPath 3.0 and later".

Please add changes entries to say "New in 4.0" for the functions: function-identity and jnode-content.

Also I assume the first change entry for function-annotations should actually say "New in 4.0" (rather than be a duplicate of the second change entry).

Please add a changes entry in the spec for the new XSLT 4.0 regex-groups function.

Standardizes on upper case for encoding names, and mentions that comparisons are case-blind.

Fix #2361

Adopted from #2351:

We have recently dropped map:pairs and map:of-pairs. With array:members, the members of arrays are returned as single-entry maps, which may confuse users. Thus, for the sake of reducing redundant functionality, do we want to keep array:members and array:of-members, or rather promote the use of for member $m and array:split/array:join instead?

If we keep the functions, we should add a dedicated record type for record(value).

Fix #2384

Fix #2378

Don’t panic, i am not suggesting a large change :)

But i would like to suggest a small change to the semantics of method calls, with the goal of third parties being able to build something much larger.

Today, we have,

A method call combines accessing a map M to look up an entry whose value is a function item F, and calling the function item F supplying the map M as the implicit value of the first argument.

I’d like to add,

If there is no such function, but there is in the map a key fn:fallback whose value is a function, then that function is called with the map, the function name, and the arity of the desired function as arguments.

In this way one could write a function that looked for an "isa" entry in the map whose value was a sequence of "class" maps, and find the function.

It’s limited in that there is no possibility of polymorphic functions, but we do not have those elsewhere in the language.

One practical benefit is that you can have a map with all your functions in it, and “instance maps’ then do not need to have, say, 40 entries for all the methods that can be called. In an application in which a map gets updated a million times (I do have one of those), adding 40 extra entries to copy is a significant burden, even though of course it’s encapsulated in a single function.

The adaptive serialization method was introduced “for the purposes of debugging query results”. For our processor, it has turned out pretty soon that it does not satisfy the requirements of our users, which is why we have introduced a custom debugging method.

I wonder what others think: Shouldn’t we relax several of the rules and let the implementation decide what to output? We haven’t defined either how the output of fn:trace needs to look like.

Some examples:

• The output of doubles often causes confusion. If parsed JSON is output, small integers will be output in exponential notation. For example, parse-json('{ "A" : 20 }') needs to be output as { "A": 2.0e1 }.
• xs:date("2001-01-01") is output as xs:date("2001-01-01"), while xs:token('x') is output as "x".
• fn() { 1 } is output as (anonymous-function)#0, whereas an implementation could prefer to use the output of fn:function-identity (see #2388), or output the original query string (if available), reproduce a string representation of the function body, etc.

I will be glad to create a PR.

The serialization specs defines rules for creating a string representation for function items. Now that we have `fn:function-identity', we should replace the rules and use this string instead.

Addresses part of issue #641

In the JSON serialization method, NaN is output as null, and infinity is output as ±1e9999.

The parse-json function adds recommendations on how to achieve round-tripping of these values.

Changes the stylesheet for generating keyword and function signature tests so that the namespace prefix "output" is explicitly declared in the test environment. This prefix is used in one of the tests and it needs to be declared if the test is to work in XPath.

Probably XQuery too. I have no idea why.

The type signature of the xsd-validator function suggests that it can be used to validate attribute nodes (as well as documents and elements), and the prose description concurs with this. However the function summary says "can be invoked to validate a document or element node against this schema."

Test case xsd-validator-092 expects an attribute node to be rejected.

Also the Notes in the specification say

The validation process is explained in more detail in the XQuery ([[XQuery 4.0: An XML Query Language]] section [4.25 Validate Expressions] and XSLT ([[XSL Transformations (XSLT) Version 4.0]] section [25.4 Validation]

but the detailed description has since been moved to F&O 17.2.4.

Note that XSLT has always allowed validation of free-standing attribute nodes, but the validate expression in XQuery allows only document and element nodes.

Per #2315:

1. Added ‘at-risk’ changes to the fn:insert-separator, array:members, and array:of-members
2. Added ‘at-risk’ changes to the XPath/XQuery section on map and array filtering
3. Added a note about what ‘at risk’ means to the status sections

These should have no effect without additional commits, but they have to be merged into main in order to have a visible effect on my subsequent PR.

In XQuery 3.1 there are two XDM types to represent binary values xs:base64binary and xs:hexBinary. The current draft adds binary literals as an additional option.

Thus, it is possible to base64 encode any value by casting a xs:base64binary to a xs:string.

xs:base64Binary("+w==") => xs:string()

There is no standard way to serialize those binary values to the URL safe variant of that encoding described in section 5 of RFC 4648.

The simplest workaround is replacing the unsafe characters of the alphabet (+ and /) and dropping the padding at the end with

xs:base64Binary("+w==") => translate("+/=", "-_")

This of course will only work for relatively small binary values. In order for processors to offer a performant and efficient way I see several options.

1. adding new type xs:base64BinaryUrlSafe whose string representation uses the adapted alphabet with - and _ and does not add padding at the end
2. a new function in fn namespace fn:encode-base64-url-safe($data as (xs:string | xs:base64Binary | xs:hexBinary)) as xs:string
3. a new function in bin namespace bin:encode-base64-url-safe($data as (xs:string | xs:base64Binary | xs:hexBinary)) as xs:string
4. add an output option that will serialize all binary values to base64 url-safe when cast to strings

Addendum

I am also wondering why binary values cannot be created from numeric literals. Especially now that we have the binary notation for integer literals and the xs:integer type is unbounded this would be a perfectly fine literal notation to create binary values from. At least as suitable as string literals that are currently allowed.

xs:hexBinary(0xfb) and xs:base64Binary(0b11111111)

In response to: QT4CG-147-02: NW to chase up DN and LQ about follow-up to the generator discussion

---

Use Case: News Feeds Aggregation Using Generators

Contents

Use Case: News Feeds Aggregation Using Generators

• Actors
• Goals
• Functional Requirements
• Constraints / Assumptions / Preconditions
• Proposed High-Level Solution
• Known Approaches that are Problematic
• Benefits of the Generators Approach
• End-to-End Flow
  • Brief Description of the Core Processes in the Pipeline
  • Notes on the Process Pipeline
• Why This Fits the Generator Datatype Extremely Well
• Alternative Flows
  • Alternative Flow-1: A Feed Temporarily Stops Producing New Items
  • Alternative Flow-2: Partial Consumption of the Pipeline
  • Alternative Flow-3: Editor Inserts or Reorders Items 11
• Exception Flows
  • Exception Flow-1: Feed Unreachable or Network Failure
  • Exception Flow-2: Malformed Feed Data
  • Exception Flow-3: Resource Exhaustion Risk
• Postconditions
• References

---

The Problem

Modern RSS/JSON aggregators must process hundreds of continuously updating feeds without excessive memory usage or latency, while supporting filtering, merging, and prioritization in real time.

---

Actors

• End-User
• Editor
• Administrator
• System components (internal processes acting as secondary actors)
• External services (RSS providers, APIs, social signals)

---

Goals

• End-User
  “As a user, I want to get the latest, up-to-the-minute news from many important sources. I want each brief news item to be presented with a link to more detailed information from the original source.”

• Editor
  “As an editor, I want to be alerted to any change in the aggregated news-stream, as it happens continuously, and to have powerful ways of inserting, reordering, appending, prepending or deleting one or more news-items.”

• Administrator
  “As an administrator, I want to start, stop, or restart the system, manage the configured feeds, and monitor operational health and error conditions.”

---

Functional Requirements

• Consume RSS / Atom / JSON-LD feeds incrementally
• Filter items by topic or sensitivity
• Merge multiple feeds chronologically
• Produce continuously updated summaries

---

Constraints / Assumptions / Preconditions

Assumptions

• Feeds may be large or unbounded
• Items arrive over time

Constraint

• Memory usage must remain bounded

Preconditions

• At least one news feed is configured
• Feeds are RSS or JSON-LD and timestamped
• Items within a feed are presented in reverse-chronological order
• Each item contains a content-link or optionally - inline content
• Items may belong to multiple categories

---

Proposed High-Level Solution

Each feed is modeled as a generator producing yield values lazily.
The ordered set of values produced by successive, demand-driven calls to move-next() is called the yield of the generator.

A generator’s yield may be finite or infinite, and may be empty for a given generator instance without implying exhaustion of the underlying data source.

Known Approaches That Are Problematic

These approaches require full materialization in memory:

• Eager sequences (XPath)
• DOM-style loading
• Materialized feeds

---

Benefits of the Generators Approach

• Bounded memory usage
• Low latency
• Composability
• Deterministic control of evaluation

---

End-to-End Flow

+-------------------------------+
| 1. Feed Fetching              |
| Input:  external providers    |
| Output: G_rawItems            |
+---------------+---------------+
                |
+---------------v---------------+
| 2. Normalization              |
| Input:  G_rawItems            |
| Output: G_normalizedItems     |
+---------------+---------------+
                |
+---------------v---------------+
| 3. Filtering                  |  <-- unwanted content removed
| Input:  G_normalizedItems     |
| Output: G_filteredItems       |
+---------------+---------------+
                |
+---------------v---------------+
| 4. Topic Classification       |
| Input:  G_filteredItems       |
| Output: G_classifiedItems     |
+---------------+---------------+
                |
+---------------v---------------+
| 5. Clustering                 |
| Input:  G_classifiedItems     |
| Output: G_clusteredItems      |
+---------------+---------------+
                |
+---------------v---------------+
| 6. Ranking                    |
| Input:  G_clusteredItems      |
| Output: G_rankedItems         |
+---------------+---------------+
                |
+---------------v---------------+
| 7. Summary Page Generation    |
| Input:  G_rankedItems         |
| Output: G_summaryPageItems,   |
|         HTML                  |
+---------------+---------------+
                |
+---------------v---------------+
| 8. Detail Page Generation     |
| Input:  G_summaryPageItems    |
| Output: HTML Detail Pages     |
+-------------------------------+

Remarks

1. The participating generator instances are named using the convention G_{name}.
2. Every stage except the final one produces a new generator.
3. Every stage except the very first uses a generator as its input.
4. Arrow semantics: the output generator of one stage is the input for the next stage.

---

Brief Description of the Core Processes in the Pipeline

Process 1 — Feed Fetching & Acquisition

Goal:
Continuously pull RSS / Atom / JSON-LD feeds from CNN, Fox, NBC, BBC, etc.

Includes:

• Periodic polling (e.g., every 5 minutes)
• Detection of new items (GUID, URL hash, published timestamps)
• N-way merging to ensure the resulting yield is sorted in reverse-chronological order
• Basic sanity validation (e.g., XML schema validity)

Output:
A generator whose yield values are raw feed items (XML / JSON documents) → input to Process 2.

---

Process 2 — Parsing & Normalization

Goal:
Convert heterogeneous raw feed items into a uniform internal format.

Normalized fields include:

• Title
• Description / Summary
• Full text (if available)
• URL
• Publication time (converted to UTC)
• Source
• Images, categories, tags
• Named entities (optional NLP-based enrichment)

Output:
A generator yielding clean, normalized NewsItem documents → input to Process 3.

---

Process 3 — Content Filtering & Exclusion Rules

Goal:
Remove unwanted items early using configurable rule sets.

Examples:

• Blocked topics: politics, celebrity gossip, violence, etc.
• Blocked entities: Donald Trump, Joe Biden, Kanye West, etc.
• Blocked publishers (optional)
• Expiration rules:
  • Tech news stale after 48 hours
  • Breaking news stale after 6 hours

Techniques:

• Keyword filtering
• Named Entity Recognition (NER)
• Sensitive-topic classifiers (ML-based)
• Freshness scoring

Output:
A generator yielding allowed, filtered NewsItem documents → input to Process 4.
Rejected items are stored separately for auditing.

---

Process 4 — Topic Classification

Goal:
Assign each item to one or more topics.

Example topics:

• Politics
• World
• Tech
• Health
• Sports
• Business
• Disasters / Urgent events
• Crime / Safety
• Entertainment

Approaches:

• Fine-tuned BERT classifier (preferred)
• TF-IDF + SVM (simpler)
• Feed-provided category tags (fallback)

Output:
A generator yielding categorized NewsItem documents → input to Process 5.

---

Process 5 — Similarity Analysis & Clustering

Goal:
Group news items from different sources describing the same event.

Techniques:

• Semantic vector embeddings (e.g., SBERT, Ada embeddings)
• Cosine similarity
• Hierarchical clustering or DBSCAN

Produces:

• Clusters of highly similar articles
• A primary (best) representative per cluster

Output:
A generator yielding clusters of related articles → input to Process 6.

Note:
To better match streaming behavior, clustering may operate within bounded windows (e.g., sliding windows) while still consuming the input generator.

---

Process 6 — Ranking, Urgency, and Freshness Scoring

Goal:
Prioritize which news appears on the Summary Page.

Computed scores:

• Freshness score (more recent → higher)
• Urgency score (disasters, crises, violence)
• Coverage score (number of sources reporting)
• Engagement score (optional: social signals)

Weighted formula:

FinalScore = a*Urgency + b*Freshness + c*Coverage + d*EditorRules

Items with the highest scores per topic are selected.

This stage does not require a full total ordering; instead a partial ordering (e.g., top-K per topic) preserves bounded memory.

Editor-driven operations (insert, remove, reorder) are modeled as generator transformations applied downstream of ranking.

Output:
A generator yielding ranked clusters → input to Process 7.

---

Process 7 — Summary Page Generation

This stage consumes the input generator and produces finite views intended for presentation.

Goal:
Build a continuously updated Summary Page (“Front Page”) containing:

• Top events per topic
• Short summaries
• Links to primary articles
• “Read similar news” (cluster siblings)
• Source icons
• Timestamp of most recent update

The page auto-refreshes and always reflects the newest items.

---

Process 8 — Detailed Pages & Cross-Links

This stage consumes its input generator and produces finite presentation views.

For each cluster:

• Canonical article (primary representative)
• Related articles across sources
• Timeline of developments
• Additional metadata (images, entities, tags)

Cross-links include:

• “More like this…”
• “Earlier developments…”
• “Follow-up stories…”

---

Notes on the Process Pipeline

• Feed Fetching typically wraps one or more data providers
  → produces G_rawItems lazily (RSS, JSON APIs, DB cursors, web services)
• Every stage is expressible as:
  • for-each, filter, append, prepend, insert-at, remove-where, concat, or fold, etc., producing a new generator derived from the previous one
• No stage requires full materialization unless explicitly demanded
  (e.g., to-array, bounded sort, pagination)
• Infinite generators are valid until stage 6; stages 7–8 typically consume finite prefixes (take(n))

---

Why This Fits the Generator Datatype Extremely Well

• The pipeline is a composition of generator transformers
• Each box maps almost 1-to-1 to generator operations
• External data providers integrate naturally at Stage 1
• Sorting can be introduced in different ways:
  • External merge-sort over generators
  • Bounded-window ranking
  • Top-K lazy ranking – e.g. using heaps.

---

Alternative Flows

Alternative Flow 1 — Feed Temporarily Stops Producing New Items

Condition:
A feed is reachable but has no new items since the last polling cycle.

Flow:

1. The feed generator advances (move-next()).
2. The data provider returns no new items.
3. The feed-generator instance yields no items during this interval.
4. Downstream generators remain operational.
5. If all feeds are empty, no new items are added downstream.

Result:
The pipeline continues uninterrupted; no special handling is required.

---

Alternative Flow 2 — Partial Consumption of the Pipeline

Condition:
Only a finite prefix of the stream is required (e.g., top N items).

Flow:

1. Downstream consumers apply take(N).
2. Upstream generators are evaluated only as needed.
3. Remaining potential yield values are never materialized.

Result:
Latency and memory usage remain bounded. The pipeline supports early termination naturally.

---

Alternative Flow 3 — Editor Inserts or Reorders Items

Condition:
An editor manually modifies the aggregated stream.

Flow:

1. Editor operations are applied as generator transformations
   (append, prepend, insert-at, remove-at, remove-where).
2. A new generator with the modified yield is produced.
3. Downstream stages consume it transparently.

Result:
Editorial control integrates seamlessly without breaking the pipeline.

---

Exception Flows

Exception Flow 1 — Feed Unreachable or Network Failure

Condition:
A feed cannot be reached during polling.

Flow:

1. The data provider reports an error or timeout.
2. The next instance of the feed generator yields no items during this polling interval.
3. The error is logged for monitoring.
4. A retry policy (e.g., exponential backoff) is applied.

Result:
The system continues operating with remaining feeds.

---

Exception Flow 2 — Malformed Feed Data

Condition:
A feed item is malformed (invalid XML/JSON or schema validation problems, e.g. missing required fields).

Flow:

1. The normalization stage detects the issue.
2. The item is discarded or quarantined.
3. Processing continues with subsequent items.

Result:
Malformed data does not propagate downstream.

---

Exception Flow 3 — Resource Exhaustion Risk

Condition:
A downstream operation risks exceeding memory limits.

Flow:

1. Bounded strategies (windowing, top-K selection) are applied.
2. Full materialization is avoided.
3. If needed, the operation degrades gracefully (e.g., reduced clustering depth).

Result:
System stability is preserved under load.

---

Postconditions

Upon successful execution:

Functional Outcomes

• End users see an up-to-date Summary Page.
• Each summary item links to a Detailed Page.
• Editors can intervene using generator operations.
• Administrators retain full system control.

Technical Guarantees

• Memory usage remains bounded.
• Latency is minimized through lazy evaluation.
• Full materialization occurs only when explicitly requested.

System State

• All generators remain composable.
• Generator composition remains valid after alternative and exceptional flows.
• Empty generators correctly represent exhaustion.
• Infinite yields are supported up to stages that require finiteness.

---

References

1. RSS 2.0 Specification
   https://www.rssboard.org/rss-specification

2. Atom Publishing Protocol (RFC 5023)
   https://www.rfc-editor.org/rfc/rfc5023

3. JSON-LD Specification
   https://json-ld.org/spec/

4. TF-IDF, “Understanding TF-IDF (Term Frequency-Inverse Document Frequency)”, https://www.geeksforgeeks.org/machine-learning/understanding-tf-idf-term-frequency-inverse-document-frequency/

5. TF-IDF + SVM, “Strengthening Fake News Detection: Leveraging SVM and Sophisticated Text Vectorization Techniques. Defying BERT?”, https://arxiv.org/html/2411.12703v1

6. Sentence-BERT (SBERT)
   Reimers, N. & Gurevych, I., 2019
   https://arxiv.org/abs/1908.10084

7. Fine-tuned BERT, “Fine-tuning a BERT model”, https://www.tensorflow.org/tfmodels/nlp/fine_tune_bert

8. Ada Embeddings (OpenAI)
   Radford et al., 2021
   https://arxiv.org/abs/2103.00020

9. Cosine Similarity
   https://en.wikipedia.org/wiki/Cosine_similarity

10. Hierarchical Clustering
    https://en.wikipedia.org/wiki/Hierarchical_clustering

11. DBSCAN
    Ester et al., 1996
    https://www.aaai.org/Papers/KDD/1996/KDD96-037.pdf

This PR adds an exported SCM version of the fos.xsd schema with an embedded license. Updates to the build script use it to validate all function-catalog.xml files.

Pro: validation

Con: any change to the fos.xsd also has to be accompanied by an update to the exported schema, which probably, only Mike or I can do.

The spec says (in both 3.1 and 4.0):

The inline elements are those included in the %inline category of any of the HTML 4.01 DTDs or those elements defined to be phrasing elements in HTML5

This could be read as defining one set of inline elements for version="4.01" and a different set of inline elements for version="5.0", or it could be read as indicating that an element is an inline element if it satisfies either of these two conditions.

F+O: editorial corrections to issues identified in #2195 (mainly corrections to examples), plus completion of missing change metadata.

Fix #2337

Fixes a number of problems from issue #2195.

Stylesheet change to allow empty NT elements, bringing them into line with other referencing elements such as termref and xnt. The markup <nt def="AxisStep"/> is treated as equivalent to <nt def="AxisStep">AxisStep</nt>. This removes a common source of error which tends to result in missing text in the spec rather than in any kind of build error.

Fix #2359

Fix #2344

Fix #2369

Currently function-catalog.xml is invalid against the schema fos.xsd. This PR updates the schema to allow "character-maps" in the enumeration of allowed context dependencies.

(Note, this should probably cause the build to fail. The problem was only spotted when using Oxygen to query the function catalog.)

F+O section 11, Processing Binary Values, is currently empty

Has something gone wrong, or should we delete the section?

Most of the changes here are to bring the change log entries up to date. Also:

Fix #2356 Fix #2366 Fix #2367

In the XSLT 4.0 spec, 3.6 Stylesheet Element says:

The optional main-module attribute is purely documentary. By including this attribute in every stylesheet module of a package, an XSLT editing tool may be enabled to locate the top-level module of the relevant package [...]

But what does "top-level module" mean? Should this say "principal stylesheet module" instead? I can see that top-level package is defined, but not "top-level module", so I'm confused.

The new serialization parameter json-lines is documented at 26.2 Serialization parameters. But please add a "changes" entry for the new attribute json-lines in the sections 25.1 Creating Secondary Results and 26.1 The xsl:output declaration in the XSLT 4.0 spec. This is currently missing.

(Note that there is already a changes entry in the Serialization spec at 3 Serialization Parameters.)

It is often useful in a function signature for an argument type to be an extensible record type (so additional fields are allowed, which the function can ignore, saving the need to check for their presence) while the return type is non-extensible (giving better static type checking for lookup expressions, for example).

Currently this requires two separate named record types to be declared, differing only in that one of them is extensible and the other not. This duplication is clearly undesirable.

One solution to this might be to have a single non-extensible definition of the name of the record type, with some way of indicating at the point where the record type is used that extensions are allowed.

For example (probably not viable syntax as written):

fn:element-to-map-plan(
       $input as element()*   
) as fn:element-to-map-conversion-plan

fn:element-to-map(
       $node as element(),
       $plan as extensible fn:element-to-map-conversion-plan
} as map(*)

Perhaps the syntax extensible(fn:element-to-map-conversion-plan) would work.

Closes #2088

Closes #2349

Closes #2355

The serializer spec says…

Serializer are required to support values of UTF-8 and UTF-16

…whereas the XQFO spec mentions utf-8 as default value for fn:serialize. Similarly, only the UTF lower-case variants are listed for fn:unparsed-text, and there may be other places.

I think we should mention the upper-case variants everywhere, and add notes that upper/case is ignored when processing the encoding string.

Is there a particular reason why the absolute slash / is defined as complicated as…

self::gnode()/(fn:root(.) treat as (document-node()|jnode())/PP

…and wouldn’t it be helpful to simplify it get rid of the treat as expression?

self::gnode()/fn:root(.)/PP

In many cases, the document node does not exist or is not really needed, and it would allow users to use the slash for nodes that would otherwise needs to wrapped into document nodes, for example:

let $as := analyze-string('abc', 'b')
return $as/fn:match[/fn:non-match]

Section 4.7.1 discusses absolute path expressions.

The first part of the section concerns leading "/", and includes the note:

If the context value includes a map or array, it is not converted implicitly to a JNode; rather, a type error occurs.

The second part concerns leading "//", and includes the statement:

Any map or array that is present in the context value is first coerced to a JNode by applying the [fn:jtree] function.

It might be inferred that "/" doesn't do this conversion, but "//" does. However, this certainly isn't stated explicitly, and there would be no logical reason for treating the two cases differently.

We should either do the conversion for both cases, or for neither.

I'm inclined to do it for neither. Partly because an implicit conversion wouldn't do any upwards navigation to a different "root" node, as users might expect; partly because doing the conversion reduces the type information available to the compiler.

Fix #2357

We use element(*) and element() interchangeably in function signatures. I propose we standardise on the simpler form, element().

Ditto attribute().

A user experimenting with xsl:for-each-group/@split-when with my 4->3 source-code transformer, inferred that the variable $group was available within the sequence constructor of the grouping instruction.

(Unfortunately due to an error in my transformer code $group was within scope in the sequence constructor, though with the wrong value ;-) - this has since been corrected.)

A close and detailed reading of the spec shows that $group and $next are implied as only in scope for the evaluation of the @split-when expression. Might I suggest that there is a small note emphasising this is the case? Similar clarification may be worthwhile for @merge-when too.