Update XSLTs for new Presentation XML format

[Intelligent2013]

As titled.

Source issues:

• Do not overwrite Semantic XML content in Presentation XML isodoc#610,
• Prevent presentation content for blocks from being added in PDF isodoc#607,
• Presentation XML caption refactoring mn-native-pdf#770.
• Refactor Presentation XML for terms, PDF isodoc#632
• Stop using Semantic XML for terms in PDF isodoc#633
• Refactor requirements, PDF mn-requirements#37
• Unify metanorma XML root tag under “metanorma” metanorma#440
• PDF: Semantic elements in Presentation XML isodoc#645
• PDF: concept markup isodoc#646
• PDF: eref and xref Presentation XML isodoc#649
• PDF: Presentation XML refactor, sourcecode isodoc#651
• PDF: amend and quote/source isodoc#652
• PDF: Preserving semantic XML in bibitem isodoc#655

[opoudjis]

With the new Presentation XML refactor coming out of metanorma/isodoc#617, metanorma-iso now breaks on trying to test for STS conversion. Should I remove that integration testing, or keep it?

[Intelligent2013]

With the new Presentation XML refactor coming out of metanorma/isodoc#617, metanorma-iso now breaks on trying to test for STS conversion. Should I remove that integration testing, or keep it?

Please keep it, I'm going to update mnconvert for the test passing as minimum. If the integration testing is stopping the release, then remove it temporarely.

[Intelligent2013]

Currently, mnconvert allows to convert ISO/NISO STS XML to Metanorma XML.
Metanorma XML is constantly changing. The changes in the Metanorma XML should be reflected in the STS XML to Metanorma XML conversion rules in the mnconvert. Errors and discrepancies between 'true' Metanorma XML and converted may occur, because the conversion between STS XML and Metanorma XML is not just 'tag-to-tag' conversion, there is complex logic often. The constantly updating of conversion isn't complex task, but it takes a time.

Does the generation STS XML to MN XML directly make sense? In which scenarios it can be used?

Metanorma XML can be generated in this workflow:

• ISO/NISO STS XML to Metanorma Adoc
• generate Metanorma XML from Adoc

Could it be to drop the ISO/NISO STS to Metanorma XML conversion in mnconvert?

@ronaldtse @opoudjis any thoughts on this?

[Intelligent2013]

With the new Presentation XML refactor coming out of metanorma/isodoc#617, metanorma-iso now breaks on trying to test for STS conversion. Should I remove that integration testing, or keep it?

@opoudjis mnconvert updated (https://github.com/metanorma/mnconvert/releases/tag/v1.65.0).

[opoudjis]

mnconvert is updated to 1.65.0 on my side, but STS generation is still crashing when running rspec in metanorma-iso:

  1) Metanorma::Iso::Processor generates STS from Metanorma XML
     Failure/Error:
       MnConvert.convert(in_fname,
                         { input_format: MnConvert::InputFormat::MN,
                           output_file: out_fname || "#{filename}.#{@suffix}" })
     
     RuntimeError:
       Input: XML (test.xml)
       Output: XML (test.sts.xml), format NISO STS
     
       Transforming...
       Validate XML against XSD NISO-STS-interchange-1-MathML3-XSD/NISO-STS-interchange-1-mathml3.xsd...
       /Users/nickn/Documents/Arbeit/upwork/ribose/metanorma-iso/test.sts.xml is NOT valid reason:
       [ERROR] org.xml.sax.SAXParseException; lineNumber: 51; columnNumber: 22; cvc-complex-type.2.4.a: Invalid content was found starting with element 'fmt-title'. One of '{title, address, alternatives, array, boxed-text, chem-struct-wrap, code, fig, fig-group, graphic, media, non-normative-note, normative-note, non-normative-example, normative-example, notes-group, preformat, supplementary-material, table-wrap, table-wrap-group, disp-formula, disp-formula-group, editing-instruction, def-list, list, tex-math, "http://www.w3.org/1998/Math/MathML":math, p, related-article, related-object, disp-quote, speech, statement, verse-group, fn-group, glossary, ref-list, sec, term-sec}' is expected.
       [ERROR] org.xml.sax.SAXParseException; lineNumber: 56; columnNumber: 11; cvc-complex-type.2.4.a: Invalid content was found starting with element 'fmt-name'. One of '{"urn:iso:std:iso:30042:ed-1":crossReference, "urn:iso:std:iso:30042:ed-1":definition, "urn:iso:std:iso:30042:ed-1":example, "urn:iso:std:iso:30042:ed-1":externalCrossReference, "urn:iso:std:iso:30042:ed-1":note, "urn:iso:std:iso:30042:ed-1":see, "urn:iso:std:iso:30042:ed-1":source, "urn:iso:std:iso:30042:ed-1":subjectField, "urn:iso:std:iso:30042:ed-1":xGraphic, "urn:iso:std:iso:30042:ed-1":xMathML, "urn:iso:std:iso:30042:ed-1":xSource, "urn:iso:std:iso:30042:ed-1":tig}' is expected.
     # ./lib/isodoc/iso/sts_convert.rb:27:in 'IsoDoc::Iso::StsConvert#convert'
     # ./lib/metanorma/iso/processor.rb:62:in 'Metanorma::Iso::Processor#output'
     # ./spec/metanorma/processor_spec.rb:166:in 'block (2 levels) in <top (required)>'

[Intelligent2013]

Confirmed. I'll fix.

[Intelligent2013]

@opoudjis mnconvert updated (https://github.com/metanorma/mnconvert/releases/tag/v1.66.0).

[opoudjis]

Currently, mnconvert allows to convert ISO/NISO STS XML to Metanorma XML. Metanorma XML is constantly changing. The changes in the Metanorma XML should be reflected in the STS XML to Metanorma XML conversion rules in the mnconvert. Errors and discrepancies between 'true' Metanorma XML and converted may occur, because the conversion between STS XML and Metanorma XML is not just 'tag-to-tag' conversion, there is complex logic often. The constantly updating of conversion isn't complex task, but it takes a time.

Does the generation STS XML to MN XML directly make sense? In which scenarios it can be used?

Metanorma XML can be generated in this workflow:

• ISO/NISO STS XML to Metanorma Adoc
• generate Metanorma XML from Adoc

Could it be to drop the ISO/NISO STS to Metanorma XML conversion in mnconvert?

@ronaldtse @opoudjis any thoughts on this?

I'm sorry to say I don't have the business context for STS XML > MN XML conversion, but I agree with you that using Asciidoc as an intermediary representation makes much more sense, and I've argued for it for years in comparable contexts, for the same reason you've given: the XML may be more expressive and more formal, but it is also volatile and will remain so. The Asciidoc is simply more stable.

[ronaldtse]

@Intelligent2013 yes indeed. We can drop the STS to MN XML conversion. It's more important that we focus on what users need, which is in this use case, the ability to re-edit and re-publish STS XML.

[opoudjis]

The ST4S apparently needs to be updated for Presentation XML for terms as well, the conversion is breaking in metanorma-iso

       Transforming...
       Validate XML against XSD NISO-STS-interchange-1-MathML3-XSD/NISO-STS-interchange-1-mathml3.xsd...
       /Users/nickn/Documents/Arbeit/upwork/ribose/metanorma-iso/test.sts.xml is NOT valid reason:
       [ERROR] org.xml.sax.SAXParseException; lineNumber: 57; columnNumber: 16; cvc-complex-type.2.4.a: Invalid content was found starting with element 'fmt-preferred'. One of '{"urn:iso:std:iso:30042:ed-1":crossReference, "urn:iso:std:iso:30042:ed-1":definition, "urn:iso:std:iso:30042:ed-1":example, "urn:iso:std:iso:30042:ed-1":externalCrossReference, "urn:iso:std:iso:30042:ed-1":note, "urn:iso:std:iso:30042:ed-1":see, "urn:iso:std:iso:30042:ed-1":source, "urn:iso:std:iso:30042:ed-1":subjectField, "urn:iso:std:iso:30042:ed-1":xGraphic, "urn:iso:std:iso:30042:ed-1":xMathML, "urn:iso:std:iso:30042:ed-1":xSource, "urn:iso:std:iso:30042:ed-1":tig}' is expected.

in spec/metanorma/processor_spec.rb, line 161

[Intelligent2013]

A week ago I've tested on ISO Rice document and the STS output XML was ok.
Ok, I'll check the XML from gem.

[Intelligent2013]

The ST4S apparently needs to be updated for Presentation XML for terms as well, the conversion is breaking in metanorma-iso

Issue fixed in https://github.com/metanorma/mnconvert/releases/tag/v1.67.0.

[Intelligent2013]

Source XML - new Presentation XML format without Semantic part
Output XML - NISO STS XML.

Validation issues:

[ERROR] org.xml.sax.SAXParseException; lineNumber: 48; columnNumber: 25; cvc-complex-type.2.4.a: Invalid content was found starting with element 'source-highlighter-css'. One of '{custom-meta-group}' is expected.
[ERROR] org.xml.sax.SAXParseException; lineNumber: 50; columnNumber: 12; cvc-complex-type.2.3: Element 'nat-meta' cannot have character [children], because the type's content type is element-only.
[ERROR] org.xml.sax.SAXParseException; lineNumber: 68; columnNumber: 7; cvc-complex-type.2.3: Element 'sec' cannot have character [children], because the type's content type is element-only.
[ERROR] org.xml.sax.SAXParseException; lineNumber: 69; columnNumber: 4; cvc-complex-type.2.4.a: Invalid content was found starting with element 'p'. One of '{sec, term-sec, fn-group, glossary, ref-list, non-normative-note, normative-note, non-normative-example, normative-example, notes-group}' is expected.
[ERROR] org.xml.sax.SAXParseException; lineNumber: 885; columnNumber: 41; cvc-complex-type.2.4.a: Invalid content was found starting with element 'p'. One of '{sec, term-sec, sub-part, "http://www.w3.org/2001/XInclude":include, sig-block}' is expected.
[ERROR] org.xml.sax.SAXParseException; lineNumber: 885; columnNumber: 41; cvc-complex-type.3.2.2: Attribute 'class' is not allowed to appear in element 'p'.
[ERROR] org.xml.sax.SAXParseException; lineNumber: 885; columnNumber: 41; cvc-complex-type.3.2.2: Attribute 'displayorder' is not allowed to appear in element 'p'.
[ERROR] org.xml.sax.SAXParseException; lineNumber: 888; columnNumber: 41; cvc-complex-type.3.2.2: Attribute 'class' is not allowed to appear in element 'p'.
[ERROR] org.xml.sax.SAXParseException; lineNumber: 888; columnNumber: 41; cvc-complex-type.3.2.2: Attribute 'displayorder' is not allowed to appear in element 'p'.
[ERROR] org.xml.sax.SAXParseException; lineNumber: 2830; columnNumber: 22; cvc-complex-type.3.2.2: Attribute 'autonum' is not allowed to appear in element 'table'.
[ERROR] org.xml.sax.SAXParseException; lineNumber: 3573; columnNumber: 109; cvc-complex-type.2.4.a: Invalid content was found starting with element 'biblio-tag'. One of '{break, bold, fixed-case, italic, monospace, num, overline, roman, sans-serif, sc, strike, underline, ruby, alternatives, inline-graphic, private-char, chem-struct, inline-formula, label, abbrev, index-term, index-term-range-end, milestone-end, milestone-start, named-content, styled-content, annotation, article-title, chapter-title, collab, collab-alternatives, comment, conf-acronym, conf-date, conf-loc, conf-name, conf-sponsor, data-title, date, date-in-citation, day, edition, email, elocation-id, etal, ext-link, fpage, gov, institution, institution-wrap, isbn, issn, issn-l, issue, issue-id, issue-part, issue-title, lpage, month, name, name-alternatives, object-id, page-range, part-title, patent, person-group, pub-id, publisher-loc, publisher-name, role, season, series, size, source, std, string-name, supplement, trans-source, trans-title, uri, version, volume, volume-id, volume-series, year, fn, target, "urn:iso:std:iso:30042:ed-1":entailedTerm, xref, sub, sup}' is expected.
[ERROR] org.xml.sax.SAXParseException; lineNumber: 3577; columnNumber: 130; cvc-complex-type.2.4.a: Invalid content was found starting with element 'biblio-tag'. One of '{break, bold, fixed-case, italic, monospace, num, overline, roman, sans-serif, sc, strike, underline, ruby, alternatives, inline-graphic, private-char, chem-struct, inline-formula, label, abbrev, index-term, index-term-range-end, milestone-end, milestone-start, named-content, styled-content, annotation, article-title, chapter-title, collab, collab-alternatives, comment, conf-acronym, conf-date, conf-loc, conf-name, conf-sponsor, data-title, date, date-in-citation, day, edition, email, elocation-id, etal, ext-link, fpage, gov, institution, institution-wrap, isbn, issn, issn-l, issue, issue-id, issue-part, issue-title, lpage, month, name, name-alternatives, object-id, page-range, part-title, patent, person-group, pub-id, publisher-loc, publisher-name, role, season, series, size, source, std, string-name, supplement, trans-source, trans-title, uri, version, volume, volume-id, volume-series, year, fn, target, "urn:iso:std:iso:30042:ed-1":entailedTerm, xref, sub, sup}' is expected.

[Intelligent2013]

In the output NISO XML found the missing tags for the terms references.

For instance, in the source BSI document (https://github.com/metanorma/mn-samples-bsi/tree/main/sources/bs-202000), in 00-introduction.adoc there are references to the terms process and standard:

SM is a through-life {{process}} that enables the drafting, development and maintenance of {{standard,standards}}. It promotes the intelligent selection and implementation of {{standard,standards}}, policy, {{process,processes}} and tools, and delivers:

The Presentation XML contains the plain text only:

<p id="_34c2a109-0b03-f25c-efc8-7ba7fd222919">SM is a through-life process that enables the drafting, development and maintenance of standards. It promotes the intelligent selection and implementation of standards, policy, processes and tools, and delivers:</p>

Previously, the Semantic part (which used as 'base' for the conversion) contained this markup:

<semantic__p id="semantic___34c2a109-0b03-f25c-efc8-7ba7fd222919">SM is a through-life <semantic__concept><semantic__refterm>process</semantic__refterm><semantic__renderterm>process</semantic__renderterm><semantic__xref target="semantic__term-process"/></semantic__concept> that enables the drafting, development and maintenance of <semantic__concept><semantic__refterm>standard</semantic__refterm><semantic__renderterm>standards</semantic__renderterm><semantic__xref target="semantic__term-standard"/></semantic__concept>. It promotes the intelligent selection and implementation of <semantic__concept><semantic__refterm>standard</semantic__refterm><semantic__renderterm>standards</semantic__renderterm><semantic__xref target="semantic__term-standard"/></semantic__concept>, policy, <semantic__concept><semantic__refterm>process</semantic__refterm><semantic__renderterm>processes</semantic__renderterm><semantic__xref target="semantic__term-process"/></semantic__concept> and tools, and delivers:</semantic__p>

I'll add the issue in the isodoc repository.

[opoudjis]

There is one remaining issue I will create after Monday's release: refactoring of footnotes. There may be other requests as @strogonoff starts using the refactored code, but after that issue, this task can be treated as completed.

[ronaldtse]

@opoudjis what's the plan for footnotes? Placing them in a unified container per document? Ideally we separate footnote content and applicable locality, in order to allow shared footnotes (and differentiation in locality).

[Intelligent2013]

mnconvert fails on the processing XML from the repository mn-samples-iso:
https://github.com/metanorma/mnconvert/actions/runs/13461620115/job/37618126610?pr=422
because XMLs in old format (iso-standard root tag). I'll run the XMLs re-generation.

[Intelligent2013]

version 2.9.5 (from Jan, 20):

<iso-standard xmlns="https://www.metanorma.org/ns/iso" type="presentation" version="2.9.5" schema-version="v1.4.1">

because Metanorma docker updated 30 days ago: https://hub.docker.com/r/metanorma/metanorma.

[Intelligent2013]

XSLT updated for old XML to STS XML conversion and validation.
Now, the issue metanorma/metanorma-plugin-lutaml#192 occurs.
https://github.com/metanorma/mnconvert/actions/runs/13475078544/job/37653234020?pr=422