Opened 10 years ago

Closed 9 years ago

Last modified 9 years ago

#152 closed enhancement (released)

Test and fix extensibility of current schemas

Reported by: ibboard Owned by: ibboard
Priority: major Milestone: WarFoundry 0.1
Component: WarFoundry-API Version: Unknown or N/A
Keywords: schema extensability Cc:
Blocked By: Blocking: #150

Description (last modified by ibboard)

ticket:150 requires an extension to the schema. Ideally, WarFoundry should be both backwards and forwards compatible (i.e. newer apps can use older files but not take advantage of new features because the files don't define it, but older apps can also use newer files and just ignore the newer features that they don't understand). This hasn't been properly tested, though, and I suspect the schemas might not be correctly defined to support this as they stand.

Attachments (1)

Empire-Extended.race (777 bytes) - added by ibboard 9 years ago.
Test Empire.race file with attributes from extra namespaces

Download all attachments as: .zip

Change History (35)

comment:1 Changed 10 years ago by ibboard

Blocking: 175 added

comment:2 Changed 10 years ago by ibboard

Description: modified (diff)
Owner: set to ibboard
Status: newaccepted

comment:3 Changed 10 years ago by ibboard

Summary: Test extensibility of current schemasTest and fix extensibility of current schemas

Extending schemas causes file load to file. Need to investigate how to get C# to recognise the anyAttribute and any tags in the schema (details)

comment:4 Changed 10 years ago by ibboard

Can't seem to get rid of

XmlSchema error: Attribute declaration was not found for http://ibboard.co.uk/warfoundry/attrib:extraAttrib XML

even when schema has <anyAttribute> and the schema for the extra attribute is loaded in the namespace manager and the schema cache.

comment:5 Changed 10 years ago by ibboard

Ooops - helps if the extra schema is actually copied to the output folder (or if Mono would say "can't find file" rather than ignoring the fact you've added a non-existant schema).

comment:6 Changed 10 years ago by ibboard

(In r277) Re #152: Test and fix extensibility of current schemas

  • Add extensibility to Race schema

Also:

  • Code tidy for WarFoundryXmlFactoryUtils

comment:7 Changed 10 years ago by ibboard

Backwards compatibility is looking possible through careful definition of new attributes with default values or new elements that are optional, but forwards compatibility needs more exploration as it will currently fail validation because the old app won't understand the new namespace.

comment:8 Changed 10 years ago by ibboard

Blocking: 175 removed

Remove #175 because that level of backward/forward compatibility is too much work for an initial beta version.

Solution for forward compatibility is XSLTs to remove things we don't understand and post-cleansing validation instead of load-time validation.

comment:9 Changed 10 years ago by ibboard

(In r304) Re #152: Test and fix extensibility of current schemas

  • Add "any" elements and "anyAttributes" to schemas where appropriate

Still need to handle cleansing with XSLT

comment:10 Changed 10 years ago by ibboard

(In r305) Re #152: Test and fix extensibility of current schemas

  • Fix use of <xs:any> by defining <core:extension> tag
  • Remove unnecessary min/maxOccurs=1 attributes

comment:11 Changed 10 years ago by ibboard

released: no
Version: N/A

XSLT method suggested in comment:8 may work, but might need extra thought about attributes outside our namespace (all extensions).

comment:12 Changed 9 years ago by ibboard

Alternatives to XSD may provide more useful functionality cleanly, but may not be supported by .Net: http://www.xfront.com/backward-forward-compatibility/

comment:13 Changed 9 years ago by ibboard

Better solution, based on discussion with a work colleague: don't try for backwards and forwards compatibility on a single schema, instead have a schema for each version with later versions adding on to and re-using the earlier ones. No parsing/prep needed, all unrecognised elements from later schemas should be ignored automatically and namespacing means that the XML XPath queries should still work fine.

If it works, dev versions of WarFoundry will be subject to change but we'll have a lock-down at each versioned release.

Changed 9 years ago by ibboard

Attachment: Empire-Extended.race added

Test Empire.race file with attributes from extra namespaces

comment:14 Changed 9 years ago by ibboard

The previous attachment adds

xmlns:ext="http://ibboard.co.uk/warfoundry/attrib" ext:extraAttrib="en"

to the standard test race, but the schema for http://ibboard.co.uk/warfoundry/attrib isn't defined, so we get

Problem validating against schema for WarFoundry data: XmlSchema error: Attribute declaration was not found for http://ibboard.co.uk/warfoundry/attrib:extraAttrib XML  Line 2, Position 155.

There will be a fix somewhere, but this effectively limits our extensibility until we can handle (ignore) any schemas that we don't know but validate against those that we do know. Method for post-validation may help.

comment:15 Changed 9 years ago by ibboard

(In r496) Re #152: Test and fix extensibility of current schemas

  • Split out some reusable parts of Race schema
  • Replace use of "xs:all" with "xs:sequence"
  • Replace use of "core:extension" with standard "xs:any"
  • Start to use xs:key and xs:keyref to get more built-in validation

comment:16 Changed 9 years ago by ibboard

(In r499) Re #152: Test and fix extensibility of current schemas

  • Refactor army schema

comment:17 Changed 9 years ago by ibboard

(In r500) Re #152: Test and fix extensibility of current schemas

  • Remove junk text used to test breaking the schema

comment:18 Changed 9 years ago by ibboard

(In r502) Re #152: Test and fix extensibility of current schemas

  • Fix complex type for unit - should be named, not IDed

comment:19 Changed 9 years ago by ibboard

(In r503) Re #152: Test and fix extensibility of current schemas

  • Add unit test for parsing extended document without XSD for namespace

comment:20 Changed 9 years ago by ibboard

Pre or post-load validation doesn't make a difference. Validation process seems to need all schemas or none (Mono source code).

I've not yet worked out how to either do partial validation (only validate against the schemas we have) or use an XSLT or similar to remove all unknown elements/attributes, but retain any elements/attributes that might be used by extensions that supply their own schemas (XSLT is a fixed document, as far as I've seen).

comment:21 Changed 9 years ago by ibboard

Recent "fixes" to move from xs:any to xs:sequence seem to work fine in Mono, but not in .Net. Latest WinForms app currently complains about ambiguity of ##any with race:unitAbilities.

comment:22 Changed 9 years ago by ibboard

(In r505) Re #152: Test and fix extensibility of current schemas

  • Make xs:anyAttribute and xs:any do lax parsing so that we don't get schema errors
  • Move Army schema away from core:extension to straight xs:any

Needs more testing

comment:23 Changed 9 years ago by ibboard

(In r506) Re #152: Test and fix extensibility of current schemas

  • Update to newer racex file for testing that matches our sequence-ordered schema

comment:24 Changed 9 years ago by ibboard

(In r510) Re #152: Test and fix extensibility of current schemas

  • Disable UPA (Unique Particle Attribution) check to get same behaviour in .Net as Mono and as new XML Schema 1.1 draft
  • Unit test with extended schema now passes. More tests needed

comment:25 Changed 9 years ago by ibboard

(In r516) Re #152: Test and fix extensibility of current schemas

  • Move (optional) categories overrides before unit list so that ordering makes a bit more sense
  • Make anyAttribute in Categories schema do lax processing

comment:26 Changed 9 years ago by ibboard

(In r517) Re #152: Test and fix extensibility of current schemas

  • Allow any element within category (lax processing)

comment:27 Changed 9 years ago by ibboard

(In r518) Re #152: Test and fix extensibility of current schemas

  • Add lots of extra attributes (from non-existant schema and from existing schema but not defined in schema)
  • Add category overrides

comment:28 Changed 9 years ago by ibboard

(In r519) Re #152: Test and fix extensibility of current schemas

  • Make race and core schemas fully extensible and forward compatible

comment:29 Changed 9 years ago by ibboard

(In r520) Re #152: Test and fix extensibility of current schemas

  • Simplify test data and test all possible tags

comment:30 Changed 9 years ago by ibboard

(In r521) Re #152: Test and fix extensibility of current schemas

  • Test system and army files
  • Test multiple unknown children in racex file

comment:31 Changed 9 years ago by ibboard

Resolution: fixed
Status: acceptedclosed

(In r522) Fixes #152: Test and fix extensibility of current schemas

  • Complete forward and backward compatibility of schemas (according to testing)

comment:32 Changed 9 years ago by ibboard

(In r525) Re #152: Test and fix extensibility of current schemas

  • Fix typo - "processContents" (plural, not singular)

no-open-ticket

comment:33 Changed 9 years ago by ibboard

released: noyes

Release as part of v0.1b7

comment:34 Changed 9 years ago by ibboard

Resolution: fixedreleased

Move to new "released" resolution for v0.1b7

Note: See TracTickets for help on using tickets.