Multiple effectiveTimes in Delta files - are they required?
Hi everyone
In the March 2020 US Edition we experienced for the first time a situation where multiple effectiveTimes were generated in the (Inferred Relationship) delta. This was a valid scenario because the Relationships were created as new in the Jan 2020 International Edition, and then correctly removed by the classifier in the March 2020 US Edition - thereby providing 2 records (with different effectiveTimes + Active flags only) in the US Edition Delta file. For example:
id | effectiveTime | active | moduleId | sourceId | destinationId | relationshipGroup | typeId | characteristicTypeId | modifierId |
12058138020 | 20200131 | 1 | 900000000000207008 | 127259005 | 128133004 | 0 | 116680003 | 900000000000011006 | 900000000000451002 |
12058138020 | 20200301 | 0 | 731000124108 | 127259005 | 128133004 | 0 | 116680003 | 900000000000011006 | 900000000000451002 |
The obvious solution is to simply code systems to take the record with the latest effectiveTime for that ID within the Delta - however as this issue has now highlighted, some vendors have already hard coded theirs to take all Delta records regardless. For now we have advised them that this is a valid (though unlikely) state, that will only happen in our only Managed Service Edition (US).
However the question now is, once we move to continuous delivery, do we still need to include multiple effectiveTimes in the Delta files that people will be able to produce for themselves, using the new Delta generation tool?
Or, are we happy with always taking the latest state in the Delta, regardless of how many (Published) changes there were in the meantime?
So, for example, if the following record is changed 3 times between January 2022 and July 2022:
id | effectiveTime | active | moduleId | sourceId | destinationId | relationshipGroup | typeId | characteristicTypeId | modifierId |
12058138020 | 20220131 | 1 | 900000000000207008 | 127259005 | 128133004 | 0 | 116680003 | 900000000000011006 | 900000000000451002 |
12058138020 | 20220431 | 1 | 900000000000207008 | 127259005 | 128133004 | 1 | 116680003 | 900000000000011006 | 900000000000451002 |
12058138020 | 20220731 | 0 | 900000000000207008 | 127259005 | 128133004 | 0 | 116680003 | 900000000000011006 | 900000000000451002 |
Should the Delta (for someone who is currently baselined on Jan 2022) generated for Jan 2022 - July 2022 include
a) ALL changes (including multiple effectiveTimes):
id | effectiveTime | active | moduleId | sourceId | destinationId | relationshipGroup | typeId | characteristicTypeId | modifierId |
12058138020 | 20220431 | 1 | 900000000000207008 | 127259005 | 128133004 | 1 | 116680003 | 900000000000011006 | 900000000000451002 |
12058138020 | 20220731 | 0 | 900000000000207008 | 127259005 | 128133004 | 0 | 116680003 | 900000000000011006 | 900000000000451002 |
b) ....Or should it only contain the latest state:
id | effectiveTime | active | moduleId | sourceId | destinationId | relationshipGroup | typeId | characteristicTypeId | modifierId |
12058138020 | 20220731 | 0 | 900000000000207008 | 127259005 | 128133004 | 0 | 116680003 | 900000000000011006 | 900000000000451002 |
When we originally all discussed this we were all in favour of a) (including all published changes in the interim period), as this follows the current RF2 spec, and allows people to generate the correct Full file even if they only ever take Delta's, and regardless of how long they wait inbetween each Delta.
However, this latest discussion has brought into question whether or not there are actually any valid use cases for this?
Do any users (certainly down at the vendor level) actually care about the interim history, or do they only ever want the latest state?
Those generating Delta's could just take the latest Full/Snapshot from the latest published release (and it will contain all changes), and those not using the Delta's will be unaffected.
One of the few use cases we can envisage is for historical ECL, but it sounds like the complexity involved here might require a rather different solution anyway, so that might not be a strong enough use case to require all vendors to change their systems to cope with multiple effectiveTimes?
So, if possible can everyone please post known use cases for including all changes (option a)) against this blog, and when we have our next TRAG call we can discuss in detail?
Thanks very much!
Andrew
Comments
I might have met and discussed SNOMED CT too many times with @Dion McMurtrie , because I agree with basically everything that he writes.
As (nowadays) a vendor I would also add that vendors like stable specifications, so it is worth the investment to implement the specifications. A change from option a to option b would imply a change of the specifications. All vendors that already have invested to implement the specifications correctly would then need to go through their implementations and do a risk assessment and potential update of their products. I would say that it would be very unfair to burden the vendors that have done a correct implementation of the specifications with this kind of work only to help the vendors that doesn’t follow the specifications.
(I assume that there are very few vendors that have done an incorrect implementation, because multiple effectiveTime is common in most extensions, so this behavior is not something new for most vendors.)
Thanks both - I'm also in complete agreement with everything you've said. I don't think that changing the spec for those few vendors who haven't catered for the possibility of multiple effectiveTimes is a sustainable approach going forward.
I also believe that encouraging those few vendors to update their systems correctly is worth the investment, given that the future is continuous delivery where multiple effectiveTimes will not only be possible, but a standard part of most releases.
If anyone has any further use cases, please post before Monday so we can finalise our conclusion on this - thanks,
Andrew
TL;DR definitely option (a), option (b) is a bad idea.
The RF2 types of Full, Delta and Snapshot have been long defined and well understood, I don't think we should be changing what they mean because of how some might be misusing them.
The Delta has been confused mostly because the International Edition has only ever released twice a year and only issues Delta files between these two release points. Therefore the international Delta files have only ever contained one effective time in the past, but this is just how the International Edition has used this file and isn't a restriction of the format.
From the spec
A delta release is a release type in which the release files contain only rows that represent component versions and reference set member versions created since the previous release date.
Notes
Each row in a delta release file represents either a new component or reference set member, or a change to an existing component or reference set member since the previous release date.
A delta release identifies differences between two versions of the same release package.
A delta release added to the previous full release is identical to the full release of the new version.
The previous release date, on which a delta release is based, is usually the date of the most recent previous release. However, that may not always be the case. For example, where interim releases are made between two major releases there may be a combined delta release covering a period since a previous major release.
I've often been bemused by the concern of "multiple effective times in delta files". This occurs regularly in extension releases when the international content is included. Extension publishers and their users are well and truly used to this and it should be a non-issue.
A special case of this is more than one version of a single component appearing in a Delta. This does necessarily happen, as you've discovered with the US extension. This has probably happened many times in the Australian extension and is perfectly normal and necessary to produce the correct NNF (previously DNF). The point is that the Delta itself cannot be processed by itself, it has to be applied to something.
The right way to use the Delta is to append it to a Full form of the data up to date with the version that the Delta is relative to, then calculate a Snapshot based on the module dependency reference set. If the user isn't prepared to do that they shouldn't use the Delta.
So in the extension cases, Deltas necessarily may have multiple effective times and possibly multiple versions of a component. If a Delta spans releases, aka a combined or "rollup" Delta, (like your January 2022-July 2022 example) it will necessarily contain multiple effective times and may contain multiple versions of a single component depending upon the content and changes.
I'd strongly advise against option (b). At the moment we know that a Delta contains all the versions between two versions of the Full form of an edition. What (b) proposes is sort of a Snapshot Delta, which buries history.
Finally this once more raises that we're missing from the release format, specifically the Delta files, a key piece of metadata - which version the Delta has been calculated from. This is critical information to correctly process the Delta, and currently is published as "the last release", which is usually predictable but can change as we've found with interim releases. Having that worked into machine readable metadata adds to safe processing of these files. We really should just patch this by adding to the Delta file naming convention to include a "from" version as well as a "to" version. This is a pretty easy fix.