“Long-term” – sustainable, scalable and maintainable

Part three: the implementation of sustainable and maintainable developments

Launching a long-term support development project is good. Putting in place the means, methods, and organisation to carry it out is even better. But the release still needs to be a success, and it still has to fit in, as well as possible, with future release and maintenance cycles.That’s what the last article in our series on iTop 2.7 Long-Term Support is about.

That’s what the last article in our series on iTop 2.7 Long-Term Support is about.

Regular and incremental deployment

An LTS version needs to be of the highest possible quality! We wanted to test the version as soon as possible in the “real world”  to get as much feedback as possible: stabilisation, corner case(s), confirmation of the choices made, etc.

Alpha, Beta, RC

We have released beta versions of our product in the past (e.g. 2.5.0-beta, 2.4.0-beta, 2.3.0-beta), but never more than one per version. These beta versions were generated at the end of upgrade developments. They were followed by a bug fixing phase and then the release of the final version.

For the 2.7.0 version which, as we saw in our previous article, introduces several major changes, we wanted to have more intermediate versions, both for in-house testing and for the community. We used fairly common industry naming conventions to define different types of milestone:

• alpha: versions that haven’t been finalised, but a number of features can be tested

• beta: a fully functioning version that has not yet been fully tested (in line with our beta definition for previous versions)
• release candidate (RC): a version that is ready for release, but we want to continue testing

We generated all these intermediate versions for 2.7.0:

• 2.7.0-alpha1 on 22/10/2019
• 2.7.0-beta on 18/12/2019
• 2.7.0-beta2 on 29/01/2020
• 2.7.0-rc on 24/03/2020
• on the same commit 2.7.0-rc2 and 2.7.0 on 25/03/2020

We released two beta versions on SourceForge to gather feedback from the iTop community:

• beta on 18/12/2019
• beta2 on 29/01/2020

The few fixes included in the beta were useful for the subsequent deployment on iTop Combodo.

Deployment on iTop Combodo

Combodo has a customised version of iTop for its own use and for customer support. This instance is nicknamed “iTop Combodo”. We decided to deploy iTop 2.7.0 beta versions on this instance to make sure we detected:

• bugs during use: all the features in this instance (Helpdesk, exports, datasync, etc.) are in constant use in the company, increasing the likelihood of problems being detected quickly!

• migration issues: iTop Combodo contains a large number of customisations, using a very wide range of iTop APIs

The first rollouts of version 2.7.0 on Combodo iTop took place two months before the release of the final version: the 2.7.0 beta version was deployed in January 2020. Several subsequent updates enabled sign off on the beta2 and RC versions. This enabled us to improve the product and above all to reassure ourselves about its quality!

Beta versions of iTop 3.0.0

This first successful experience prompted us to make even greater use of the pre-release mechanism for iTop 3.0.0. We created:

• an alpha version (public)
• 8 beta versions, 4 of which were public
• the installation of certain beta versions on iTop Combodo between July 2021 and the final version’s release in January 2022: beta1 (01/07), beta2 (13/07), beta3 (31/08), beta5 (04/10), beta7 (24/11), final (26/01)

And to further involve the Community in the process, a forum for beta versions was created in March 2020 just before the publication of the first 3.0.0 beta.

The forum was very well attended and provided a wealth of feedback from our community, Combodo partners and customers. Thank you all for your involvement! 🤗

Given our commitment to communicate and share information with our users, we added the following to the release plan:

• dedicated announcements on Sourceforge
• a list of known issues on our Wiki, or updates under development

However, the work on version 2.7.0 does not stop with the release of the final version. A new phase begins in parallel with release 3.0. Users need support to update their application.

Migrations

A version that has a development cycle of over a year contains a lot of changes (see Internal “Long-term” organisation for a top-quality LTS version) and the new features and changes must therefore be documented and communicated as well as possible, both to customers and to the internal staff at Combodo.

What we did

Migration Notes are a very important document for users and the support department: this is the information that will be used to prepare and perform client migrations.
Writing this wiki page is therefore a very important part of the release process. The steps involved are as follows:

1. The document is completed as development progresses, both by the R&D team and the Product team
2.   Before the first beta version is published, the document is structured by the two teams to ensure maximum readability
3.   The migration is tested during iTop Combodo updates with the beta and RC versions
4.   During version acceptance (at the RC stage), migration tests are performed on test instances containing representative customisations

We followed the same process as for previous versions to create the migration notes for version 2.7.0. But it has to be said that the usual process was inadequate this time.

Feedback

In fact, in the months that followed the publication of the final version 2.7.0, with the successive migrations we noticed:

• Some omissions in the migration notes, detected in specific cases that had not been covered in our migration tests during the RC: and therefore some unpleasant surprises to deal with as a matter of urgency.
• Migration notes that are becoming difficult to understand:
  • a large number of points to check, some of which are very technical and poorly explained (in particular: when and why to make certain modifications)
  • unclear task allocation between the machine administrator, iTop administrator, and iTop customisation developers (extensions, PHP code included in iTop Designer)

• Paradigm shift: while with many previous versions of iTop we could simply update and refer to the migration notes in the event of problems, with version 2.7.0 it’s now imperative to prepare the migration before carrying it out !

In addition, our support and consultancy teams received inadequate training and support during early client migrations due to a lack of anticipation of the scale of these migrations.

Result : the difficulties encountered during the initial migrations to iTop 2.7 had a negative impact on our clients’ perception of the product.

What has been put in place since

It was clear things had to change! For the development of the next version, 3.0.0, we made the following choices.

In-house training and skills transfer

• Progress presentations were made throughout the development of version 3.0.0, as well as presentations on the key features
• More in-depth training sessions were organised for our staff
• Migration Assistance :
  • The Product team gave each person in the Support and Consulting teams individual training on a customer’s actual migration
  • Migrations were also carried out with our partners

Tools

A major innovation at Combodo was the addition of a “migration audit” tool. The rules-based tool checks for a regular expression or XPath on:

• the XML generated by ITSM Designer (datamodel modification in XML, overloaded methods, snippets, etc.)
• the PHP code for the modules installed on this instance

Each rule is linked to a version of iTop.

Example rule for iTop 2.7.0:

/itop_design/menus/menu[parent = "AdminTools"]

If you want to check for errors for a migration to iTop >= 2.7.0, and this rule is triggered in the client licence, it will raise an error.

This set of rules is entered simply and quickly during development, and validated at the end of the version: exactly like the migration notes mentioned earlier.
The difference here is that it becomes easy to plan a customer migration, by listing all the problem points in the form of a report.
This tool will also be a great help when migrating several major versions (from 2.6.* to 3.0.*, for example).

Documentation

For version 3.0.0 we created a new “developer” migration notes page : Migrate an Extension to 3.0

• These migration notes for extension developers are divided into several clear categories: iTop XML, PHP APIs, Twig, JS APIs, and CSS APIs.
• In each of these categories, a dedicated chapter lists API deletions (“breaking changes”), and another lists the deprecations.

Also an extension compatibility page has been developed: Extensions compatibility with iTop 3.0 to help alpha and beta testers, and so that community users can migrate more easily after delivery of the final version

Prospects

Delivery process

The delivery process for an iTop version (Professional, Essential, Community) is a complex operation: several departments are involved (R&D, Products, Marketing), and a large number of tasks need to be carried out in a specific order!

To help monitor the process, our internal iTop instance was adapted to link a list of new Work Order objects to each release. The orders contain mainly:

• a designated team 
• a duration,
• any dependencies on other Work Order objects
• and of course a status.

In this way, everyone can see the planned timetable and monitor progress.

The R&D team needs to perform several dozen tasks, some of which should be automated. We narrowed the list down by developing:

• scripts ( .make directory in our repository)

• PHPUnit tests informing us of forgotten tasks ( test/integration/ directory in our repository)

This automation and automatic validation work should continue.

Minor versions and upgrades

Until 2.7.0, our rule was to only include upgrades in major versions (x.y.0). But a few months after the release of 2.7.0, we agreed that we should also integrate behavioural changes in the subsequent corrective versions. This was the case with 2.7.1 and 2.7.2. Version 2.7.0 introduced many changes,  so perhaps this was inevitable?

So, which version should we switch to LTS? Should we have waited for a 2.7.2, instead of declaring LTS status as early as 2.7.0?
Also, concerning the API deprecation/deletion policy (you will remember we talked about it in the 2nd article): will there be no deprecated APIs in the next iTop LTS version?

These questions remain open and in discussion with our customers and partners.

Take-up of iTop versions

Today, two and a half years after the release of version 2.7.0, around 80% of our customers are in a maintained version (in the sense of bug fixes) and over 90% of our customers are running the last three versions.

In comparison, in 2019 the last 2 major versions (2.6 and 2.5) accounted for only 64% of our customers’ installations, and versions 2.4 and 2.3 still accounted for 34%.

Conclusion

If we were to sum up this long-term support experience in one sentence, hindsight shows that the LTS gamble has worked. 

Take-up is undeniable, despite the difficulties encountered following the initial migrations and the risks of losing our users’ confidence. In addition, combining short-term and long-term support versions has definitely helped Combodo improve its development and release management, as well as its internal organisation.

Let’s build on what we’ve learned as we move on to the next LTS 3.X. To be continued…