“Long-term” – sustainable, scalable and maintainable

Part three: the implementation of sustainable and maintainable developments

Launching a long-term support development project is good. Putting in place the means, methods, and organisation to carry it out is even better. But the release still needs to be a success, and it still has to fit in, as well as possible, with future release and maintenance cycles.That’s what the last article in our series on iTop 2.7 Long-Term Support is about.

That’s what the last article in our series on iTop 2.7 Long-Term Support is about.

Regular and incremental deployment

An LTS version needs to be of the highest possible quality! We wanted to test the version as soon as possible in the “real world”  to get as much feedback as possible: stabilisation, corner case(s), confirmation of the choices made, etc.

Alpha, Beta, RC

We have released beta versions of our product in the past (e.g. 2.5.0-beta, 2.4.0-beta, 2.3.0-beta), but never more than one per version. These beta versions were generated at the end of upgrade developments. They were followed by a bug fixing phase and then the release of the final version.

For the 2.7.0 version which, as we saw in our previous article, introduces several major changes, we wanted to have more intermediate versions, both for in-house testing and for the community. We used fairly common industry naming conventions to define different types of milestone:

• Alpha: An unfinished version where many features are already available for testing.

• Beta: A fully working version that still requires thorough testing (just like in our earlier beta releases).
• Release candidate (RC): A version considered ready for release, while we keep testing to ensure stability.

We generated all these intermediate versions for 2.7.0:

• 2.7.0-alpha1 on 22/10/2019
• 2.7.0-beta on 18/12/2019
• 2.7.0-beta2 on 29/01/2020
• 2.7.0-rc on 24/03/2020
• on the same commit 2.7.0-rc2 and 2.7.0 on 25/03/2020

We released two beta versions on SourceForge to gather feedback from the iTop community:

• beta on 18/12/2019
• beta2 on 29/01/2020

The few fixes included in the beta were useful for the subsequent deployment on iTop Combodo.

Deployment on iTop Combodo

Combodo has a customised version of iTop for its own use and for customer support. This instance is nicknamed “iTop Combodo”. We decided to deploy iTop 2.7.0 beta versions on this instance to make sure we detected:

• bugs during use: all the features in this instance (Helpdesk, exports, datasync, etc.) are in constant use in the company, increasing the likelihood of problems being detected quickly!

• migration issues: iTop Combodo contains a large number of customisations, using a very wide range of iTop APIs

The first rollouts of version 2.7.0 on Combodo iTop took place two months before the release of the final version: the 2.7.0 beta version went live in January 2020. Several subsequent updates enabled sign off on the beta2 and RC versions. This enabled us to improve the product and above all to reassure ourselves about its quality!

Beta versions of iTop 3.0.0

This first successful experience prompted us to make even greater use of the pre-release mechanism for iTop 3.0.0. We created:

• an alpha version (public)
• 8 beta versions, 4 of which were public
• the installation of certain beta versions on iTop Combodo between July 2021 and the final version’s release in January 2022: beta1 (01/07), beta2 (13/07), beta3 (31/08), beta5 (04/10), beta7 (24/11), final (26/01)

And to further involve the Community in the process, we created a forum for beta versions in March 2020 just before the publication of the first 3.0.0 beta.

The forum brought together an active community and produced valuable feedback from our community, Combodo partners and customers. Thank you all for your involvement! 🤗

Given our commitment to communicate and share information with our users, we added the following to the release plan:

• dedicated announcements on Sourceforge
• a list of known issues on our Wiki, or updates under development

However, the work on version 2.7.0 does not stop with the release of the final version. A new phase begins in parallel with release 3.0. Users need support to update their application.

Migrations

A version that has a development cycle of over a year contains a lot of changes, and the new features and changes must therefore be documented and communicated as well as possible, both to customers and to the internal staff at Combodo.

What we did

Migration Notes are a very important document for users and the support department: this information will serve as the basis to prepare and perform client migrations.
Writing this wiki page is therefore a very important part of the release process. The steps involved are as follows:

1. The R&D team and the Product team complete the document as development progresses.
2. Before publishing the first beta version, the two teams structure the document to ensure maximum readability.
3. The team tests the migration during iTop Combodo updates with the beta and RC versions.
4. During version acceptance (at the RC stage), the team performs migration tests on test instances containing representative customisations.

We followed the same process as for previous versions to create the migration notes for version 2.7.0. This time, the usual process proved inadequate.

Feedback

In fact, in the months that followed the publication of the final version 2.7.0, with the successive migrations we noticed:

• We detected some omissions in the migration notes in specific cases that our RC migration tests had not covered: and therefore some unpleasant surprises to deal with as a matter of urgency.
• Migration notes that are becoming difficult to understand:
  • a large number of points to check, some of which are very technical and poorly explained (in particular: when and why to make certain modifications)
  • unclear task allocation between the machine administrator, iTop administrator, and iTop customisation developers (extensions, PHP code included in iTop Designer)

• Paradigm shift: while with many previous versions of iTop we could simply update and refer to the migration notes in the event of problems, with version 2.7.0 it’s now imperative to prepare the migration before carrying it out !

In addition, our support and consultancy teams received inadequate training and support during early client migrations due to a lack of anticipation of the scale of these migrations.

Result : the difficulties encountered during the initial migrations to iTop 2.7 had a negative impact on our clients’ perception of the product.

What has been put in place since

It was clear things had to change! For the development of the next version, 3.0.0, we made the following choices.

In-house training and skills transfer

• The team presented progress throughout the development of version 3.0.0, as well as presentations on the key features
• We organised more in-depth training sessions for our staff
• Migration Assistance :
  • The Product team gave each person in the Support and Consulting teams individual training on a customer’s actual migration
  • The team also performed migrations alongside our partners

Tools

A major innovation at Combodo was the addition of a “migration audit” tool. The rules-based tool checks for a regular expression or XPath on:

• the XML generated by ITSM Designer (datamodel modification in XML, overloaded methods, snippets, etc.)
• the PHP code for the modules installed on this instance

Each rule is linked to a version of iTop.

Example rule for iTop 2.7.0:

/itop_design/menus/menu[parent = "AdminTools"]

If you want to check for errors for a migration to iTop >= 2.7.0, and this rule is triggered in the client licence, it will raise an error.

This set of rules is entered simply and quickly during development, and validated at the end of the version: exactly like the migration notes mentioned earlier.
The difference here is that it becomes easy to plan a customer migration, by listing all the problem points in the form of a report.
This tool will also be a great help when migrating several major versions (from 2.6.* to 3.0.*, for example).

Documentation

For version 3.0.0 we created a new “developer” migration notes page : Migrate an Extension to 3.0

• We divide these migration notes for extension developers into several clear categories: iTop XML, PHP APIs, Twig, JS APIs, and CSS APIs.
• In each of these categories, a dedicated chapter lists API deletions (“breaking changes”), and another lists the deprecations.

We also developed an extension compatibility page: Extensions compatibility with iTop 3.0 to help alpha and beta testers, and so that community users can migrate more easily after delivery of the final version

Prospects

Delivery process

The delivery process for an iTop version (Professional, Essential, Community) is a complex operation: several departments are involved (R&D, Products, Marketing), and we need to carry out a large number of tasks in a specific order!

We adapted our internal iTop instance to help monitor the process in order to to link a list of new Work Order objects to each release. The orders contain mainly:

• a designated team 
• a duration,
• any dependencies on other Work Order objects
• and of course a status.

In this way, everyone can see the planned timetable and monitor progress.

The R&D team needs to perform several dozen tasks, some of which should be automated. We narrowed the list down by developing:

• scripts ( .make directory in our repository)

• PHPUnit tests informing us of forgotten tasks ( test/integration/ directory in our repository)

This automation and automatic validation work should continue.

Minor versions and upgrades

Until 2.7.0, our rule was to only include upgrades in major versions (x.y.0). But a few months after the release of 2.7.0, we agreed that we should also integrate behavioural changes in the subsequent corrective versions. This was the case with 2.7.1 and 2.7.2. Version 2.7.0 introduced many changes,  so perhaps this was inevitable?

So, which version should we switch to LTS? Should we have waited for a 2.7.2, instead of declaring LTS status as early as 2.7.0?

These questions remain open and in discussion with our customers and partners.

Take-up of iTop versions

Today, two and a half years after the release of version 2.7.0, around 80% of our customers are in a maintained version (in the sense of bug fixes) and over 90% of our customers are running the last three versions.

In comparison, in 2019 the last 2 major versions (2.6 and 2.5) accounted for only 64% of our customers’ installations, and versions 2.4 and 2.3 still accounted for 34%.

Conclusion

If we were to sum up this long-term support experience in one sentence, hindsight shows that the LTS gamble has worked. 

Take-up is undeniable, despite the difficulties encountered following the initial migrations and the risks of losing our users’ confidence. In addition, combining short-term and long-term support versions has definitely helped Combodo improve its development and release management, as well as its internal organisation.

Let’s build on what we’ve learned as we move on to the next LTS 3.X. To be continued…