Athento is a complex product that has been under development and evolution for several years. Until now, Athento’s technology stack was based on Django 1.11 and Python 2.7, versions of these technologies that we needed to renew. In addition to the support issues we faced when using technologies that were already a few years old, we had limitations in terms of the functionality we could offer to our customers: many of the packages and libraries we wanted to use to implement cool features required newer versions of these technologies. Although we often managed to find a workaround, the time had come: we had to upgrade.
Our starting point was a product in these technologies, with a hundred installations. We had to upgrade the technological stack with the least impact on the more than 10K users who work with Athento daily. More than 230,000 lines of code to migrate, 80+ Django apps, and an additional challenge: to continue providing support to our customers, applying fixes, and implementing some new features in the old version of the product.
Our first steps were to evaluate the strategy to follow and the upgrades we were going to apply, and in what order. None of us had done an upgrade of this caliber before. It was clear to us that any plan we might devise at this point was going to be subject to change and that we needed to be flexible and patient in trying new ideas. Our first question was: Python first or Django? or both at the same time?
We solved this question by studying compatibilities. We found that Django 1.11 supported Python versions up to 3.7. Having Python 3.7 allowed us to increase our Django version up to 3.2. While not the most modern versions of these technologies, they were a vast improvement to the versions we had with probably less effort than it would take to migrate to more modern versions. In addition, the candidate versions of Python and Django already had enough experience and background to give us peace of mind. So we decided to migrate the entire project to Python 3.7 with Django 1.11 and, in a second step, to migrate from Django 1.11 to Django 3.2.
Upgrading the Python version from 2.7 to 3.7 involved two main tasks: updating the code and updating all the pip requirements of the project.
To update the code we had to do a complete analysis of Athento with the help of tools like 2to3 that basically, analyze the code and suggest the syntax changes required to convert from Python 2 to Python 3.
Analyzing the results returned by the tool, we decided not to opt for automatic conversion, but to get the suggested changes and apply them if they made sense. The reason for this decision is that some changes did not seem reasonable to us: function calls such as print added a double parenthesis or the results of certain iterators or Django QuerySets were converted to lists (this sometimes means a loss of performance if the evaluation was lazy). As Athento consists of more than 2000 Python 2 source files, we divided the task between the whole team to finish it as soon as possible. The most common changes we had to make to the code were:
Using external packages that we installed with pip presented another challenge. First of all, certain packages are not available in Python 3, or the versions we had are not compatible. Also, since Athento is a complex product, it can happen that upgrading package versions results in some incompatibility between them, which we had to work around. The lessons learned are the following:
Despite all our efforts and the changes we made, there were several packages that, although compatible with Python 3.7, were still incompatible with Django 1.11, and we had to plan a migration from Django earlier than expected. It is also worth pointing out a flaw in our procedure that we could have targeted differently: when most of the requirements were installed, part of the team continued fixing the rest of the requirements while others were dedicated to testing some of the functionality that could already work. Perhaps it would have been more appropriate to start the testing phase once the complete migration to the new version of Django was finished.
The Django upgrade, in general, was lighter than the Python upgrade. Django-related requirements had to be reviewed, again, as some of them were already incorporated natively and others required updating.
A couple of cases that created some uncertainty were the Hijack package, used to implement the user login impersonation functionality and others that served to extend the Django administration functionality. These functionalities are very important within Athento as they are widely used by our project and support teams to quickly debug possible bugs or facilitate the implementation of the platform for our customers. Finally, we found a suitable version of Hijack and chose to use Grapelli as the administration system.
Once the code problems were solved, the versions were updated, and the project was started, testing had to begin. In this first phase, the whole team got involved, testing each and every feature of Athento. Although now the syntax of the project was correct with Python 3, semantically there are things that have changed and did not work well.
Most of the encode and decode problems on strings that we had changed during the code migration, caused unwanted effects: the conversion from bytes to str and vice versa sometimes returned the wrong type of data and certain sections of the code had to be rewritten. Related to the above, opening files in binary or text mode causes several problems. We had to study well the io package of Python 3.7 and rewrite part of the code using the BytesIO, TextIO, and TextIOWrapper classes, among others.
It should also be noted that while the whole team was doing tests and corrections on this new version of Athento, we had to keep adding functionality and providing support to users of the previous version, in Python 2. This was a challenge for the day-to-day work of the development team since we had to prepare the database and our entire development environment to support both versions, and sometimes we encountered certain problems. One of them was the Django cache, which could cause us problems when switching from one version to another. We solved this by clearing this cache after each change:
from django.core.cache import cache; cache.clear()
Another common problem is bringing to the new version of Python 3 all the hotfixes or new features that are added to the Python 3 project. To do this we apply the following procedure: add to git a remote of the Python 2 version of the project, place ourselves in the main branch in the Python 3 project, and merge the changes. This is a costly process, as there are often some conflicts or code written that is not compatible with Python 3 and needs to be rewritten. This process is now part of our workflow until Athento’s Python 2 version is deprecated and all our customers have migrated to the new version.
$ git remote add firstname.lastname@example.org:/.git
$ git fetch
$ git switch -c merge/merge-py2-py3
$ git merge /fast-track
Our next objective, once all the fixes had been applied, was to install Athento on a new clean server. The idea was to make it easier for system colleagues to install and configure environments in the new version and, at the same time, to allow support and project teams to test the product functionalities in the updated version.
Based on the existing for the previous version, we performed a review and update of the required OS level package. The most significant change is that many of the installed packages have different versions depending on whether Python2 or Python3 is used. As an example, we can take the zbar package, where we had to change from python-zbar to python3-zbar.
As mentioned above, deploying the new version of Athento allowed the rest of the teams within the company to test the product in an environment as close as possible to production. For a few days, we iterated between tests and fixes, following the established testing protocol.
Once Athento was working correctly in its Python 3 version, and having already placed this version in several new servers for different customers, we had to proceed with the upgrade of servers that have Athento with Python 2.7 and Django 1.11 to the new versions. To do so, we established the following protocol with the help of the company’s systems team:
Migrations are usually generated by Django and have to be applied because some models may have changed their implementation, especially those models handled by external dependencies of the project. We have encountered different problems in this regard. For example, when generating new migrations and applying them, we get errors because the models already exist (remember, we must use the database already installed so that the client does not lose any data). Finally, we have proposed a couple of strategies that will have to be evaluated depending on the state of each server. On the one hand, we can use the -fake flag when applying migrations. In this way, they are registered as applied, although they do not change the data model stored in the database. Then, an application testing process is performed and any inconsistent models are fixed. Another alternative is to generate migrations from the old version of the product and copy them to the new one. We then generate them again in the new version, so they will be added after the previous ones and leave the database models consistent. The problem with this approach is that migrations generated in the Python 2 version of the product may contain code that is not executable in Python 3 and has to be corrected by hand. Following one strategy or the other is going to depend heavily on the individual servers.
After all this adventure and learning process we continue to update our customers on the new version of the product, on which we will be adding new functionality. As a product team, we will continue to update this new version with all the fixes that are added to the previous version. In addition, we will continue to collaborate with the project, support, and systems teams in bringing more and more customers to this version of the product. While all this is happening, we will be adding new features to Athento to make it a more complete, useful, fast, and reliable tool. Python 3 will also allow us to delve deeper into technologies such as Machine Learning, another of the great incentives we have had to undertake this migration.
We hope that this experience will help the community and all those who are about to embark on a journey like the one we have undertaken.