Upgrade error form 21.02 to 21.05, database upgrade

Hello,

Upgrading from 21.02 to 21.05 with wazo-upgrade cli command.
Two errors happened:

2021-05-03 12:21:21,367 [23754] (INFO) (wazo_call_logd.main_migrate_db): Removing old tables...
2021-05-03 12:21:21,422 [23754] (CRITICAL) (root): ERREUR:  ne peut pas supprimer les objets désirés car d'autres objets en dépendent
DETAIL:  contrainte cel_call_log_id_fkey sur table cel dépend de table call_log
HINT:  Utilisez DROP ... CASCADE pour supprimer aussi les objets dépendants.
Traceback (most recent call last):
  File "/bin/wazo-call-logd-migrate-db", line 11, in <module>
    load_entry_point('wazo-call-logd==1.2', 'console_scripts', 'wazo-call-logd-migrate-db')()
  File "/usr/lib/python3/dist-packages/wazo_call_logd/main_migrate_db.py", line 77, in main
    migrate_call_log_tables(config, options.get('max_entries'))
  File "/usr/lib/python3/dist-packages/wazo_call_logd/main_migrate_db.py", line 222, in migrate_call_log_tables
    cel_cur.execute(query)
psycopg2.InternalError: ERREUR:  ne peut pas supprimer les objets désirés car d'autres objets en dépendent
DETAIL:  contrainte cel_call_log_id_fkey sur table cel dépend de table call_log
HINT:  Utilisez DROP ... CASCADE pour supprimer aussi les objets dépendants.

Enabling Wazo services...

Executing upgrade script /usr/share/wazo-upgrade/post-start.d/45-confd-sync.sh...
2021-05-03 12:22:02,042 [24631] (CRITICAL) (root): (psycopg2.ProgrammingError) ERREUR:  la colonne tenant.slug n'existe pas
LINE 1: SELECT tenant.uuid AS tenant_uuid, tenant.slug AS tenant_slu...
                                           ^
 [SQL: 'SELECT tenant.uuid AS tenant_uuid, tenant.slug AS tenant_slug, tenant.sip_templates_generated AS tenant_sip_templates_generated, tenant.global_sip_template_uuid AS tenant_global_sip_template_uuid, tenant.webrtc_sip_template_uuid AS tenant_webrtc_sip_template_uuid, tenant.webrtc_video_sip_template_uuid AS tenant_webrtc_video_sip_template_uuid, tenant.registration_trunk_sip_template_uuid AS tenant_registration_trunk_sip_template_uuid, tenant.twilio_trunk_sip_template_uuid AS tenant_twilio_trunk_sip_template_uuid \nFROM tenant'] (Background on this error at: http://sqlalche.me/e/f405)
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
    cursor, statement, parameters, context
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 536, in do_execute
    cursor.execute(statement, parameters)
psycopg2.ProgrammingError: ERREUR:  la colonne tenant.slug n'existe pas
LINE 1: SELECT tenant.uuid AS tenant_uuid, tenant.slug AS tenant_slu...
                                           ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/bin/wazo-confd-sync-db", line 11, in <module>
    load_entry_point('wazo-confd==0.1', 'console_scripts', 'wazo-confd-sync-db')()
  File "/usr/lib/python3/dist-packages/wazo_confd/sync_db.py", line 79, in main
    for tenant in session.query(Tenant).all():
  File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 2925, in all
    return list(self)
  File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 3081, in __iter__
    return self._execute_and_instances(context)
  File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 3106, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 980, in execute
    return meth(self, multiparams, params)
  File "/usr/lib/python3/dist-packages/sqlalchemy/sql/elements.py", line 273, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1099, in _execute_clauseelement
    distilled_params,
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1240, in _execute_context
    e, statement, parameters, cursor, context
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1458, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 276, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
    cursor, statement, parameters, context
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 536, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) ERREUR:  la colonne tenant.slug n'existe pas
LINE 1: SELECT tenant.uuid AS tenant_uuid, tenant.slug AS tenant_slu...
                                           ^
 [SQL: 'SELECT tenant.uuid AS tenant_uuid, tenant.slug AS tenant_slug, tenant.sip_templates_generated AS tenant_sip_templates_generated, tenant.global_sip_template_uuid AS tenant_global_sip_template_uuid, tenant.webrtc_sip_template_uuid AS tenant_webrtc_sip_template_uuid, tenant.webrtc_video_sip_template_uuid AS tenant_webrtc_video_sip_template_uuid, tenant.registration_trunk_sip_template_uuid AS tenant_registration_trunk_sip_template_uuid, tenant.twilio_trunk_sip_template_uuid AS tenant_twilio_trunk_sip_template_uuid \nFROM tenant'] (Background on this error at: http://sqlalche.me/e/f405)
Executing upgrade script /usr/share/wazo-upgrade/post-start.d/50-invalidate-wazo-confgend-cache.sh...
Module 'res_pjsip' reloaded successfully.

You database is not up-to-date.
Can you send us the complete upgrade log? We should see something around Setting up xivo-manage-db...

Also, can you give us the output of:

debconf-show wazo-auth xivo-manage-db | grep db-skip

And if db-skip is false, can you give us the output of:

su postgres -c "psql asterisk"
# select * from alembic_version;
# select * from alembic_version_auth;
debconf-show wazo-auth xivo-manage-db | grep db-skip
  wazo-auth/db-skip: false
* xivo-manage-db/db-skip: true

su postgres -c "psql asterisk"
psql (11.11 (Debian 11.11-0+deb10u1))
Saisissez « help » pour l'aide.

asterisk=# select * from alembic_version;
 version_num  
--------------
 e5281e98b300
(1 ligne)

asterisk=#  select * from alembic_version_auth;
 version_num  
--------------
 56cea38f6815
(1 ligne)

full upgrade log

===========================================
wazo-upgrade started at 2021-05-03_12:11:36
===========================================

Upgrading wazo-upgrade
Lecture des listes de paquets…
Construction de l'arbre des dépendances…
Lecture des informations d'état…
Les paquets suivants seront mis à jour :
  wazo-upgrade
1 mis à jour, 0 nouvellement installés, 0 à enlever et 95 non mis à jour.

Dépaquetage de wazo-upgrade (21.05~20210415.150227.477fdf6.deb10) sur (21.01~20210118.165126.de5339a.deb10) ...
Paramétrage de wazo-upgrade (21.05~20210415.150227.477fdf6.deb10) ...
installed version : 21.02
proposed update	: 21.05
Would you like to upgrade your system (all services will be restarted) [Y/n]? Executing upgrade sc

Executing upgrade script /usr/share/wazo-upgrade/pre-start.d/05-run-xivo-check-db.sh...
Checking database...
	NOK (current revision is e5281e98b300)
Executing upgrade script /usr/share/wazo-upgrade/pre-start.d/10-install-dahdi.sh...
Executing upgrade script /usr/share/wazo-upgrade/pre-start.d/10-run-xivo-update-config.sh...
Updating Wazo config
Generating configuration
'dhcp' configuration OK, skipping.
'mail' configuration OK, skipping.
'dhcp' configuration OK, skipping.
'system' configuration OK, skipping.
DONE
Executing upgrade script /usr/share/wazo-upgrade/pre-start.d/20-wazo-update-keys.sh...
Executing upgrade script /usr/share/wazo-upgrade/pre-start.d/30-fix-paths-rights.sh...
Executing upgrade script /usr/share/wazo-upgrade/pre-start.d/31-fix-twisted-reactor.sh...
Executing upgrade script /usr/share/wazo-upgrade/pre-start.d/40-remove-unused-packages.sh...
Lecture des listes de paquets…
Construction de l'arbre des dépendances…
Lecture des informations d'état…
Les paquets suivants seront ENLEVÉS :
  xivo-sysconfd*
0 mis à jour, 0 nouvellement installés, 1 à enlever et 0 non mis à jour.
Après cette opération, 0 o d'espace disque supplémentaires seront utilisés.
(Lecture de la base de données... 93173 fichiers et répertoires déjà installés.)
Purge des fichiers de configuration de xivo-sysconfd (1:21.02~20210129.202137.aab54b8.deb10) ...
dpkg: avertissement: lors de la suppression de xivo-sysconfd, le répertoire « /var/backups/xivo-sysconfd » n'était pas vide, donc il n'a pas été supprimé
Executing upgrade script /usr/share/wazo-upgrade/pre-start.d/41-remove-dahdi-packages.sh...
Executing upgrade script /usr/share/wazo-upgrade/pre-start.d/50-restrict-postgres-hba.sh...
Executing upgrade script /usr/share/wazo-upgrade/pre-start.d/60-migrate-call-log-db.sh...
2021-05-03 12:21:18,972 [23750] (INFO) (wazo_call_logd.main_migrate_db): Migrate call log index...
2021-05-03 12:21:18,974 [23750] (INFO) (wazo_call_logd.main_migrate_db): Call log index migrated
2021-05-03 12:21:19,507 [23754] (INFO) (wazo_call_logd.main_migrate_db): Migrate call log tables. This may take a while...
2021-05-03 12:21:19,512 [23754] (INFO) (wazo_call_logd.main_migrate_db): Migrate call_log table...
2021-05-03 12:21:20,169 [23754] (INFO) (wazo_call_logd.main_migrate_db): call_log table migrated in 0:00:00.656783
2021-05-03 12:21:20,169 [23754] (INFO) (wazo_call_logd.main_migrate_db): Migrate call_log_participant table...
2021-05-03 12:21:21,363 [23754] (INFO) (wazo_call_logd.main_migrate_db): call_log_participant table migrated in 0:00:01.193143
2021-05-03 12:21:21,363 [23754] (INFO) (wazo_call_logd.main_migrate_db): Verifying everything is ok...
2021-05-03 12:21:21,367 [23754] (INFO) (wazo_call_logd.main_migrate_db): Removing old tables...
2021-05-03 12:21:21,422 [23754] (CRITICAL) (root): ERREUR:  ne peut pas supprimer les objets désirés car d'autres objets en dépendent
DETAIL:  contrainte cel_call_log_id_fkey sur table cel dépend de table call_log
HINT:  Utilisez DROP ... CASCADE pour supprimer aussi les objets dépendants.
Traceback (most recent call last):
  File "/bin/wazo-call-logd-migrate-db", line 11, in <module>
    load_entry_point('wazo-call-logd==1.2', 'console_scripts', 'wazo-call-logd-migrate-db')()
  File "/usr/lib/python3/dist-packages/wazo_call_logd/main_migrate_db.py", line 77, in main
    migrate_call_log_tables(config, options.get('max_entries'))
  File "/usr/lib/python3/dist-packages/wazo_call_logd/main_migrate_db.py", line 222, in migrate_call_log_tables
    cel_cur.execute(query)
psycopg2.InternalError: ERREUR:  ne peut pas supprimer les objets désirés car d'autres objets en dépendent
DETAIL:  contrainte cel_call_log_id_fkey sur table cel dépend de table call_log
HINT:  Utilisez DROP ... CASCADE pour supprimer aussi les objets dépendants.

Enabling Wazo services...
Closing port 5060.
Waiting for services to stop successfully...
Waiting for services to start successfully...
	starting dahdi ... ignored
	starting wazo-plugind ... OK
	starting wazo-webhookd ... OK
	starting wazo-sysconfd ... OK
	starting wazo-confgend ... OK
	starting wazo-confd ... OK
	starting wazo-auth ... OK
	starting wazo-dxtora ... OK
	starting wazo-provd ... OK
	starting wazo-agid ... OK
	starting asterisk ... OK
	starting wazo-amid ... OK
	starting wazo-call-logd ... OK
	starting wazo-agentd ... OK
	starting wazo-dird ... OK
	starting wazo-phoned ... OK
	starting wazo-calld ... OK
	starting wazo-websocketd ... OK
	starting wazo-chatd ... OK
Opening port 5060.
Wazo fully booted
Executing upgrade script /usr/share/wazo-upgrade/post-start.d/10-upgrade-official-plugins.py...
Executing upgrade script /usr/share/wazo-upgrade/post-start.d/20-reset-unassociated-devices-to-autoprov.py...
Executing upgrade script /usr/share/wazo-upgrade/post-start.d/25-provd-reconfigure-all-devices.py...
Executing upgrade script /usr/share/wazo-upgrade/post-start.d/45-confd-sync.sh...
2021-05-03 12:22:02,042 [24631] (CRITICAL) (root): (psycopg2.ProgrammingError) ERREUR:  la colonne tenant.slug n'existe pas
LINE 1: SELECT tenant.uuid AS tenant_uuid, tenant.slug AS tenant_slu...
                                           ^
 [SQL: 'SELECT tenant.uuid AS tenant_uuid, tenant.slug AS tenant_slug, tenant.sip_templates_generated AS tenant_sip_templates_generated, tenant.global_sip_template_uuid AS tenant_global_sip_template_uuid, tenant.webrtc_sip_template_uuid AS tenant_webrtc_sip_template_uuid, tenant.webrtc_video_sip_template_uuid AS tenant_webrtc_video_sip_template_uuid, tenant.registration_trunk_sip_template_uuid AS tenant_registration_trunk_sip_template_uuid, tenant.twilio_trunk_sip_template_uuid AS tenant_twilio_trunk_sip_template_uuid \nFROM tenant'] (Background on this error at: http://sqlalche.me/e/f405)
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
    cursor, statement, parameters, context
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 536, in do_execute
    cursor.execute(statement, parameters)
psycopg2.ProgrammingError: ERREUR:  la colonne tenant.slug n'existe pas
LINE 1: SELECT tenant.uuid AS tenant_uuid, tenant.slug AS tenant_slu...
                                           ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/bin/wazo-confd-sync-db", line 11, in <module>
    load_entry_point('wazo-confd==0.1', 'console_scripts', 'wazo-confd-sync-db')()
  File "/usr/lib/python3/dist-packages/wazo_confd/sync_db.py", line 79, in main
    for tenant in session.query(Tenant).all():
  File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 2925, in all
    return list(self)
  File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 3081, in __iter__
    return self._execute_and_instances(context)
  File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 3106, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 980, in execute
    return meth(self, multiparams, params)
  File "/usr/lib/python3/dist-packages/sqlalchemy/sql/elements.py", line 273, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1099, in _execute_clauseelement
    distilled_params,
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1240, in _execute_context
    e, statement, parameters, cursor, context
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1458, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 276, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
    cursor, statement, parameters, context
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 536, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) ERREUR:  la colonne tenant.slug n'existe pas
LINE 1: SELECT tenant.uuid AS tenant_uuid, tenant.slug AS tenant_slu...
                                           ^
 [SQL: 'SELECT tenant.uuid AS tenant_uuid, tenant.slug AS tenant_slug, tenant.sip_templates_generated AS tenant_sip_templates_generated, tenant.global_sip_template_uuid AS tenant_global_sip_template_uuid, tenant.webrtc_sip_template_uuid AS tenant_webrtc_sip_template_uuid, tenant.webrtc_video_sip_template_uuid AS tenant_webrtc_video_sip_template_uuid, tenant.registration_trunk_sip_template_uuid AS tenant_registration_trunk_sip_template_uuid, tenant.twilio_trunk_sip_template_uuid AS tenant_twilio_trunk_sip_template_uuid \nFROM tenant'] (Background on this error at: http://sqlalche.me/e/f405)
Executing upgrade script /usr/share/wazo-upgrade/post-start.d/50-invalidate-wazo-confgend-cache.sh...
Module 'res_pjsip' reloaded successfully.

===========================================
wazo-upgrade stopped at 2021-05-03_12:22:03
===========================================

I don’t see anything strange around xivo-manage-db

Paramétrage de libisccfg163:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u5) ...
Paramétrage de libavahi-client3:amd64 (0.7-4+deb10u1) ...
Paramétrage de libirs-export161 (1:9.11.5.P4+dfsg-5.1+deb10u5) ...
Paramétrage de python3.7 (3.7.3-2+deb10u3) ...
Paramétrage de xivo-manage-db (21.05~20210401.205653.eed7623.deb10) ...
Paramétrage de libbind9-161:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u5) ...
Paramétrage de wazo-auth (21.05~20210415.160410.3443e0c.deb10) ...
Installation de la nouvelle version du fichier de configuration /etc/wazo-auth/config.yml ...
Upgrading wazo-auth database ...

Yes missing output is what is strange. For a weird reason, you have db-skip to true, so your DB will never be updated automatically
To fix it:

echo "xivo-manage-db xivo-manage-db/db-skip boolean false" | debconf-set-selections
xivo-update-db
xivo-check-db

Then you can retry wazo-upgrade

In your case, only the cleanup has failed, so if you rerun the script, it may tell you that Migration already done
After look at your database that everything is migrated

# should be equivalent
select count(*) from call_log;  # old table
select count(*) from call_logd_call_log;  # new table

# should be equivalent
select count(*) from call_log_participant;  # old table
select count(*) from call_logd_call_log_participant;  # new table

Then you can drop old tables
Only if everything else is OK and the script doesn’t want to be executed

DROP TABLE call_log, call_log_participant;
DROP TYPE call_log_participant_role;

I did all, and it worked perfectly.
Old tables were removed with xivo-update-db or wazo-upgrade, no need to drop them by hand.

Thank you a lot

1 Like