Various issues with upgrading from 18.03 to 19.14

I am testing upgrading an existing Wazo installation from 18.0. to 19.13 using this guide. I’ve run into some headaches that I thought I’d ask about here:

  1. After running wazo-upgrade and installing the wazo-ui package some admin users from before the upgrade are not are able to log in. I am able to use wazo-auth-cli user list to view a list of users, but I don’t see any of the old admin accounts. I am able to login with my old account that is not listed. My current account is in the auth_user table and I have the wazo_default_admin_policy policy on my account, but another account that worked pre-upgrade does not have that policy. Is there a place in the newer interface to manage admin accounts?

  2. After running wazo-dist-upgrade and rebooting, I can no longer log in to the web interface. The end of the process printed /usr/bin/wazo-dist-upgrade-buster: line 201: wazo-check-conffiles: command not found. Trying to login to the web interface fails and logs the following into /var/log/wazo-ui.log:

2019-11-14 15:36:59,014 [1295] (ERROR) (wazo_ui.errors): Unexpected error:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/wazo_ui/plugins/authentication/form.py", line 43, in validate
    response = auth_client.token.new(expiration=60 * 60 * 12)
  File "/usr/lib/python3/dist-packages/wazo_auth_client/commands/token.py", line 28, in new
    self.raise_from_response(r)
  File "/usr/lib/python3/dist-packages/xivo_lib_rest_client/command.py", line 25, in raise_from_response
    response.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: BAD REQUEST for url: https://localhost:9497/0.1/token

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/lib/python3/dist-packages/flask_classful.py", line 231, in proxy
    response = view(**request.view_args)
  File "/usr/lib/python3/dist-packages/flask_classful.py", line 202, in inner
    return fn(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/wazo_ui/plugins/authentication/view.py", line 25, in post
    return self._login()
  File "/usr/lib/python3/dist-packages/wazo_ui/plugins/authentication/view.py", line 33, in _login
    if form.validate_on_submit():
  File "/usr/lib/python3/dist-packages/flask_wtf/form.py", line 101, in validate_on_submit
    return self.is_submitted() and self.validate()
  File "/usr/lib/python3/dist-packages/wazo_ui/plugins/authentication/form.py", line 54, in validate
    raise ValidationError(l_('Error with Wazo authentication server: %(error)s', error=e.message))
AttributeError: 'HTTPError' object has no attribute 'message'

Trying to list users with the wazo-auth-cli user list command prints 400 Client Error: BAD REQUEST for url: https://localhost:9497/0.1/token, and trying to get a token with curl gives an internal server error:

HTTP/1.1 500 INTERNAL SERVER ERROR
Content-Type: application/json
Content-Length: 37
Access-Control-Allow-Origin: *
Date: Thu, 14 Nov 2019 22:40:20 GMT
Server: Cheroot/6.5.4

{"message": "Internal Server Error"}

Tailing the /var/log/wazo-auth.log file fills the page with 404 errors as well.

  1. Trying to check for updates after upgrading to Buster with wazo-upgrade fails, and I have to run wazo-dist -m pelican-stretch again. Running wazo-upgrade then prints that xivo-upgrade is already the newest version (19.14~20190925.180051.ce3a5c8.deb9)..

Hi!

  1. Do you see the old accounts in wazo-auth-cli user list --recurse ? --recurse means: “list all users in the master tenant and all subtenants”. Since wazo-auth-cli is in the master tenant (which is more like a system tenant) and all other tenants are subtenants of master, --recurse tells wazo-auth-cli to effectively list all user accounts. In the wazo-ui, you can manage users in Credentials > Identities. Be sure to select the correct tenant in the top left dropdown list.

  2. There was a big problem during your upgrade and it did not finish correctly. Can you show us the file /var/log/wazo-upgrade.log that contains the logs of the wazo-dist-upgrade? All other weird behaviors most likely derive from this failed upgrade.

  3. Once you upgrade to Buster, any package from Stretch will be in a lower version and wazo-upgrade, which uses the Debian tools under the hood, will not upgrade anything.

  1. Do you see the old accounts in wazo-auth-cli user list --recurse

I do see them! They account that current works and the accounts that do not have the same tennant uuid and they are all enabled. Going into the Credentials > Identities menu lets me find the old users. Going into my currently working user has wazo_default_admin_policy, but the other users do not have any policies. Trying to add one only shows one policy.

Upon looking at the auth_policy table I see that the devault wazo and xivo policies all have one uuid (the master tenant), while the one non-admin policy has a different uuid (the wazo tenant). My user is in the wazo tenant but still has a policy that exists in the master tenant, while others do not. Opening the dropdown in the top left dropdown under the wazo logo does not show any master tenant, just the wazo one.

  1. I will add another reply with the log file, it takes a while to run. I’m currently getting a different error related to the flask wtf package install.

  2. Maybe the error is tied to the failure in step 2?

I’ve found that if I run wazo-upgrade and then wazo-dist-upgrade without installing the wazo-ui package in between, it seems like I have no upgrade errors.

Upgrading, installing wazo-ui, and then doing the dist upgrade causes the wazo-check-conffiles: command not found error to happen. I am unable to attach the log file to this comment because the forum only allows image files, but here’s a couple errors (with line numbers when I left out lines) from it:

[line 5897]
Fetched 375 MB in 2min 5s (2,986 kB/s)
E: Failed to fetch http://ftp.us.debian.org/debian/pool/main/p/python-defaults/python2_2.7.16-1_amd64.deb  Hash Sum mismatch
   Hashes of expected file:
    - SHA256:d412efde85b1bbdec65f7a4f4f3fc9754eb33b4712218d51eaaa8f6b950613e0
    - MD5Sum:02f7d78d3b6c41ec8d4eb8c1ededbaf2 [weak]
    - Filesize:41616 [weak]
   Hashes of received file:
    - SHA256:de1cbe042b66525eac2601c69fb32e9c7ecb9403aa953429e1b54269ed21138f
    - MD5Sum:5679cd3928179938a59bd9e5f0505f72 [weak]
    - Filesize:5841 [weak]
   Last modification reported: Mon, 04 Mar 2019 21:12:39 +0000
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
[line 6003]
Waiting for services to start successfully...
	starting dahdi ... OK
	starting wazo-plugind ... OK
	starting wazo-webhookd ... OK
	starting xivo-sysconfd ...Job for xivo-sysconfd.service failed because the control process exited with error code.
See "systemctl status xivo-sysconfd.service" and "journalctl -xe" for details.
 NOK
[line 6012]
Executing upgrade script /usr/share/xivo-upgrade/post-start.d/09_provd_add_phonebook_ip.sh...
No handlers could be found for logger "xivo.token_renewer"
Error while connecting to wazo-provd: HTTPSConnectionPool(host='localhost', port=8666): Max retries exceeded with url: /0.2/pg_mgr/install/installable (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f2cb4f2c610>: Failed to establish a new connection: [Errno 111] Connection refused',))
Executing upgrade script /usr/share/xivo-upgrade/post-start.d/18_upgrade_official_plugins.py...
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 80, in create_connection
    raise err
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 70, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 841, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 301, in connect
    conn = self._new_conn()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 168, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f237a037e80>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=9497): Max retries exceeded with url: /0.1/token (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f237a037e80>: Failed to establish a new connection: [Errno 111] Connection refused',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/xivo-upgrade/post-start.d/18_upgrade_official_plugins.py", line 58, in <module>
    main()
  File "/usr/share/xivo-upgrade/post-start.d/18_upgrade_official_plugins.py", line 40, in main
    token_data = auth_client.token.new(expiration=300)
  File "/usr/lib/python3/dist-packages/wazo_auth_client/commands/token.py", line 25, in new
    r = self.session.post(self.base_url, headers=headers, json=data)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 581, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='localhost', port=9497): Max retries exceeded with url: /0.1/token (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f237a037e80>: Failed to establish a new connection: [Errno 111] Connection refused',))
Executing upgrade script /usr/share/xivo-upgrade/post-start.d/20_reset_unassociated_devices_to_autoprov...
Traceback (most recent call last):
  File "/usr/share/xivo-upgrade/post-start.d/20_reset_unassociated_devices_to_autoprov", line 36, in <module>
    token_data = auth_client.token.new(expiration=300)
  File "/usr/lib/python2.7/dist-packages/wazo_auth_client/commands/token.py", line 25, in new
    r = self.session.post(self.base_url, headers=headers, json=data)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 581, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='localhost', port=9497): Max retries exceeded with url: /0.1/token (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f4c3c66a290>: Failed to establish a new connection: [Errno 111] Connection refused',))
[line 6124]
dpkg: warning: while removing xivo-dist, directory '/usr/lib/python2.7/dist-packages/xivo_dist' not empty so not removed
/usr/bin/wazo-dist-upgrade-buster: line 201: wazo-check-conffiles: command not found

Maybe .log files need to be authorized for this forum to allow me to attach it?

Using wazo-upgrade, installing wazo-ui, removing wazo-ui, and using wazo-dist-upgrade appears to succeed, though I have barely tested it. There are no errors in the upgrade log and the server appears to have the most recent version of wazo-ui, unlike before when it had the 19.13 version. I can also successfully run wazo-upgrade without issue, though it does allow me to update to the same proposed version (19.15) as the currently installed version.