Upgrade to vCloud director 9.5 fails with "Exception encountered while converting the roles"




ERROR  :

During a vCloud 8.20 upgrade to 9.5 we got the following error:

Found one or more disabled ESX/ESXi hosts, upgrading will automatically enable all disabled hosts after the upgrade. Do you wish to continue with upgrading the database? [Y/N]  y
The next step in the upgrade process will change the vCloud Director database schema.
Backup your database now using the tools provided by your database vendor.
Enter [Y] after the backup is complete. y
Running 5 upgrade tasks
Executing upgrade task:
Successfully ran upgrade task
Executing upgrade task:
Successfully ran upgrade task
Executing upgrade task:
Successfully ran upgrade task
Executing upgrade task:
.............[13]
Unable to upgrade the database: java.lang.IllegalStateException: Exception encountered while converting the roles


ANALYSIS :

The vcloud-container-debug.log will contain the below entries

2019-03-22 23:24:10,523 | DEBUG    | pool-1-thread-1           | ConvertRoles                   | Loaded 2,865 roles and their rights mappings |
2019-03-22 23:24:12,772 | DEBUG    | pool-1-thread-1           | ConvertRoles                   | Converting any users assigned directly to role templates to role template instances |
2019-03-22 23:24:12,848 | DEBUG    | pool-1-thread-1           | ConvertRoles                   | Adding Defer to Identity Provider instance for 969691a6-f825-4152-8515-dd16b9606a4d |
2019-03-22 23:24:12,874 | DEBUG    | pool-1-thread-1           | ConvertRoles                   | Adding Defer to Identity Provider instance for 8f1954cf-1a89-44d6-b45f-159538264aeb |
2019-03-22 23:24:12,876 | DEBUG    | pool-1-thread-1           | ConvertRoles                   | Converting the role templates |
2019-03-22 23:24:12,942 | WARN     | pool-1-thread-1           | SerialAggregateTask            | Convert to version 2.0 of RBAC: Task failed due to uncaught exception |
java.lang.IllegalStateException: Exception encountered while converting the roles
        at com.vmware.vcloud.upgrade.tasks.vulcan.ConvertRoles.call(ConvertRoles.java:84)
        at com.vmware.upgrade.task.SerialAggregateTask.call(SerialAggregateTask.java:65)
        at com.vmware.upgrade.task.AbstractDelegatingTask.doCall(AbstractDelegatingTask.java:144)
        at com.vmware.upgrade.sql.task.TransactionTask.doCall(TransactionTask.java:95)
        at com.vmware.upgrade.task.AbstractDelegatingTask.call(AbstractDelegatingTask.java:123)
        at com.vmware.upgrade.task.SerialAggregateTask.call(SerialAggregateTask.java:65)
        at com.vmware.upgrade.task.AbstractDelegatingTask.doCall(AbstractDelegatingTask.java:144)
        at com.vmware.upgrade.factory.GraphUpgradeDefinitionFactory$UpgradeTask.doCall(GraphUpgradeDefinitionFactory.java:127)
        at com.vmware.upgrade.task.AbstractDelegatingTask.call(AbstractDelegatingTask.java:123)
        at com.vmware.upgrade.task.SerialAggregateTask.call(SerialAggregateTask.java:65)
        at com.vmware.upgrade.task.SerialAggregateTask.call(SerialAggregateTask.java:65)
        at com.vmware.upgrade.task.SerialAggregateTask.call(SerialAggregateTask.java:65)
        at com.vmware.upgrade.task.SerialAggregateTask.call(SerialAggregateTask.java:38)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: The UPDATE statement conflicted with the REFERENCE constraint "fk_orgmember_to_role". The conflict occurred in database "vcloud", table "dbo.org_member", column 'app_role_id'.
        at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:372)
        at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2820)
        at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2258)
        at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:632)
        at net.sourceforge.jtds.jdbc.JtdsStatement.processResults(JtdsStatement.java:584)
        at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQL(JtdsStatement.java:546)
        at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeUpdate(JtdsPreparedStatement.java:504)
        at com.vmware.vcloud.upgrade.tasks.vulcan.ConvertRoles.convertRoleTemplatesToGlobalRoles(ConvertRoles.java:252)
        at com.vmware.vcloud.upgrade.tasks.vulcan.ConvertRoles.call(ConvertRoles.java:76)
        ... 16 more



CAUSE

This happens when the users has been assigned a role template as a role

From the above logs, It looks like the check for assignment of role templates was executed, and it found two instances of a user being assigned the "Defer to Identity Provider" role template before the exception is thrown (see the highlighted "Adding Defer to Identity Provider instance" entries). 


HOW TO FIX :


Run the SELECT query against the database. This should find users who've been assigned a role template as a role.

Note: This does no harm to the database.

SELECT org_member.member_name, organization.name, role.name
 FROM org_member, organization, role
 WHERE organization.org_id = org_member.org_id
 AND org_member.app_role_id = role.role_id
 AND role.role_type_enum = 5

The output to query would similar to this

 member_name     org_name                               role_name
orgadmin ABC              Defer to Identity Provider
orgadmin MYF                        Defer to Identity Provider
orgadmin XYZ              Organization Administrator
orgadmin MLB                              Organization Administrator


From the above output name represents the Organization name. Go ahead and reassign those users to a different role (not a role template). 


Restore the vCloud director backup and retry the upgrade to 9.5. 


Hope this helps!! 
Cheers :) 

Comments

Popular posts from this blog

vCloud director 9.7 appliance deployment step by step guide

How to Install and Uninstall Guest agent for vRA Windows machines

Unable to change Provider VDC virtual hardware support beyond version 10