On February 26th 2019, we discovered a serious security issue in which 129 Calpendo users' passwords were exposed to four groups of people:
The staff of Exprodo Software
Those with admin rights on their Calpendo
For those universities that host their own Calpendo, the IT staff that manage the Calpendo server
Anybody who can eavesdrop on email sent from your Calpendo to Exprodo Software.
For those using Calpendo's built-in user management (also known as "local" users) where Calpendo stores the passwords of your users, a very small percentage of users were affected. Where they are, it's their Calpendo password that has been exposed and so their Calpendo password will need to be changed.
For those using an external authentication method (e.g. linking to your university’s single sign on system), a higher percentage of users were affected, and it is the user's single-sign-on password that has been exposed which means affected users will need to change their SSO password.
There were two bugs that contributed to this issue:
If anything went wrong while handling a user's login or password change request, the password was stored in clear text in a system event in Calpendo's database. Further, when errors happen in the server, Calpendo is normally configured to send an email to Exprodo Software to tell us about it so that we can find out about some problems and fix them without requiring users to tell us. In this case, the passwords were also embedded in the email sent to us.
When a session times out, it asks you for your password so you can continue your session. If you're using a single-sign-on system, then Calpendo should not ask you for your password, but it did. When faced with Calpendo asking for your password (in a page that looks nothing like your university’s SSO page), some users, but not all, entered their SSO password. Calpendo then could not process the request because it doesn't normally deal with SSO passwords, so this never worked and required the user to refresh their browser to continue. This then fell foul of the first bug, which means whatever the user entered for their SSO password would end up in Calpendo's database and be emailed to Exprodo Software.
This incident has been reported to the UK’s Information Commissioner’s Office in accordance with European GDPR legislation. The problem was identified by Exprodo Software (details below) with nobody else reporting this to us. We are not aware of any users’ passwords being misused as a result of this incident.
What have we done about it?
Both bugs have been fixed in version 8.4.27
When your Calpendo first booted 8.4.27, it went through all the recorded system events and looked for any that contained user passwords. When it found them, it removed the details from the database and made a note of the username of the affected user. A system event was logged at the end to record the list of affected users.
We have removed any trace of users’ passwords from our systems.
What is the timeline of the events?
The bugs were introduced when your Calpendo was upgraded to 8.4. This would have been between August 12th 2018 and around November 2018.
The problem was first identified on Tuesday 26th February 2019 by Paul Robinson, Exprodo Software’s founder, who noticed that an email of an exception report contained what looked like a password.
To reduce the potential impact of the security breach we considered the appropriate course of action to be to fix the bugs and ensure all systems were upgraded to the fixed version before announcing the details to admins. This was to avoid putting Calpendo admins into the position of knowing how to find users’ passwords.
On the afternoon (UK time) of Thursday 28th February, we started rolling out the fix to those systems we host. This process finished in the morning of Friday 1st March, with all systems upgraded out of hours for their time zone.
At 9.30am on Friday 1st March, an email was sent to all those that host their own Calpendo to tell them to upgrade. The email told them the upgrade was urgent but did not tell them what the problem was. The last of those at risk upgraded their Calpendo on Friday 8th March.
We intend to start the process of notifying those in charge of all Calpendos on Monday 11th March. (This being written on 9th March)
Why didn't you tell us as soon as you knew?
Admins have the ability to look at Calpendo system events. If we had told you what happened before you upgraded, then you would have been able to go looking for user passwords.
We thought it better for both compromised users and Calpendo administrators that all user passwords were removed from all Calpendos before telling anybody about the details of the problem.
How did this happen?
We have had protection against exactly this happening for all of Calpendo's history.
What went wrong here was that the method of communicating between the server and a user's browser is being completely changed for version 9.0. A part of the change in the way that communication takes place was done in version 8.4 and involved moving the code that protects against passwords going into the system events. In moving the code, it was broken.
Exactly who could have seen user passwords?
Passwords were stored in Calpendo's database, in a form accessible by all the administrators on your Calpendo. Where this happened, the passwords were buried inside larger messages without anything indicating it contained a password. This made it possible to look at the message without realising it contained a password.
Anybody who handles the backups of your Calpendo database could also access the passwords.
If you have left on the feature that sends exception reports back to Exprodo Software, (and we request that you leave this turned on, even now) then we will have had emails containing the passwords. Those passwords were stored in our email systems and our CRM system as well as in backups of these systems. Passwords have now been completely removed from these internal Exprodo Software systems.
What are the implications for users?
The admins of their Calpendo and the staff at Exprodo Software may have seen users’ passwords.
If this has happened, it would then be possible that those people used that password. In the case of SSO passwords being exposed, it is not possible to tell the extent of the damage that could have been caused. Other people could potentially have logged on to any of your SSO services as you.
It should be noted that it is fairly unlikely that this has happened. However, now the details have been disclosed, the people who look after database backups could go looking for those passwords. Therefore, all users whose passwords were exposed should update their passwords quickly to mitigate the risk.
Why haven't you already forced users to change their password?
For those using SSO, we have no way to do this.
For those using Calpendo's built-in authentication system, we could have forced those users to change their password as soon as 8.4.27 was installed. However:
We wanted to give all sites a chance to upgrade before the nature of the problem was disclosed so that all databases could have the passwords removed from them, removing the most likely way that passwords could be accessed.
If Calpendo forced a lot of users to change their passwords as soon as 8.4.27 was installed, the local admins would have faced a lot of questions from users. That would have made it hard to ensure all systems had the chance to be upgraded before the details were given out. Also, since it would have generated lots of questions to admins, it is only fair that they found out what happened first.
How do I find out who is affected?
Only a Calpendo admin can see who was affected. They can do this as follows:
Log in to your Calpendo and go to Admin->System Events.
Click on the "Source" button, and at the bottom of the drop-down, click the red [-] button to deselect everything. Then tick the DatabaseUpgrader item.
Change the dates at the top to cover the period when your Calpendo was upgraded. If you don't know when this is, put in 28th Feb 2019 to now, and that will definitely cover it.
Click the Go button.
Look for an entry that says "Some problems found" or "No problems found". If you only see "No problems found", then nobody in your Calpendo was affected. If you see "Some problems found", then click on that row.
The details that appear in the bottom half of the screen will include the number of incidents and a list of the login names of the affected users.
What is the scale of this incident?
A total of 129 users had their passwords exposed, of whom 22 users were on Calpendos that we host, and 107 users were on systems hosted by our customers.
What needs to be done now?
Affected users need to be told to change their passwords immediately. If they use the same password for other services then they would ideally change their password on those services as well.
For those Calpendos we have access to, we could contact users ourselves. However, we would like to offer Calpendo admins the option to contact their users themselves instead of having Exprodo Software do it.
If you do want us to do this for you, please let us know.
The affected users should change their passwords quickly to reduce the risk that there could be further damage.
If you host your own Calpendo and there are affected users on your Calpendo, then:
any backups of the Calpendo database you hold will contain those user passwords. You might like to consider whether you should or even whether you can remove those backups.
If your IT system keeps a copy of outgoing emails, then it will contain the user passwords and should be cleaned if possible.
If your university requires that breaches be reported to a central group, then that should happen now, ideally citing this page and the number of users you have had affected.
We are very sorry for having had this issue and for the distress this might cause to those using Calpendo. We are devastated that it has happened.
The fixes for the problems have been thoroughly tested to make sure that it's working properly now, and also in the up-coming version 9.0.