helsinki1.jpg

New modules were enabled on www.hiit.fi causing the site to stop responding until the database was restored from backup. Any changes made to web pages between 6:30 and 12:10 were lost.

HIIT IT services will investigate the issue to prevent it from happening again.

A support contract of HIIT's current disk array system EMC LCARiiON CX4-480, called Mithlond, ends at 2015-01-31. After that time neither hardware nor software failures will be fixed. Renewing the support contract was too expensive, thus it wasn't renewed.

To avoid problems due to end-of-support (EOS) of Mithlond, all data is to be relocated. To facilitate that HIIT IT services (HIIT ITS) recently acquired a new NetApp FAS8040 storage system, called Númenor, to which all data will be moved.

Virtual machines

All virtual machine storage needs to be relocated Númenor. A break is needed to change disk type of a virtual machine from physical RDM to virtual RDM. After this is done the virtual machine storage will be moved, without a break, to Númenor.

Physical machines

All servers that use disks (LUNs) from storage array network (later SAN) need to relocate their data from to other storage space using LUNs from new disk array system (Númenor). New disks will be provided as agreed with the server administrator. A short break in services will happen at the time when the last syncing of data is performed.

All breaks are quite likely shorter than announced and they will be scheduled beforehand.

Update at 2015-01-23 10:40 by Pekka Tonteri: The following services will suffer a short break in between 11:30 - 12:00:

• wiki.hiit.fi
• web services (www.hiit.fi, www.reknow.fi, etc.)

Update at 2015-01-23 12:00 by Pekka Tonteri: Web services downtime 11:30-11:44, wiki.hiit.fi downtime 11:49-11:54.

Update at 2015-01-23 16:18 by Markus Nuorento: Vcs.hiit.fi will suffer a short break in between 16:20 - 17:00.

Update at 2015-01-23 16:28 by Markus Nuorento: Vcs service downtime 16:21 - 16:27.

Update at 2015-01-26 17:40 by Pekka Tonteri: fs.hiit.fi, shell.hiit.fi and static web sites (note about static web sites added at 2015-01-27 15:37) will suffer a break at 2015-01-27 09:00 at which time the last synchronisation of home directories takes place.

Update at 2015-01-27 15:37 by Pekka Tonteri:

• fs.hiit.fi's home and group synchronisations are still underway and are expected to be finished in a couple of hours.
• sites share was restored at 14:50 thus bringing static web sites on-line.
• scratch share will not be migrated due to it being temporary data.

Update at 2015-01-27 16:30 by Pekka Tonteri: Home directory synchronisation finished and thus users' static home pages started to be visible to the Internet.

Update at 2015-01-30 15:00 by Markus Nuorento: Break in hand.hiit.fi data disk migration from 15:00 to 16:00.

Update at 2015-01-30 15:50 by Markus Nuorento: Break in hand.hiit.fi is over.

Shell.hiit.fi will be rebooted due to kernel security update. Please see related CVEs from references section.

Update at 13:03 By Markus Nuorento: The break is over.

Due to scheduling error SSL certificate used in Radius authentication expired.

Update at 2015-01-12 16:26 by Sami Niemimäki: HIIT eduroam profile has been updated and is now available in eduroam configuration assessment tool (http://cat.eduroam.org).

Update at 2015-01-16 12:30 by Pekka Tonteri: HIIT eduroam and certificates wiki pages have been updated to contain information about new SHA-2 root certificates as well as information about transition to certificates using SHA-2 hash algorithm instead of an older and (more) vulnerable SHA-1.