Break reports

Here are listed all breaks in HIIT's IT services.

Break in meriadoc.it.hiit.fi at 2012-04-05 08:00 - 08:10

Schedule: 
2012-04-05 08:00 to 08:10
Duration: 
10 min
Affected services: 
rwin.hiit.fi
Description: 

Following updates will be installed

* Cumulative Security Update for Internet Explorer 8 for Windows Server 2008 R2 x64 Edition (KB2675157)
Security issues have been identified that could allow an attacker to compromise a system that is running Microsoft Internet Explorer and gain control over it.

* Definition Update for Windows Defender - KB915597 (Definition 1.123.1430.0)
Install this update to revise the definition files used to detect spyware and other potentially unwanted software.

*Security Update for Microsoft .NET Framework 3.5.1 on Windows 7 and Windows Server 2008 R2 for x64-based Systems (KB2656372)
A security issue has been identified that could allow an authenticated remote attacker to compromise your system and gain control over it.

* Security Update for Microsoft .NET Framework 4 on XP, Server 2003, Vista, Windows 7, Server 2008, Server 2008 R2 for x64 (KB2656368)
A security issue has been identified that could allow an authenticated remote attacker to compromise your system and gain control over it.

* Security Update for Windows Server 2008 R2 x64 Edition (KB2653956)
A security issue has been identified that could allow an unauthenticated remote attacker to compromise your system and gain control over it.

*Update for Microsoft Office 2007 suites (KB2598306)
Microsoft has released an update for Microsoft Office 2007 suites . This update provides the latest fixes to Microsoft Office 2007 suites.

Break in hasufel.it.hiit.fi at 2012-04-11 11:00 - 11:10

Schedule: 
2012-04-11 11:00 to 11:10
Duration: 
10 min
Affected services: 
wsus.hiit.fi
Description: 

Following updates will be installed

* Security Update for Microsoft .NET Framework 3.5.1 on Windows 7 and Windows Server 2008 R2 for x64-based Systems (KB2656372)
A security issue has been identified that could allow an authenticated remote attacker to compromise your system and gain control over it.

* Cumulative Security Update for Internet Explorer 8 for Windows Server 2008 R2 x64 Edition (KB2675157)
Security issues have been identified that could allow an attacker to compromise a system that is running Microsoft Internet Explorer and gain control over it.

* Security Update for Windows Server 2008 R2 x64 Edition (KB2653956)
A security issue has been identified that could allow an unauthenticated remote attacker to compromise your system and gain control over it.

Break in arod.it.hiit.fi at 2012-04-11 10:30 - 10:40

Schedule: 
2012-04-11 10:30 to 10:40
Duration: 
10 min
Affected services: 
none
Description: 

Following updates will be installed

* Security Update for Microsoft .NET Framework 3.5.1 on Windows 7 and Windows Server 2008 R2 for x64-based Systems (KB2656372)
A security issue has been identified that could allow an authenticated remote attacker to compromise your system and gain control over it.

* Cumulative Security Update for Internet Explorer 8 for Windows Server 2008 R2 x64 Edition (KB2675157)
Security issues have been identified that could allow an attacker to compromise a system that is running Microsoft Internet Explorer and gain control over it.

* Security Update for Windows Server 2008 R2 x64 Edition (KB2653956)
A security issue has been identified that could allow an unauthenticated remote attacker to compromise your system and gain control over it.
 

Break in elrond.it.hiit.fi at 2012-04-13 07:30 - 07:31

Schedule: 
2012-04-13 07:30 to 07:31
Duration: 
1 min
Affected services: 
wiki.hiit.fi
Description: 

Apache and Tomcat servers will be booted to force SSL on all wiki.hiit.fi connections.

Autowatch feture will be disabled by default. Autowatch sends an email about changes in a content you contribute to. This feature is enabled by default after Confluence version 3.5. If you want to enable autowatch again that is possible from personal settings.

Update at 07:31: The break is over.

Break in file service fs at 2012-03-31 10:41 - 16:15

Schedule: 
2012-03-30 10:41 to 16:15
Duration: 
5h 34min
Affected services: 
File services (fs), cluster file services (hpc-fs), web services, entry server shell.
Description: 

Our disk array system suffered from performance problems on thin-LUN. In order to fix this both storage processors were rebooted (separately, of course). During this HIIT's file server Frodo's multipathd handled the reboot of first storage processor (SPA) just beautifully but dumped core when the paths via the second storage processor (SPB) went down. /etc/init.d/multipath-tools was restarted and functionality of multipathd was restored but too late; nfsd erroneously continued to access SAN- disks via failed paths and obviously failed, causing kernel to hung those nfsd processes. This was the reason for having to reboot Frodo.

Frodo's /group file system was previously resized from 9TB to 11TB on-line. Because Linux's resize2fs isn't able to update all superblocks of ext3 filesystem when on-line resize is performed, a file system check is needed on next mount. Thus /group will be checked. Duration of the file system check is expected to be approximately 12 hours.

Other file systems, excluding /group will be up sooner.

Updates on this break report will be added as the situation progresses.

Update at 13:42: It appears that previously described superblock update problem with resize2fs has been fixed and thus running file system check on /group is not neccessary. Connections for services using disks from file server are currently being restored and the break is expected to end much sooner than previously announced.

Update at 14:42: File services(fs), cluster file services (hpc-fs) working normally. There are still problems with entry sever shell and web services.

Update at 15:22: Entry shell server and web services working normally.

Update at 15:56: We install upgrades and boot server in same break.

Update at 16:15: The break is now over.

Pages