HIIT - Break reports

Break in shell.hiit.fi at 2017-11-10 11:10 - 11:20

pmtonter — Fri, 10 Nov 2017 09:10:00 +0000

Schedule:

2017-11-10 11:10 to 11:20

Duration:

10 min

Affected services:

shell.hiit.fi

Description:

General purpose GNU/Linux server shell.hiit.fi will be rebooted due to kernel upgrade. All other pending updates will be installed as well.

Break in LDAP logins using secondary LDAP server at 2016-11-15 15:57 - 2016-11-16 10:20

niemimak — Tue, 15 Nov 2016 13:57:00 +0000

Schedule:

2016-11-15 15:57 to 2016-11-16 10:20

Duration:

18h 23min

Affected services:

Web services that use LDAP authentication including wiki.hiit.fi, www.hiit.fi, www.hict.fi, www.reknow.fi, wiki.hiit.fi.

Description:

Secondary LDAP server lose.it.hiit.fi stopped responding at 2016-11-15 16:00 and caused logins to fail on services that use only that server for LDAP authentication.

Break in faramir.it.hiit.fi at 2016-10-13 16:34 - 16:38

niemimak — Fri, 21 Oct 2016 13:34:44 +0000

Schedule:

2016-10-21 16:34 to 16:38

Duration:

4min

Affected services:

www.hiit.fi, www.reknow.fi, www.hict.fi

Description:

faramir.it.hiit.fi was rebooted due to updates.

Break in shell.hiit.fi at 2016-10-20 11:11 - 11:13

niemimak — Thu, 20 Oct 2016 08:11:00 +0000

Schedule:

2016-10-20 11:11 to 11:13

Duration:

2min

Affected services:

Shell.hiit.fi

Description:

Shell.hiit.fi was booted due to kernel update.

Break in faramir.it.hiit.fi at 2016-10-13 16:23 - 16:25

niemimak — Thu, 13 Oct 2016 13:23:00 +0000

Schedule:

2016-10-13 16:23 to 16:25

Duration:

2min

Affected services:

www.hiit.fi, www.reknow.fi, www.hict.fi

Description:

Web server faramir.it.hiit.fi was booted due to updates.

Break in knee.hiit.fi at 2016-09-05 17:00 - 17:02

nuorento — Mon, 05 Sep 2016 13:38:40 +0000

Schedule:

2016-09-05 17:00 to 17:02

Duration:

2min

Affected services:

knee.hiit.fi

Description:

Project virtual server knee.hiit.fi needs more resources and due to that it needs to be rebooted.

Update at 2016-09-05 17:02 by Markus Nuorento: The break is now over.

Break in Drupal sites on faramit.it.hiit.fi

niemimak — Wed, 10 Aug 2016 13:09:57 +0000

Schedule:

2016-08-10 16:10 to 16:40

Duration:

30min

Affected services:

www.hiit.fi, www.hict.fi, www.reknow.fi, figs.hiit.fi, fics.hiit.fi

Description:

Authorization configurations will be updated on Drupal sites running on faramir.it.hiit.fi. During the update logged in user may not be available to create, delete or edit any content.

Break in toe.hiit.fi at 2016-06-08 16:40 - 20:07

nuorento — Wed, 08 Jun 2016 15:04:00 +0000

Schedule:

2016-06-08 16:40 to 20:07

Duration:

3h 27min

Affected services:

toe.hiit.fi

Description:

Due to failed systemd configuration after reboot virtual server couldn't find it's root volume group and thus root volume.

We are currently investigating reason for this.

Update at 2016-06-08 20:07 by Markus Nuorento: toe.hiit.fi is operational again. At the same break server was upgraded to Debian GNU Linux (Jessie) 8.5.

Shutdown of HIIT's OS X update service

niemimak — Mon, 30 May 2016 22:00:00 +0000

Schedule:

2016-05-31 23:59

Duration:

indefinite

Affected services:

Managed Software Center or Managed Software Update (Munki)

Description:

HIIT IT has provided updates to OS X clients via munki-packages.hiit.fi since August 2012, but this service has become redundant as both mother universities offer similar service. The shutdown of munki-packages.hiit.fi affects all HIIT's Macs, which haven't already been reinstalled to use services provided by Aalto University or University of Helsinki. Everyone is encouraged to have their Macs reinstalled as soon as possible, but not later than May 31st 2016.

Break in HIIT production networks in Otaniemi area in between 2016-03-10 10:00 and 2016-03-14 13:27

pmtonter — Thu, 10 Mar 2016 08:00:00 +0000

Schedule:

2016-03-10 10:00 to 2016-03-14 13:27

Duration:

51h 27min

Affected services:

HIIT production networks in Otaniemi area.

Description:

There will be a short break in HIIT's production networks in Otaniemi area due to them being configured to be ran over an MPLS tunnel instead of a light-path. This is done to ensure more redundant L2 configuration.

Update at 2016-03-10 09:55 by Pekka Tonteri: Break's schedule was updated from 9-12 to 10-14.

Update at 2016-03-14 13:15 by Markus Nuorento: There are problems with connections between Kumpula and Otaniemi. Quick fix before we get them fixed is to drop MTU of network interface to 1428.

Update at 2016-03-14 13:27 by Markus Nuorento: The break is over. Connection with standard MTU setting (1500B) works normally. Jumbo frames (9000B) don't work.

Break in Kumpula-Otaniemi light-paths at 2016-03-09 14:22 - 2016-03-10 02:20

pmtonter — Wed, 09 Mar 2016 12:22:00 +0000

Schedule:

2016-03-09 14:22 to 2016-03-10 02:20

Duration:

11h 58min

Affected services:

Network connections running over Kumpula-Otaniemi light-paths.

Description:

The following services are affected by a broken light-path in between Kumpula and Otaniemi:

HIIT's network connections in Otaniemi region, including Open Innovation House (OIH), T- and TUAS- building.
Testbed.

The schedule to fix the broken fibre is not available.

To remediate the situation and to gain redundancy to production network connections, all other VLANs except the testbed will be relocated from the Kumpula-Otaniemi light-path to MPLS tunnel at at 2016-03-10. The MPLS tunnel will be ran from University of Helsinki to Aalto university and it will carry the traffic over IP using FUNET links of both universities. FUNET links are redundant, a.k.a. there are two (2) of them in each university so breakage of one link will not cause a break in connections.

Update at 2016-03-10 03:10 by Pekka Tonteri: Light-paths seem to function again and the break is over. :) The production network will see some downtime in the morning due to MPLS configuration. That break should, however, be quite short. A separate break report will be issued due to MPLS configuration.

Update at 2016-03-10 03:23 by Pekka Tonteri: A break report for MPLS configuration has been created.

References:

FUNET break report.

Break in Open Innovation House's network connections at 2016-02-22 13:36 - 2016-02-23 12:05

pmtonter — Mon, 22 Feb 2016 11:36:00 +0000

Schedule:

2016-02-22 13:36 to 2016-02-23 12:05

Duration:

22h 29min

Affected services:

Network connections of Open Innovation House

Description:

A lightpath in between Kumpula and Otaniemi is broken. No estimate about the duration of the break currently exists.

More information from the FUNET break report.

Update at 2016-02-23 by Pekka Tonteri: The light path has been fixed at 12:05.

References:

FUNET break report.

Break in shell.hiit.fi at 2016-02-19 13:30 - 13:45

nuorento — Fri, 19 Feb 2016 09:30:34 +0000

Schedule:

2016-02-19 13:30 to 13:45

Duration:

15min

Affected services:

shell.hiit.fi

Description:

Shell.hiit.fi need to be rebooted due to glibc library security update.

References:

https://www.debian.org/security/2016/dsa-3481

Mitre's CVE dictionary: CVE-2015-7547, CVE-2015-8776, CVE-2015-8778, CVE-2015-8779.

Break in eomer.it.hiit.fi at 2016-02-19 09:00 - 09:04

nuorento — Fri, 12 Feb 2016 10:19:56 +0000

Schedule:

2016-02-19 09:00 to 09:04

Duration:

4min

Affected services:

wiki.hiit.fi

Description:

There is a short break in faramir.it.hiit.fi as we reboot the server to enable non-critical kernel updates.

Update at 2016-02-19 09:05 by Markus Nuorento: Server succesfully booted. The break is over.

Break in range.hiit.fi at 2016-02-18 10:00 - 11:00

nuorento — Sun, 07 Feb 2016 11:28:04 +0000

Schedule:

2016-02-18 10:00 to 11:00

Duration:

Affected services:

range.hiit.fi

Description:

There will be one hour break in range.hiit.fi due to the OS upgrade. Range.hiit.fi will be upgraded to Debian GNU/Linux v8.3.

Break in eye.hiit.fi at 2016-02-17 10:00 - 11:00

nuorento — Sat, 06 Feb 2016 11:28:04 +0000

Schedule:

2016-02-17 10:00 to 11:00

Duration:

Affected services:

eye.hiit.fi

Description:

There will be one hour break in eye.hiit.fi due to the OS upgrade. Eye.hiit.fi will be upgraded to Debian GNU/Linux v8.3.

Break in elbow.hiit.fi at 2016-02-16 10:00 - 10:30

nuorento — Fri, 05 Feb 2016 11:28:04 +0000

Schedule:

2016-02-16 10:00 to 10:30

Duration:

30min

Affected services:

elbow.hiit.fi

Description:

There will be one hour break in elbow.hiit.fi due to the OS upgrade. Elbow.hiit.fi will be upgraded to Debian GNU/Linux v8.3.

Update at 2016-02-16 10:30 by Markus Nuorento: The break is over.

Break in b-course.hiit.fi at 2016-02-15 10:00 - 11:00

nuorento — Fri, 05 Feb 2016 10:28:04 +0000

Schedule:

2016-02-15 10:00 to 11:00

Duration:

Affected services:

b-course.hiit.fi

Description:

There will be one hour break in b-course.hiit.fi due to the OS upgrade. B-course.hiit.fi will be upgraded to Debian GNU/Linux v8.3.

Break in boole12.hiit.fi at 2016-01-27 17:55 - 2016-01-28 12:36

nuorento — Thu, 28 Jan 2016 08:01:39 +0000

Schedule:

2016-01-27 17:55 to 2016-01-28 12:36

Duration:

18h 41min

Affected services:

boole12.hiit.fi

Description:

Boole12.hiit.fi hang again after disk controller replacement. We are currently investigating reasons for this.

Update at 2016-01-28 10:51 by Markus Nuorento: Service for the server has been ordered and it will arrive later today.

Update at 2016-01-28 12:36 by Markus Nuorento: Battery backed cache module was changed from HP E200i Samrt Array disk controller and the server is again operational.

Break in boole12.hiit.fi at 2016-01-22 22:30 - 2016-01-26 12:55

nuorento — Tue, 26 Jan 2016 14:17:21 +0000

Schedule:

2016-01-22 22:30 to 2016-01-26 12:55

Duration:

3d 14h 25min

Affected services:

boole12.hiit.fi

Description:

There is a break in boole12.hiit.fi due to hardware failure.

Update at 2016-01-26 12:55 by Markus Nuorento: Smart Array e200i disk controller was replaced and the server operationality was restored.