One Ticking Second Until Your Network Crashes

AdobeStock 203577294 scaled - Network CrashesRemember the Y2K panic? That was a bit of a pain to deal with, but in the end, everything was solved and all of the nuclear weapons in the world didn’t go off. Hooray!

However, in June 2015 a panic overcame IT departments as thousands of networks crashed in the blink of an eye. Overworked and stressed network and system admins arrived to work and found their end users unable to access their networks.

The problem boiled down to a single issue that keeps ticking on in IT departments.

That issue? The leap second.

Morty-ClockFor those uninitiated, a leap second is occasionally added to our Coordinated Universal Time (UTC), to keep our clocks in sync with tiny changes in Earth’s rotation speed. Now, don’t worry, the Earth isn’t coming to a screeching halt any time soon. According to The International Earth Rotation & Reference Systems Service (IERS), changes in Earth’s speed are nothing to worry about, at least not as far as “end-of-all-life” scenarios are concerned. However, astronomers (like all scientists) are in the business of being precise and therefore want to account for those minuscule differences in time, which is where leap seconds come in.

Leap seconds began to be added in 1972 and that year became its own “year zero”. When calculating leap seconds, IERS uses that year as the reference point. Since 1972, a total of 26 leap seconds have been added.

Generally, leap seconds are added on the last day of month in either June or December. Unlike leap days, leap seconds occur all at once, which can be quite tricky in certain places. The leap second is nestled in right at 11:59 p.m. Coordinated Universal Time (UTC), which is 6:59 p.m. Eastern Standard Time (EST). Therefore, on the East Coast the change might not be even be felt by most users. It falls just at the right time where IT guys have enough time at night to make sure all goes smoothly.

However, 11:59 p.m. UTC is 8:59 a.m. in Japan (Japan Standard Time) and is right at the beginning of their workday, giving IT people no time to fix this issue before their users get to work. If the leap second isn’t accounted for, their servers might be disrupted. Unlike adding leap years, which are added every four years, adding a leap second doesn’t follow an apparent schedule. Since 1972, there have been years where experts added two leap seconds in one year (1972) and then there were periods where zero leap seconds were added. For instance, there were zero leap seconds added between the years 2008 and 2012.

Infrequently adding leap seconds stems from the Earth’s temperamental and varying rotation speed. Climatic and geographical events greatly influence Earth’s rotation speed. For instance, scientists believe that the 2004 Indian Ocean earthquake shortened the rotation speed by 2.68 microseconds. Because of the unpredictability of Earth’s rotation, it’s decided merely six months in advance to add a new leap second.

The reason for that? The leap second is based on the discrepancy between an atomic clock and the Earth’s rotation. You see, it takes the Earth around 84,000 seconds to rotate to what we call one “day”. The way atomic clocks work is that they measure time from the time it takes a Cesium-133 atom to oscillate precisely 9,192,631,770 times — big words that mean it measures atom rotation, not the whole Earth.

This means that the nuclear clocks and regular clocks don’t always align perfectly, but most digital clocks try to keep up with atomic clocks anyway. In short, due to the aforementioned geological events, the difference between Earth’s rotation and atomic rotation averages about one second every 1.5 years.

Sick-Clock-2

While it seems that adding leap seconds shouldn’t be a big deal, as we saw in June, this can be catastrophic for some networks. This is because many computing systems use something called the “Network Time Protocol”, or NTP, to keep themselves in time with the world’s atomic clocks. When that extra second is sprinkled onto it, they just can’t compute, for lack of a better term, and crash.

Most famously, Linux and Java have the worst time with this extra second. This is what took down many sites during the last leap second, and with the addition of a leap second in 2012, Reddit, Huffington Post, and other major sites went down.

While you might not have had crashes on your own system, you may have seen it happen to those sites — that is because those sites likely run on the operating systems that seem to be more susceptible to the leap second crash, such as Linux and Java-based systems.

In 2015, however, these sites learned from past leap seconds and had their own solutions for the June 2015 jump. Google had its own system working to make sure that no users felt any sort of issue during the day.

Standard-Clock-1-e1452183203153

Because IT departments have to deal with leap seconds so infrequently, no one has created a standard system or program to implement leap seconds. Therefore there’s no real automated solution for adding this second to networks, which is what causes them to bite it. There are some pieces of software out there that can help fix your systems, but even those have been known to fail when the time actually comes for the second.

Without a standardized solution, each IT department creates a unique approach to dealing with the extra second. However, these different methods only add to the problem as it can take computers and networks out of sync with each other — sometimes, the different methods don’t interact well — and cause more crashes.

So…how do we fix this?

What if we told you that we should just get rid of the leap second? A proposed solution that’s been floating around suggests eliminating leap seconds once and for all. This would be specifically for computing reasons and uses, and it would provide accurate time as well as what networks around the world need in order to stay in sync: continuity.

The International Telecommunication Union, or ITU, meets every three to four years to review and change the Radio Regulations. They’re an entity of the United Nations, and they last met in 2012. The issue of the leap second was brought up (which is under their jurisdiction), and the concept of a continuous timescale was proposed. This timescale would basically just start from its own reference point, and continue going. There would be no turning the clock forward, and all operating systems would operate on this new timescale.

What happened then?

The delegates, who are representatives of different countries from around the world, voted. And they decided… to postpone the decision.

In fact, they decided to push the proposal of a solution until the conference in 2023. Yikes!

Why the wait? Well, the ITU is to conduct studies to “consider the feasibility of achieving a continuous reference time-scale, whether by the modification of UTC or some other method.”

Pipe-Bomb-2023

There’s over ten organizations working to gather this data and suitable solutions for this issue for the 2023 conference, where a solution will be proposed and, with any luck, accepted.

In the meantime, Sys Admins will have to just keep using their clever fixes, or external software, to help them with their leap second transition. Hopefully by 2023, we’ll have had enough time to figure out a better solution for this ticking time bomb of an issue.

Ed Clark| IT Advocate for LinkTek | LinkMail@LinkTek.com

Try Now!

Related Posts

AdobeStock 890754272 - Network Crashes

The Falcon Fiasco: CrowdStrike Outage Brings Millions to Their Knees

Jul 29, 2024

IT Tips & Tricks The Falcon Fiasco: CrowdStrike Outage Brings Millions to Their Knees Published 29 July 2024 Share on Facebook Share on Twitter Share…

Read More
Why SharePoint Online Migration Is So Widespread Header - Network Crashes

Why SharePoint Online Migration Is So Widespread

Jun 27, 2024

IT Tips & Tricks Why SharePoint Online Migration Is So Widespread Is It Time To Bid Farewell to That Beloved (or Not-So-Beloved) File Server? Published…

Read More
Migration Pointers - Network Crashes

SharePoint Online Migration Checklist: Practical Pointers

Jun 17, 2024

Considering a migration to SharePoint Online? Want to ensure the smoothest possible transition? This practical migration checklist can help get you there.

Read More
box - Network Crashes

Release of New Version of Link-Fixing Software Offers Support for Migrations To or From Box.com

Feb 25, 2022

Data migrations can be tricky. Migrating to or from Box is no different. Get some help.

Read More
AdobeStock 514831438 - Network Crashes

Latest Release of LinkFixer Advancedâ„¢ Streamlines Cloud Data Migrations

Jan 27, 2020

Migrating to the cloud? Link-fixing software helps prevent post-migration data-loss. Chat with us about options, including a free webinar.

Read More
Server c - Network Crashes

Newest Release of Automatic Link Fixing Software Simplifies the Process of Protecting Links When Moving to SharePoint On-Premise or Online

Oct 2, 2018

Whether on-premises or online, a migration to SharePoint could result in data loss without the best tool for the job. Chat with us about your options.

Read More