Cork Internet eXchange
Industry: Data Centres
Locations: Cork, Ireland
For co-location data centres, nothing indicates reliability like an Uptime Institute tier score. The Institute is generally regarded as the leading international certification body with respect to data centre uptime, or “availability.”
Although there aren’t yet any Institute-rated data centres in Ireland, the industry is booming and savvy data centre operators like Jerry Sweeney, managing director of Cork Internet eXchange (CIX) and Uptime Institute certified Accredited Tier Specialist, are using the logic behind the Institute’s data centre assessment system to increase their centres’ resiliency. CIX has invested in robust cogeneration and is building towards full fault-tolerance, eliminating power outage threats one at a time.
So when Jerry enrolled CIX in demand response (DR) in 2012—a program that pays energy consumers to cut back their usage during moments of grid strain—it caught people’s attention. After two years of successful participation, Jerry Sweeney believes demand response hasn’t adversely affected CIX’s reliability in any way; in fact, it’s become a competitive advantage.
CIX is a 16,000 square foot facility with 1.8MVA of generative capacity. Since opening in 2008, CIX has grown to become the prime IP connectivity centre for the entire Munster region of Ireland. Companies such as Vodafone, BT, COLT, and Hibernia Networks use the facility as a major regional point of presence. An estimated 40,000 homes and businesses derive connectivity from the centre via 2,000 fibre pairs and a 30 meter telecommunications tower. CIX offers co-location services to government and business customers and also provides dedicated and virtual servers as well as a firewall service.
The centre has invested heavily in a 24/7/365 network operations centre to minimise outage threats. “If you’re a data centre you’re paranoid about outages,” said Jerry Sweeney, CIX’s managing director. “Former Intel CEO Andy Grove’s book, ‘Only the Paranoid Survive,’ is keenly applicable to the data centre industry. No financial reward will encourage you to risk an outage.”
Ensuring continuous availability often means hefty energy bills. For CIX, after wages and connectivity, energy is the third largest expense—currently €300,000 and growing by 25% annually. CIX was interested in offsetting some of these costs through demand response participation, but had initial concerns about its ability to curtail any of its load.
When the Company dug deeper into demand response with EnerNOC, CIX discovered that the opportunity posed no threat to its uptime—there is no penalty for not curtailing, only a missed opportunity to generate revenue. Moreover, EnerNOC would audit CIX’s generation capacity to help ensure it was fully prepared to go off-grid without a moment’s downtime. For Sweeney, EnerNOC’s commitment to assuring his system’s reliability resonated with his own attitude: “If you don’t trust your system at all times, then there’s something wrong with your system.”
Demand response also resonated with CIX’s active participation in renewable and alternative energy. Since inception, the data centre has implemented a number of green energy projects, including becoming the first data centre in Ireland designed from the ground up to have cold aisle containment and free cooling chillers. The data centre achieves an impressive annual average power usage effectiveness (PUE) of 1.18.
CIX has publicly promoted involvement in demand response in its capacity to advance renewable energy penetration onto the national grid.
How Demand Response Strengthens Data Centres Through Grid Disruption Advance Notice
The tier framework views utility power simply as the most economically effective power source, but not necessarily the most reliable source of disruption-free power. The Uptime Institute actually assumes data centres will face periodic grid disruptions, and any data centre manager can attest to their inevitability. “Your resilience comes from your on-site generation,” explained Sweeney, “not the reliability of your grid.” Treating grid disruptions as an expected operational condition means having extensive synchronous generation sources.
EnerNOC adds resilience to data centres cogeneration by giving its customers advance notice of grid problems. EnerNOC provides alerts about grid capacity shortages in time for CIX to either prepare for a trigger-event or preemptively move to back-up power for the duration of the grid instability. “Going off-grid while the system is still available and checking to see that all your systems are up is much better than waiting for the grid to fail,” said Sweeney on the value of receiving grid emergency updates.
Not all grid emergencies are equally manageable for data centres. “UPSs are very good at dealing with clean outages. But what really bothers them is brownouts, short outages, and over voltages,” explained Sweeney. “So if you can exit the market completely when those events are likely to happen, you have a huge advantage.” Fundamentally, EnerNOC’s dispatch intelligence helps data centre operators mitigate power source interruption risk, which Sweeney notes “has a positive benefit to your actual uptime.”
Using Unplanned Testing with EnerNOC to Ensure System Reliability
Back-up generation testing is an essential part of any data centre manager’s operations. CIX is no exception, testing each of its two 900kVA GenSets every two weeks for a minimum of 30 minutes. Testing is often scheduled well in advance and staff are prepared for the simulated shutdown. EnerNOC demand response implements a live, on-load testing regime that more accurately replicates the utility fail risk without exposing the site to an actual utility fail (the grid connection can be retained at all times during a Demand Reduction Event).
Rather than always testing the generators on a scheduled basis, the site is able to fortify its infrastructure and operations by carrying out generator runs outside the normal scheduled test time window. Fortunately, Jerry’s team was already implementing an extremely rigorous on-load testing regime. “Moving a percentage of your load to generation is absolutely necessary, and if you can sell your generative capacity then that is an added bonus,” said Sweeney.
As a participant in the grid operator Eirgrid’s demand response program, CIX can guarantee 300kW of curtailed load for up to 20 dispatches in a given year. In the rare event that the grid encounters difficulties, a demand response call is initiated. EnerNOC immediately relays that call to CIX, which already has a curtailment plan they can enact quickly. Each year, CIX earns north of €10,000 in capacity payments—payments made simply for being on stand-by—in addition to a rebate on any diesel fuel costs. As CIX and EnerNOC continue working together, and as CIX continues to grow, the data centre will be able to increase the annual revenue it earns for participation in the scheme.
Tier I & II Explained
Both Tier I and Tier II sites are limited by single power and mechanical distribution systems. A typical Tier I example would be a small in-house IT department with single unit cooling capacity, a single generator, a single universal power supply (UPS) backing up the utility supply, and a single power circuit to the servers. A Tier II site has non-redundant subsystem components and a single distribution path servicing the IT floor. It will also have redundant critical power (i.e., an additional generator and UPS plant) and cooling capacity components to reduce the risk of IT process disruptions due to failure of site infrastructure.
Tier III & IV Explained
Tier III infrastructure allows for “concurrent maintenance,” meaning all capacity and distribution components necessary to support IT needs can undergo individual maintenance without impacting the IT environment. Consequently, this entails duplicating the mechanical and power distribution paths in addition to the redundant subsystem component requisites for a Tier II site. Tier IV classification adds the additional requirement of fault tolerance to Tier III, such that each and every system or component that supports IT operations should be capable of experiencing a failure or unscheduled outage on any single component or distribution path at any time, without negatively impacting IT availability.