When Dark Data Turns Darker:
Data’s Impact on Climate Change
As the world continues to debate how to address the existential threat from climate change, an unlikely and often overlooked threat hides in the data storage shadows: it is known as dark data. This giant mass of unused data, stored indefinitely and often needlessly, reduces business efficiency, increases the chances and seriousness of security issues, detracts from the true value of data, and contributes negatively to environmental issues in ways that might surprise you.
In a corporate context, businesses have moral and, increasingly, legal obligations to do better when it comes to the environment, but there’s also a strong financial incentive as businesses with strong ESG initiatives are quite rightly seen as forward thinking and are therefore more attractive to certain investors (we could of course debate the impact and indeed authenticity of some of these efforts but let’s look past that for now).
Tackling environmental issues in some organisations from a digital or data delivery perspective rarely goes further than optimising the front-end of a website or perhaps in more mature organisations lightweight ‘serverless’ architecture is more of a consideration; however, in this blog post I want to enlighten you about the more sinister side of dark data (an issue for almost all organisations) and the uncomfortable truth about its impact on our planet, notwithstanding the potential business intelligence being left unexploited.
Between 60%-80% of data is dark or unused
Experts from Clouded II documentary
“In Ireland, data centers are using almost 20% of all electricity. It’s an extreme example, but it’s a warning sign of where things could head if not carefully managed.”
Ishmael Burdeau,
Lead Business Sustainability Architect
DWP Digital
The Darkest Side of Data Storage
Dark data includes vast amounts of data assets collected by organisations that never gets analysed or used downstream. Think of all the emails, documents, data files, videos, and transactions that are saved “just in case.” The environmental footprint of this unused data is staggering. The energy needed to store and maintain this dark data contributes directly to increased carbon emissions.
The environmental footprint of data centers worldwide is becoming increasingly alarming and is now responsible for approximately 3% of global emissions — a figure comparable to that of the airline industry. This burden is expected to grow as the global reliance on data continues to expand.
Cloud migration, which is often thought of in the industry as a greener solution than on-premise, is not immune to this problem. While cloud services offer efficiencies and potential reductions in an organisations’ carbon footprint, they also become repositories for an ever-growing amount of dark data. Without strategic data management, the migration to Cloud – especially if performed as a lift and shift and not transformed using serverless – can shift the burden of dark data rather than reduce it.
The Energy Cost of Failing to Act
The proliferation of dark data is not only a symptom of poor data management, but a significant environmental issue. Data centres, which power the internet and store our data, consume a colossal amount of energy. According to a recent report from the International Energy Agency (IEA), data centres worldwide consumed about 220 terawatt-hours (TWh) of electricity in 2023, representing roughly 1.3% of global electricity demand. The storage of dark data clearly makes this problem worse by driving up energy consumption.
To put this into perspective, the IDC produced a report back in 2018 that said data worldwide is expected to grow to 175 zettabytes by 2025. The report said: “If a single zettabyte were stored on DVDs, the stack of DVDs would circle the Earth 222 times”. With such rapid data growth estimates appearing conservative the environmental impact of dark data has become much worse than expected and that trend is expected to worsen since AI has become mainstream (the size of the dataset AI requires is staggering).
The hyper-scale cloud vendors have taken steps to address the carbon footprint of their data centres. For example, Microsoft has committed to becoming carbon negative by 2030, with plans to remove all the carbon it has emitted either directly or through electricity use since its founding by 2050. Despite these efforts, the issue of dark data is still pervasive and is largely unaddressed. However, recent data suggests that these goals may not be met and Microsoft now admits they cannot meet their 2030 goals due to the energy-intensive requirements of AI and data growth. Furthermore, the environmental impact of data centers could be much higher than officially reported. According to a report from The Guardian, some data center gas emissions are 662% higher than what they publish.
Organisations can unlock insights and drive business value from dark data by integrating advanced analytics and AI tools.
Unlocking Hidden Value
Dark data isn’t just a burden to organisations; it holds huge, untapped potential value if correctly managed and exploited. Organisations can unlock insights and drive business value from dark data by integrating advanced analytics and AI tools. This process can support environmental goals by identifying unnecessary data and eliminating it.
For example, our team worked with a client to improve their health and safety practices by leveraging previously unused data. We analysed extensive data sets, including health data, sensor data, incident data, and location data, to optimise the amount of time spent using equipment that could cause fatigue or injury, optimising scheduling routines, and enabling strategic decision making – ultimately maintaining compliance and worker safety. This approach not only boosted their health and safety results but also supported sustainability by using existing data and reducing waste.
GreenOps: A Solution to Dark Data
To reduce the environmental impact of dark data, organisations must adopt more sustainable data practices – grounded in robust data governance and data management practices. This should start with an objective assessment of what data is necessary, rolling out data lifecycle management policies, performing regular data audits, and pivoting to more responsible data retention practices.
This is where GreenOps approaches can really help. GreenOps focuses on optimising operational efficiency while minimising environmental impact. It encompasses practices such as efficient resource usage, energy-saving measures, and the reduction of waste. When applied to data management, GreenOps can help organisations tackle the problem of dark data by promoting sustainable data practices.
A crucial component of GreenOps is the integration of FinOps principles. By aligning financial incentives with sustainable practices, FinOps encourages organisations to consider the cost of storing and managing dark data. This dual approach ensures that both financial and environmental impacts are reduced.
Hyper-scale Cloud providers like Google, Amazon, and Microsoft provide tools like Intelligent-Tiering, which automatically moves data to the most cost-effective storage tier based on changing access or usage patterns, thereby reducing the environmental impact. These technology advancements highlight the potential for technology to play a key role in managing dark data more sustainably.
It’s vital to adopt sustainable data practices and policies to address the environmental damage caused by our exponential reliance on data assets.
The Role of Regulation and Corporate Responsibility
Regulators must mandate more rigorous data management standards. As with other environmental regulations, there should be a framework that encourages or even forces organisations to reduce their “dark data footprint”. Policies could include incentives for companies that demonstrate a commitment to reducing their digital waste and penalties for those that do not address this issue.
Corporate responsibility must extend beyond compliance. Organisations must understand that their data practices have far-reaching implications for the environment. By investing in more efficient data management technologies and fostering a culture of sustainability, organisations can play a pivotal role in combating the environmental impacts of dark data.
It’s Time to Act
The fight against climate change requires addressing every source of emissions, including those in our data centres. Dark data’s impact has been overlooked. As we move into a scaled, AI-enabled future that’s fuelled by data, it’s vital to adopt sustainable data practices and policies to address the environmental damage caused by our exponential reliance on data assets.
We need to move past just optimising websites and stylesheets (this is akin to the focus on front-end UI veneers that we experienced during the Covid pandemic, which was a missed opportunity!) to tackling the harder yet more impactful challenges, such as addressing dark data, designing efficient, lighter weight architecture that carries a reduced footprint, and leveraging data and AI (effective data management is an essential enabler to tackle dark data) as a means of identifying and optimising energy use in organisations e.g., For Pivotl, we built an environmental data dashboard that pulls into various data sets and layers the data against industry standard emissions reporting standards.
GreenOps offers a potential solution by integrating financial and operational efficiencies with sustainability goals. By embracing GreenOps, organisations can ensure that their digital and data practices are both economically and environmentally on point, enabling a more sustainable future and a more performant organisation.
