This challenge has been expedited over the last decade, as Big Data snowballed with the rise in use of social media platforms and connected devices. This snowball of data is still in full effect, changing the way in which data is stored. Relying on traditional, on-prem servers is still an option for organizations but will inevitably reach a breaking point. The microchip can only shrink in size insofar as modern science can redefine atomic composition. Just as any other physical object, it will no longer be able to produce the level of computing power that will soon be required of it -- forcing the tech industry to look to more innovative ways to store data.
In floats the cloud, hailed as a solution to the Big Data conundrum, and rightfully so. The cloud does away with the physical limitations and costs of maintaining on-prem servers, offering at times a more affordable data storage solution, depending on the size and scope of the organization utilizing it. Industries across the nation have taken notice, with 83% of workloads expected to be on the cloud by next year.
Data compliance is yet another advantage of migrating to the cloud, and depending on the provider, built-in compliance features such as pseudonyms and extended data storage periods make it simpler to meet GDPR and CCPA requirements, among other compliance laws.
The issue many cloud-eager organizations overlook is ensuring that the kind of central log management they deploy is one that makes data searchable, easy to analyze, and scalable alongside the growth of their business. In terms of tracking and managing events, SIEMs are true to their name, and are effective at detecting actions, potential threats, and overall activity within a network. However, this is where a SIEM’s capabilities stop, and SOAR’s begin.
SOAR goes beyond the means of a traditional SIEM by more intuitively connecting the dots between independent, yet suspicious events across a network, producing an automatic response that shuts down potentially nefarious activity. Both of these platforms do their jobs well, however neither is native to the cloud, making any data that is tracked and stored less apt to be effectively and seamlessly analyzed for future use. Integrating the scalability of the cloud with the detection tools of a SIEM creates a powerful central log management platform that operates with greater agility, speed, and accuracy, allowing for data to be leveraged in a more practical manner.
As data collection and storage practices currently stand in the cyber world, visibility has been placed on the backburner. Data is indeed being stored, but in a rather stagnant, inaccessible manner, often referred to as a “data lake”. The problem with the lake model lays not in capacity, but rather in intelligent pattern detection and the accompanied network orchestration required to leverage it. A more efficient and scalable model is that of a free-flowing data river, with machine learning and data consumption capabilities that grant far greater levels of visibility, and consequently, threat detection. Relying on this model creates an easy-to-navigate network of searchable data, and due to its cloud-based structure and pattern detection features, allows for more specific tracking of who, what, when, where and how information is being produced.
Keeping this in mind, grappling with the onslaught of Big Data and the ever-expanding cloud-based market has become unavoidable for organizations and businesses across all industries. As greater amounts of data are generated by individuals, devices, AI, and more, finding ways to compliantly yet accessibly store data will become inextricably linked to whether an institution survives or thrives in years to come.
Chris Jordan founded Endeavor Security, a cutting-edge, threat detection and analysis company focused on helping enterprises and governments protect their most sensitive networks. Acquired by McAfee in 2009, he then continued with a role as Vice President of Threat Intelligence. Well known for establishing some of the largest government security operations centers, Chris changed his career, starting a security service company in 2003 and a research & development company in 2004. Both companies have since been acquired, and with retiring from McAfee in 2012 founded Fluency with longtime friend and coworker Kun Luo.
