Don’t Be Scared Of GDPR
Regardless of where your business is physically located, if your sales and marketing efforts target people resident in the European Union you’re already aware of the EU’s General Data Protection Regulation, or GDPR, and its implications for your business. What I’m interested in is how GDPR affects business marketing – for better or worse. The vast majority of GDPR concerns the way companies use, protect, store, and keep personal customer data. Not only does it fundamentally change what marketers can and cannot do with data, but it will penalize companies that misuse that information. So how did we get here? Why do we need the GDPR? The answer is simple: Because of all of us having to deal with crappy marketers doing crappy marketing for the past decade or so. With GDPR in place, any consumer resident in the European Union has the right to request a copy of the data any organization holds on them. Any business sending marketing messages to EU individuals needs to prove receipt of explicit consent that they can use the data for marketing purposes. So what’s the flip-side of the coin? By having all of this data open, harmonized, and accessible to the appropriate people offers the possibility of a whole new level of next-gen marketing tech innovations. Yes, GDPR is there to prevent unscrupulous businesses abusing customer data. GDPR is, in effect, creating a de facto worldwide standard for consumer data protection. With users having the ultimate say in how their data is used, the quality and relevance of marketing campaigns will have no choice but to improve. Blanket email blasts using rented third-party data and poorly-considered retargeting campaigns will be swiftly punished by consumers exercising their new data rights. Consumers are more likely to accept better, more targeted, more relevant marketing messages if they know they can turn the tap off quickly and easily if they wish. The adoption of new marketing technologies and innovations is just a small part of what GDPR promises. For businesses of every shape and size, GDPR provides perhaps the single biggest opportunity to improve marketing performance since the introduction of broadband internet.
Handling GDPR with Apache Kafka: How does a log forget?
If you follow the press around Apache Kafka you’ll probably know it’s pretty good at tracking and retaining messages, but sometimes removing messages is important too. This raises a very obvious question: how do you delete arbitrary data from Kafka? After all, its underlying storage mechanism is an immutable log. The simplest way to remove messages from Kafka is to simply let them expire. Businesses increasingly want to leverage Kafka’s ability to keep data for longer periods of time, say for Event Sourcing. In such cases it’s important to understand how to make long lived data in Kafka GDPR compliant. Deleting a message from a compacted topic is as simple as writing a new message to the topic with the key you want to delete and a null value. To ‘forget’ a customer, simply lookup their Orders and either explicitly delete them from Kafka, or alternatively redact any customer information they contain. You might roll this into a process of your own, or you might do it using Kafka Streams if you are so inclined. There is a less common case, which is worth mentioning, where the key (which Kafka uses for ordering) is completely different to the key you want to be able to delete by. Then you can delete messages using the mechanism discussed earlier using the [ProductId][CustomerId] pair as the key. Quite often you’ll be in a pipeline where Kafka is moving data from one database to another using Kafka Connectors. If you’re using CDC this will just work: the delete will be picked up by the source Connector, propagated through Kafka and deleted in the sinks. One final consideration is that partitions in Kafka are made from a set of files, called segments, and the latest segment isn’t considered for compaction. Kafka provides immutable topics where entries are expired after some configured time, compacted topics where messages with specific keys can be flagged for deletion and the ability to propagate deletes from database to database with CDC enabled Connectors. Despite being built on immutable logs as its fundamental underlying abstraction, Kafka provides tools that accommodate GDPR requirements easily and elegantly.
The problem is about to get a whole lot worse, as the soon-to-be-enforced General Data Protection Regulation will turn the skills gap into a chasm. Under GDPR, your business could be one mistake away from a breach that could cost you up to €20million or 4% of your global revenue. A fine-worthy breach includes data hacks, loss or misuse of data. To make the issue even worse, to prepare for GDPR itself you’re going to need staff that are adequately trained in data protection, data management and GDPR compliance. As GDPR affects nearly every company in the EU, people who have knowledge of all of the above are going to be in extremely high demand. A quick search online brings up several GDPR training courses you can send your IT team and other technical staff on so they can get clued up on GDPR and its requirements. Some businesses will need a dedicated Data Protection Officer. There is the further issue of your staff potentially leaking customer data, misusing it or storing it incorrectly. As part of your GDPR preparations, you will have to ensure all staff are aware of GDPR, its implications and what GDPR-compliance looks like. You’ll have to go into detail over what constitutes a breach, as well as put in place policies on bring-your-own equipment and data governance that all staff will have to be trained in. You should consider holding a few different training sessions with your employees based on how tech literate they are and how clued up they are on GDPR. You’ll also have to schedule in regular refresher sessions in case anything changes and to really ensure compliance and include GDPR in induction sessions for new employees. Organiztations should focus on solid data infrastructures. Between setting up employee training and finding yourself a DPO, it’s very easy to forget about the main preparations for GDPR readiness. You’ll need to carry out a data audit to ensure all your data is stored correctly and securely, is easily transferable when requested and has all the required consent. There’s a significant amount of work to be put in before the May 2018 deadline when GDPR is enforced.