Baby Steps About …Ops

Within 3 weeks, I’ve gotten a title “DataOps and Platform Architect,” and let me be honest, that was the coolest title that I’ve ever gotten before. But what is this title? What does this title mean exactly?

Somehow, I should manage data with a pipeline orchestration, monitoring, and maintaining security end-to-end between services. I implement DevOps and its culture in my hobby project, but DataOps is new to me.

So, I got a pen and paper. I’ve tried to create a pipeline to make data more secure, monitorable, and automation-ready. And let me be honest, I’ve failed as hell. Because I didn’t have a clue how to create a pipeline for a service’s output for another service’s input. Automation is not a new place in my tech skills, but you know, I always automate tools and my needs as an engineer, not a pipeline. In that time, my posture was like “The Thinker” statue, and I couldn’t find a way to implement and integrate my skills into the project that I was working on as a DataOps (engineer?). I’ve just taken a breath and created a to-do for the project.

First, understand how GitHub actions actually work
Create many and many workflows, such as manipulating files, triggering an action chain, creating a file, pushing a Dockerfile…
Refresh your mind on Kubernetes and try your test cluster for several purposes
Check the latest methodologies for continuous deployment
Learn to protect data quality in a pipeline
Try to define data standards early
Design for scalability and re-create pipelines if needed
Learn monitoring pipelines
Make practices as much as you can

And, surprisingly, it worked. I got more confident. My mouth was open for planning, developing, integrating, testing and releasing/deploying. I’ve just realized that I've gotten used to the cycle of DataOps step by step. I’m totally new in this place but it’s fun to learn how we should adapt to changing needs.

Asking the right questions is the key because, as a newbie, there was no problem that I couldn't solve with asking questions. Ask questions over and over again. What is the current structure of the data? Has any change been integrated? Is any data missing? What is the idea beyond this microservice?

And, from my experience, how more you use artificial intelligence in your learning process, you are going to forget as fast as possible. Please write down your problems, questions, and solutions in a notebook and read them over and over regularly.

So, DataOps is an umbrella term that includes almost everything about data, automation and operations. Somehow, the task is to create, deliver and manage the data for applications. So to do that, you must learn how to collaborate with other teammates, especially data-related engineers.

In this place, there are many puzzles I have to solve. I'll let you know more things about my journey in DataOps. But for now, Thanks for reading my blog.