Folks seem to instantly understand "GitOps" when they hear the word, even without reviewing a case study or reading any publication by an expert. I think this phenomenon stems from thinking along the lines of: "Git is central to the processes in my operations environment, so what we do must be GitOps." To avoid this misguided thought process, I recommend a free book called The Path to GitOps by Christian Hernandez. The book is very concise, but still I will highlight some sections that I find to be the most overlooked in case you want to drill right into the content that will challenge your intuition.
Principles
Christian highlights 4 principles of GitOps: Declarative, Versioned and Immutable, Pulled Automatically, and Continuously Reconciled. Of the four, I think only one is generally well understood: Versioned and Immutable. Here is what I think is misunderstood about the others:
Declarative: This is another word that lulls folks into a false sense of understanding. "It's represented with YAML, of course it's declarative!". Well, this depends on what state you are describing. You might have a declarative representation of a CI/CD pipeline, but if that pipeline is procedural (and most are), then you do not have a declarative representation of the state that pipeline is intended to achieve. The book describes the term declarative quite eloquently:
The state of a system is stored as a set of declarations without procedures for how that state will be achieved.
Pulled Automatically: I often find organizations using a tool like ArgoCD as if it is a replacement for kubectl. This is quite convenient for converting a traditional CI/CD pipeline into "GitOps" in the same way that Russian election ceremonies have successfully converted the nation into a thriving democracy.
GitOps software agents (which I sometimes call GitOps controllers) check the desired state by pulling declarations from the state store at regular intervals, which means polling as well as pulling
Continuously Reconciled: Quite simply, this is GitOps nirvana. GitOps means observation of both the desired and actual state. Reconciliation occurs whenever there is divergence.
GitOps triggers reconciliation whenever there is a divergence
GitOps and CI/CD: GitOps cannot possibly replace CI/CD in its entirety, which usually must include some procedural processing. As Christian puts it, GitOps participates in CI/CD.
Under GitOps, the CI process just needs either to make a commit directly to your state store or to mark that the change is ready for deployment in a Git repository.
Git Workflows
I find organizations most resistant to the following Git-workflow related suggestions.
Separate Your Repositories:
The first is to keep your application code in a separate repository from your YAML configurations. This might seem counterintuitive initially, but most teams that start with code and configurations together quickly learn it’s better to separate them.
Separate Development in Directories, Not Branches:
Another best practice that surprises many programmers is to separate environments– such as test and production–into different directories, but not create branches for them.
Trunk-based Development:
The recommended workflow for implementing GitOps with Kubernetes manifests is known as trunk-based development. This method defines one branch as the “trunk” and carries out development on each environment in a different short-lived branch.
CI/CD with GitOps
CI and CD Can Be Decoupled: I think this is one of the most difficult habits to break.
People typically glom CI and CD together, because popular DevOps tools like Jenkins aim to do both. Jenkins has successfully created a platform to manage both CI and CD using a single management system. But with the advent of Kubernetes, microservices, and cloud-native architecture, many paradigms are starting to be decoupled, including CI/CD.
A Mindset Shift:
[W]hen you’re thinking about promoting your application from environment to environment, think about it instead as promoting manifests. Because Kubernetes relies on immutable containers, the code isn’t what really gets promoted; rather, it’s the desired state of the application running on the cluster. This is a subtle change in mindset, but an important one.
Conclusion
I really hope you read Christian's book The Path to GitOps. Articles about GitOps are everywhere, and there's so much vendor-washing, it can be tough to cut through the BS. While this book is sponsored by Red Hat, it's not a giant advertisement for some proprietary tool they want you to buy, it is a concise and eloquent description of GitOps and the most important concepts therein.
Oh and as promised by my click-bait-y title: What is the hardest part about GitOps? Having the humility to admit to yourself that you might not know what it means.