In the ever-evolving landscape of software development, one truth remains constant: outages and issues are inevitable. However, the frustration that comes from debugging code—especially when it's not your own—can often feel like an insurmountable challenge.
This article, inspired by my journey and experiences, aims to provide an approach for software developers and engineers, who are often not in the decision-making seat, on how they can advocate for better observability practices in their team or organization.
My Observability Journey
My curiosity about observability was ignited when my team started instrumenting some of our code for metrics, introducing me to the term. Eager to learn more, I delved into research, leading me to attend KubeCon NA 2023 and Observability Day. There, I explored how various organizations were implementing observability, the benefits they gained, and the practices they adopted. This journey brought me to OpenTelemetry and the OTEL end-user working group, where I joined a vibrant community of end users. Returning to my team armed with these new insights and perspective, I recognized a significant gap in our observability practices. Realizing the untapped potential, I've have been advocating for enhanced observability practices within my team ever since.
Developers' Nightmare
As software developers, we find ourselves in a constant bug-fix mode, trapped in a cycle where solving one problem often introduces another. This cycle is not just frustrating; it's a significant drain on our time, resources, and morale.
The lack of adequate tools and practices for troubleshooting also means we're often the last to know when our systems fail, learning about issues only when customers report them. This reactive stance not only damages customer trust but also hinders our ability to maintain and improve system reliability.
Envisioning The Ideal
Imagine a world where developers have access to comprehensive telemetry data at their fingertips. In this world, metrics alert us to problems, traces guide us to their source, and logs provide the context needed to understand and resolve issues swiftly. This world is not a fantasy; it's attainable through the adoption of observability practices and standards, such as OpenTelemetry, which offers seamless integration with a plethora of observability tools, allowing for flexibility and ease of switching as needs evolve.
From Dream To Reality
Transitioning from a problem-laden environment to one enriched with observability practices can be daunting, especially when your management team is not aware of observability and your pain points, and is not willing to invest money or effort into adopting these practices. How can you then convince them and secure buy-in?
Here is a simple 3-step approach that can help you drive your point across.
Understand
The first step is to deeply understand your team, your code, and the observability practices that can address your specific challenges. This understanding forms the foundation for effective advocacy, enabling you to communicate the value of observability in terms that resonate with your team and management.
Proof
Next, create a Proof of Concept (PoC) with one or more of your services. This PoC serves as tangible evidence of the benefits of observability, demonstrating its potential to improve troubleshooting efficiency, enhance system reliability, and reduce the time spent in bug-fix cycles.
Plan
Finally, armed with your PoC and a deep understanding of observability's benefits, create a detailed proposal for your team. This plan should outline the telemetry data you aim to collect, the tools you'll use, and how these tools will integrate into your existing systems. Highlight the benefits to management, such as improved customer retention and confidence, while also considering cost and the practicalities of adoption. Also consider pitching this plan to your team mates first since they also feel the pain of not having these tools and wining them on your side will provide better amnesty when advocating to management.
Conclusion
The journey to adopting observability is not without its challenges, from overcoming resistance to convincing management of its value. However, the rewards—increased system transparency, improved troubleshooting efficiency, and enhanced developer satisfaction—are well worth the effort. By understanding the landscape, proving the concept, and planning strategically, we can make observability an integral part of our development process, moving from a world of reactive fixes to one of proactive insight and improvement.