Cracking the Code to a High-impact Career in Data

How do you navigate a career in data with all the hype and the information overload? How do you identify the breadth and depth of skills to acquire and master? How much time to invest in learning? How do you grow in your current role? How do you prevent yourself from going around in circles like a dog chasing its tail?

Well, I went from being a siloed data and software engineer to leading technical product teams. I could not believe how simple it was as a mental model. By simple I don’t mean easy; I mean clear and achievable.  All it took was focused effort and direction.

Navigating professional growth has its complexities. Market conditions are always in flux. It is even more complex in technology, data, and growth.

New ideas come up at an unprecedented pace and scale. Tools and technology vendors, consultants, and competition among the bigger players fuel these ideas to products and features that ride the hype curve – it’s a never-ending cycle.

By the end of this short article, you’ll have a solid mental model to approach this problem and adapt the solution to your unique situation. This is the post I needed as I navigated my data career in 2010 and beyond.

Data career expectations

You will encounter tremendous hype on data roles, including high pay, and skill shortages. This is true in other areas too but our focus is on data. Database administration, data engineering, data analysis, data science, and similar labels are touted as a dream career.

The first thing to do is to get grounded and specific on a basic set of things that you would like from your career and things that you would NOT like. In my experience, it is easier to start with things that you don’t wish for, than to be specific about what you want. Try to start with a list of options based on current skill levels and things that you are curious about.

Trust but verify

Whether it’s roles, skills, or technologies, there will always be noise about new and popular ideas disguised as game changers and silver bullets.

By default don’t take anyone’s word for anything, and validate claims empirically before buying into the ideas. Always begin by evaluating who is presenting the idea and what is their incentive. Embrace a healthy skepticism and spend a bit of time thinking about the counterarguments to popular ideas online.

For example, you can compare and contrast software engineering, data engineering, data analysis, and data science. What are the common patterns in these roles, and what are the differences in the expected outcomes, baseline day-to-day responsibilities, challenges, and risks? Talk to some folks in the roles to get insights from someone who is further along in their career.

Apply the same parameters to this article. Now that we have the basics let me share the mindset power-up.

The code to strategic thinking

A severe gap in most data careers is strategic thinking. It’s not that data folks are not strategic, but there is too much work to do. Too many ad-hoc requests. Too many fires. And insufficient business context.

But strategic thinking is expected of folks working with data by default.

And it’s an incredible power-up to a data professional’s career to talk through different scenarios on the board, the different positions and moves, and the possible results.

Trends and hypes are all about change. But long-term strategy needs farsight – it’s based on things that don’t change as often.

There are two tools that you can use to build your strategic muscle: working backward and zooming in and out.

Working backward

Working backward is famously associated with Amazon, but many others have embraced this mental model. It relies on deeply understanding the context and the pains of the consumer, translating them into tangible ideas and workflows that can potentially solve those problems, and finally, bringing in the requisite technology and data.

The first step of working backward is to outline a solution to unsolved consumer problems in a market.

Focus on:

• Getting the basics right.

• Clear articulation of problems.

• Consistent and predictable execution.

Deeply understand:

• The customer

• The business

• The market

• The data

Spend a couple of hours each week learning about the market. I prefer searching news aggregators like Google News with long tail keywords like “Healthcare IT data challenges,” “eCommerce analytics market reports” etc. to read about the market that is relevant to my product and business.

Spend a couple of hours digging into customer support tickets, customer success reports, and sales reports. Ask customer-facing colleagues in growth, product management, and customer success about the customer problems, and needs. Ask business-focused colleagues in operations, and finance about the business. Learn about the challenges and difficulties.

When working on build requests, set aside some time to consider the economics. 

How much are the consumers willing to invest? How much do they expect in return? This is basic demand economics. To figure out the supply, you need to know what data and technology are needed to create the insights that the market demands. And you need to estimate the cost. Cost of talent, cost of infrastructure, and so on

Zooming in and out

Prevent garbage in garbage out. Adjust your lens.

Zoom in when you are estimating complexity, time to build, test, and ship. Zoom out when you are designing user workflows, looking at use case patterns, and identifying valuable problems.

Zoom in when you are evaluating data quality, privacy, and regulatory compliance. Zoom out when you are considering the architecture of the data platform and the systems that need to interoperate for things to work.

In my experience, the majority of data problems are caused by applying the wrong lens to solving misunderstood problems.

What has not changed in data

Here are 5 elements in data that have remained unchanged over the past 3 decades of software and web:

Customer Needs: Consumers of data demand complete datasets to analyze and make informed decisions. They're not interested in black boxes that they can’t understand. Also, no one wants to pay more and get less in return.

Stakeholder Needs: With the constant buzz around data-related innovation, stakeholders have high and sometimes unrealistic expectations. This can be a powerful motivator if managed effectively. It also presents an opportunity to calibrate data savviness through hands-on learning experiences. Ultimately, leaders are accountable for profitable returns on their investments.

Product Needs: Quality, scalability, reliability, and security (including privacy) remain perennial priorities. As technology advances, these expectations will only become more demanding, particularly with the growing emphasis on privacy and ethical considerations. Systems must be reliable at scale while delivering accurate insights without compromising privacy or trust.

Unit Economics: While technology has improved and costs have been optimized, the basic economic principles of cost-benefit analysis, supply, and demand remain unchanged. The value of datasets lies in their enrichment and granularity. Combining data from multiple sources at scale is not simple or cheap.

Capacity Constraints: The gap between capacity, talent pool, and advanced skill set persists. There are no shortcuts to address this issue; effective professional development, performance management, and organizational design are key.

Whatever your data organization model, remember that a consistent pattern is crucial. Power it with an engine that minimizes entropy (waste) for optimal results.

Conclusion: 3 muscles to build with practice

1. Leverage consistent professional development.
2. Prioritize a deep understanding of the market and the customers.
3. Strive for simplicity in your tech stack and make it a priority to collaborate across the organization.

When your system is in place, with an efficient engine powering the flywheel, your organization will not only be well-positioned to capitalize on relevant trends but also set new ones.

{{line}}

Join our community to learn from Deb and other data leaders!