Questions and answers: my career change to data science (Part 1)

I've been working as a data scientist for some time now. My background is in physics, and my previous work was mostly in education, teaching in the STEM fields. This is an exciting period in my life. Career changes often are, and data science in particular has been called "the sexiest job of the 21st century" - a description that I'll not complain about.

All this excitement has lead to people asking a number of questions about my job - in fact, I recently ran into an old friend who was contemplating a similar kind of career change, and I told her that I loved my new job. She then asked me the following questions:

1) Why do you love being a data scientist? Why does the job fit you like a glove?
2) What do you do day-to-day? What's a typical week like?
3) What were the challenges starting out? What are they now, some time later?
4) What are the best and worst parts of the job?
5) What are the top 3 actions you took to successfully land a job in data science?
6) If you could give career advice to yourself from a couple of years ago, or to a fellow scientist or mathematician now, what would it be?
7) Anything else I'm missing?

So here are my answers:

1) Why do you love being a data scientist? Why does the job fit you like a glove?

Doing data science is like all the fun parts of doing physics, without any of the drudgery. Iteration times are much faster. If I'm curious about something, I can just look in the data, instead of having to go get the liquid nitrogen, and wait for the vacuum system to pump down, and for the coarse approach to finish, etc. I can typically produce something meaningful in hours or days, instead of weeks or months.

I never was very good in the lab, with all the physical equipment. I was better in a more theoretical, abstracted setting. I felt more at home when sitting in front of a computer, or thinking about a math problem on a whiteboard, rather than working with physical lab equipment. Collecting physical data is hard - it's what ended up taking all the time and effort in physics. But in data science, there's already so much data out there that much more of my time could be spent on exploring, processing, analyzing, and acting on that data. I can do things like that for hours or days on end. So a data science job fits well with my talents and temperament.

The lengthy and time-consuming nature of physical data collection was another part of science that I didn't particularly like. It was hard to stay motivated - I can work hard on a problem for an extended period of time, but if there was no real sign of progress after a few days of focused effort, that would be very discouraging to me. This happened to me frequently in physics, and it probably ultimately prevented me from doing well in grad school.

In contrast, data science projects generally give me some real return on my investment within hours of putting in the effort. In addition, working in industry as opposed to academia means that the results of my work impact the real world much more quickly. This higher rate of reward for my efforts helps me stay motivated and is much more suited for my personality.

On a more big-picture level, I like the idea that my primary job is to think soundly about data. That, in the abstract, is a profound activity, closely connected to the depth of the nature of the universe. In that sense I still feel that I am trying to "know the thoughts of God", in much the same way that I felt when I was exploring the "big questions" of physics.

2) What do you do day-to-day? What's a typical week like?

I usually do some programming, some SQL data pulls, visualize data by making graphs, attend meetings, etc. These pieces get put together to answer a data question that's relevant to my employer, like "How effective is this feature in our business in serving our customers?" or "Which of our potential customers would benefit the most from which of our offerings?" I usually work on answering several of these questions per week, with some of the larger questions taking several weeks to answer completely.

There is also an element of teaching and education, as part of the role of a data scientist is helping others understand what the data is saying and what actions we can take in response to it. This is great for me, as it scratches my itch for teaching and personal interactions - more things that I couldn't often get while sitting in the basement in a physics lab.

3) What were the challenges starting out? What are they now, some time later?

Coming from a background in physics, there were some new things I had to pick up. Obviously, I had to ramp up on the programming end. I had picked up a smattering of obscure, little-used programming languages throughout my physics career, but I finally learned more useful languages like Python or JavaScript some time before switching to data science. Apart from general programming, there's also some specific sets of modules used by data scientists, and I had to become proficient in those as well. I also picked up some statistics.

In all these thing my physics background served me well. A great deal of things was just learning to implement the things that I could already think of to do, in a specific development environment. But the foundation of the necessary abstract concepts and the mathematical tools were already in place.

Now that I've been in the field for a while, I'm still challenged to continually improve and add to my skill set. I see more of what's out there now - there's so much to branch out to that it's hard to say what I should learn next. There's also the other, more nebulous, non-technical questions that I have to answer, like "how can I maximize the impact that my work will have in my company?" or "how should I split my time between exploring new ideas and doing what others have asked of me?" I didn't have much time to contemplate these questions when I first started, but I find that they're becoming more important now.

(to be continued in the next post)

---

You may next want to read:
Make the most of your time and your life. Number your days.
Basic Bayesian reasoning: a better way to think (Part 1)
Another post, from the table of contents