Selecting Scientific Tools

This page started as a lab meeting discussion of how to choose among different options for scientific tools (mostly focused on programming).

Question Framing

• When should I learn python/R/Stan/etc.?
• How does {tool x} integrate into my workflow?
• If I don’t learn {tool x}, what am I doing instead?
• What are the benefits for knowing how to use {tool x}
• What are the benefits for learning how to use {tool x}

Identify Alternatives

• some possible scenarios
  • I have a specific research problem that doesn’t appear to solvable with my current toolset
  • Lots of people are using {tool x}, and I don’t want to be left behind
  • No one else is using {tool x}, and I want to be the leading expert in my lab/department/field
• what are some other considerations beyond solving my specific problem
  • speed
  • generality
• how do I find out about other options?
  • google +3
  • twitter +3
  • literature search
  • seminar talks, conferences (people tend to be a little more “in the weeds”, more receptive for engagement & dialogue)
  • workshops
• what are the people around me using?
  • peers, colleagues, collaborators
  • similar enough level for learning/teaching

Evaluate the utility of knowing a new tool

Ok, @cmMcConnaughy is right - enough language war. My language-agnostic concluding thoughts. Choose tools that:
1) Have strong communities around them
2) Are proven but adaptive, so you can use them long-term
3) Facilitate persistent reproducibility, esp. w/ your future self

— Thomas J. Leeper (@thosjleeper) February 4, 2019

properties of communities to consider:

• communities to avoid (non-inclusive, wrong level)
• asking about what communities are set up around tools
• how diverse is the steering committee for the tool
• what are the practices of the community?
• If you are going to spend time in a community, will you be able to grow and take on leadership roles? (is that important for you)

What is the long-term trajectory of a tool:

• is it actively being developed?
• established history of developers
• support for older versions

Evaluate the utility of learning a new tool

• humans are habit-forming
  • defer learning new tools –> (decades later) “I’m too old to learn {tool x}”
  • always learning new tools –> never get around to using anything in practice
• exposure to new communities builds your professional network
• exposure to new ideas gives added context to what you already knew before
• “learning tools” is also a skill! (which means you can practice it and get better)
• “if you don’t use it, you lose it” -

Things to keep in mind

Decision making is difficult (but it is a skill you can practice and improve on!)

• imperfect knowledge

  • no information source is incomplete
  • information sources aren’t completely tuned to your individual situation

• outcomes are imperfectly measured

  • how useful was learning {tool x}? (no ideal way to measure this)
  • no measurement of what would have occurred with an alternative

• limited time/attention

  • spending (more) effort to improve decisions can also be suboptimal

• when to abandon bad decisions (sunk cost fallacy)

Tips

• find good heuristics
  • be aware of biases (cognitive and otherwise)
  • reflect on past decisions (successful and unsuccessful)
• engage in deliberate practice
  • identify appropriate goals
  • evaluate outcomes
  • get targeted feedback
  • repeat
• embrace failure
  • “People don’t fail, processes do”
  • debug your processes (learn from mistakes, aim for long-term growth)
• learn to fail fast
  • for large/complex projects - is {tool x} suitable long-term?
  • test whether the {tool x} addresses the most difficult aspect of the project
    • if this doesn’t work, it is better to know sooner!
• exposure vs. learning
  • goal-oriented or not?