Experiment with overparameteized models in domains that may not necessarily make sense, particularly from a first-principles perspective. One example: I have peer-reviewed a paper where 1-dimensional convolutional neural networks were unleashed on on tabular data: this makes no sense from a first-principles perspective, because in tabular data, there's usually no semantically meaningful spatial correlations between columns. These experiments, to the trained practitioner, leave the impression of the experimenter trying too hard to shoehorn a problem into a newly learned tool, with the model's assumptions not being sufficiently understood before being applied.

In the spirit of Finding the appropriate model to apply is key, I'd rather see models custom-built for each problem. After all, Every well-constructed model is leverage against a problem

I think we need to develop a sense for "when to apply which model".

The key skill here is to be able to look at a problem and very quickly identify what class of problem it falls under, and what model classes are best suited for it.

By problem class, I mean things like:

• Input/inverse design problems
• Supervised learning problems
• Unsupervised learning problems
• Statistical inference problems
• Pure prediction problems

I think that those who claim that "the end of programming is near" likely have a deeply flawed view of how models of all kinds are built.

I think that the place where overparametrized models make the most sense for application are not in models with "high explanatory power" (see The goal of scientific model building is high explanatory power). Rather, their best realms of application are a bit more pedantic. We don't always make these large, overparametrized models to explain the world, per se, but to automate some task that could have been done by a human being. In other words, we come right back to the notion of using computing to automate routine tasks. Basically one of the two classes of tasks:

1. Generating representations of input data, which falls under the umbrella term of Representation Learning.
2. Automation of routine, manual tasks, via methods such as semantic segmentation.

Both of these are things that would have originally required human intervention.

Seems to me, then, that many so-called "AI" applications are merely (yes, merely) about automating tasks that would have otherwise taken much longer with humans.

Other references:

• The end goals of business data science

What are the end goals of research data science?

One of them is being able to explain the world, ideally in a causal fashion. (See: The goal of scientific model building is high explanatory power)

The kind of data scientist we need for this kind of work is different from that of business data science (see: The end goals of business data science).

The context in which research data science operates is one where business processes (and their corresponding business outcomes) are oftentimes not well defined. Thus, defining the ROI is a less straightforward task than it might otherwise be. As such, we need to view research data science as an investment for the future, just like any research organization is.

Also a thing to read: Researchers think mechanistically about the world

We've started off in research data science with a wave of model-focused advancements and applications, particularly into finding applications of deep learning in primarily imaging and natural language fields. These are saturating, I think, as we move forward into the problems that don't have the same set of inductive biases as learning from text and images.

What are the end goals of business data science?

I think one of them is automation, being able to capture value from existing business processes by automating out manual, repetitive tasks, so that humans can be free to do other things.

Data science, as applied directly in most business contexts, usually revolves around business goals that are easily-defined in monetary terms, and where well-established processes are available to optimize. Put bluntly, to be able to save on costs in a business process, or to increase the amount of profit extracted from the business process.

Why does mechanistic thinking matter? In The end goals of research data science, we are in pursuit of the invariants, i.e. knowledge that stands the test of time. (How our business contexts exploit that knowledge for win-win benefit of society and the business is a matter to discuss another day).

When we build models, particularly of natural systems, predictive power matters only in the context of explanatory power, where we can map phenomena of interest to key parameters in a model. For example, in an Autoregressive Hidden Markov Model, the autoregressive coefficient may correspond to a meaningful properly in our research context.

Being able to look at a natural system and find the most appropriate model for the system is a key skill for winning the trust of the non-quantitative researchers that we serve. (ref: Finding the appropriate model to apply is key)

This was inspired by my participation in the TAO Data Science Panel.

I'm starting to see a bifurcation in research vs business data science.

• Research vs Business Data Science

How this translates to training needs and hiring

• Researchers think mechanistically about the world
• Learn how to learn fast
• Operate outside your pay grade
• Data scientists should know the data generating process behind their data
• Data scientists should learn how to write good code
• Avoiding technical debt with ML pipelines

And notes for managing data scientists:

• How to motivate your problem solvers