Understand the hierarchy of the Data Process

 

Kubeflow architecture

 

Model development life cycle

 

The Zen of Python, by Tim Peters

• Beautiful is better than ugly.
• Explicit is better than implicit.
• Simple is better than complex.
• Complex is better than complicated.
• Flat is better than nested.
• Sparse is better than dense.
• Readability counts.
• Special cases aren't special enough to break the rules.
• Although practicality beats purity.
• Errors should never pass silently.
• Unless explicitly silenced.
• In the face of ambiguity, refuse the temptation to guess.
• There should be one—and preferably only one—obvious way to do it.
• Although that way may not be obvious at first unless you're Dutch.
• Now is better than never.
• Although never is often better than *right* now.
• If the implementation is hard to explain, it's a bad idea.
• If the implementation is easy to explain, it may be a good idea.
• Namespaces are one honking great idea—let's do more of those!

TOGAF

 

Data-Driven Decision

 

Decision Problems

A decision problem consists of a specification of a subset of the possible instances. Given an instance, one is required to determine whether the instance is in the specified set (e.g., the set of prime numbers, the set of connected graphs, or the set of sorted sequences). For example, consider the problem where one is given a natural number, and is asked to determine whether or not the number is a prime. One important case, which corresponds to the aforementioned search problems, is the case of the set of instances having a solution; that is, for any binary relation R ⊆ {0, 1}∗ × {0, 1}∗ we consider the set {x : R(x) = ∅}. Indeed, being able to determine whether or not a solution exists is a prerequisite to being able to solve the corresponding search problem (as per Definition 1.1). In general, decision problems refer to the natural task of making a binary decision, a task that is not uncommon in daily life (e.g., determining whether a traffic light is red). In any case, in the following definition of solving decision problems, the potential solver is again a function; that is, in this case the solver is a Boolean function, which is supposed to indicate membership in a predetermined set.

Search Problems

A search problem consists of a specification of a set of valid solutions (possibly an empty one) for each possible instance. That is, given an instance, one is required to find a corresponding solution (or to determine that no such solution exists). For example, consider the problem in which one is given a system of equations and is asked to find a valid solution. Needless to say, much of computer science is concerned with solving various search problems (e.g., finding shortest paths in a graph, sorting a list of numbers, finding an occurrence of a given pattern in a given string, etc.). Furthermore, search problems correspond to the daily notion of “solving a problem” (e.g., finding one’s way between two locations), and thus a discussion of the possibility and complexity of solving search problems corresponds to the natural concerns of most people.

Polynomial time and exponential time

 Below are some common Big-O functions while analyzing algorithms.

• O(1) - constant time
• O(log(n)) - logarithmic time
• O((log(n))^c) - polylogarithmic time
• O(n) - linear time
• O(n²) - quadratic time
• O(n^c) - polynomial time
• O(cⁿ) - exponential time
• O(n!) - factorial time

(n = size of input, c = some constant)

Here is the model graph representing Big-O complexity of some functions

O(1) = O(yeah)
O(logn) = O(nice)
O(n) = O(k)
O(n^2) = O(my)
O(2^n) = O(no)
O(n!) = O(mg)
O(n^n) = O(sh*t!)

Algorithm Design

Algorithmic problems form the heart of computer science, but they rarely arrive as cleanly packaged, mathematically precise questions. Rather, they tend to come bundled together with lots of messy, application-specific detail, some of it essential, some of it extraneous. As a result, the algorithmic enterprise consists of two fundamental components: the task of getting to the mathematically clean core of a problem, and then the task of identifying the appropriate algorithm design techniques, based on the structure of the problem. These two components interact: the more comfortable one is with the full array of possible design techniques, the more one starts to recognize the clean formulations that lie within messy problems out in the world. At their most effective, then, algorithmic ideas do not just provide solutions to well-posed problems; they form the language that lets you cleanly express the underlying questions.

Computational Neuroscience

Computational neuroscience is an approach to understanding the information content of neural signals by modeling the nervous system at many different structural scales, including the biophysical, the circuit, and the systems levels. Computer simulations of neurons and neural networks are complementary to traditional techniques in neuroscience. This book series welcomes contributions that link theoretical studies with experimental approaches to understanding information processing in the nervous system. Areas and topics of particular interest include biophysicalmechanisms for computation in neurons, computer simulations of neural circuits, models of learning, representation of sensory information in neural networks, systems models of sensory-motor integration, and computational analysis of problems in biological sensing, motor control, and perception.

The Butterfly

 

Thinking is more interesting than knowing, but lee interesting than looking.

 

“The explosive freedom we gain from describing reality in mathematical

metaphors is breathtaking. We can discover truths about ourselves that we

could never have learned as poets writing in English. ”

Paul Rapp, Get Smart: Controlling Chaos, 1989

On the Role of Counterfactuals in Learning

The following is a hypothesis regarding the purpose of counterfactual reasoning (particularly in humans). It builds on Judea Pearl's three-rung Ladder of Causation (see below).

One important takeaway from this hypothesis is that counterfactuals really only make sense in the context of computationally bounded agents.

Summary

Counterfactuals provide initializations for use in MCMC sampling.

Preliminary Definitions

Association (model-free):

$$Pr(Y=y\mid X=x)$$

Intervention/Hypothetical (model-based):

$$Pr(Y=y\mid do(X=x\left)\right)$$

Counterfactual (model-based):

$$Pr(Y=y\mid do(X=x),Y=y^{\prime })$$

In the counterfactual, we have already observed an outcome $y^{\prime }$ but wish to reason about the probability of observing another outcome $y$ (possibly the same as $y^{\prime }$) under $do(X=x)$.

Note: Below, I use the terms "model" and "causal network" interchangeably.  Also, an "experience" is an observation of a causal network in action.

Assumptions

1. Real-world systems are highly complex, often with many causal factors influencing system dynamics.
2. Humans minds are computationally bounded (in time, memory, and precision).
3. Humans do not naturally think in terms of continuous probabilities; they think in terms of discrete outcomes and their relative likelihoods.

Relevant Literature:

Lieder, F., Griffiths, T. L., Huys, Q. J., & Goodman, N. D. (2018). The anchoring bias reflects rational use of cognitive resources. Psychonomic bulletin & review, 25(1), 322-349.

Sanborn, A. N., & Chater, N. (2016). Bayesian brains without probabilities. Trends in cognitive sciences, 20(12), 883-893.

Theory

Claim 1.

From a notational perspective, in going from a hypothetical to a counterfactual, the generalization lies solely in the ability to reason about a concrete scenario starting from an alternative scenario (the counterfactual).  In theory, given infinite computational resources, the do-operator can, on its own, reason forward about anything by considering only hypotheticals.  Thus, a counterfactual would be an inadmissible object under such circumstances.  (Perfect knowledge of the system is not required if one can specify a prior.  All that is required is sufficient computational resources.)

Corollary 1.1.

Counterfactuals are only useful when operating with limited computational resources, where "limited" is defined relative to the agent doing the reasoning and the constraints they face (e.g., limited time to make a decision, inability to hold enough items in memory, and any such combinations of these constraints).

Corollary 1.2.

If model-based hypothetical reasoning (i.e. "simulating") is a sufficient tool to resolve all human decisions, then all of our experiences/observations should go toward building a model that is as accessible and accurate as possible, given our computational limitations.

By Assumption 1, the vast majority of human decision-making theoretically consists in reasoning about a "large" number of causal interactions at once, where “large” here means an amount that is beyond the bounds of the human mind (Assumption 2).  Thus, by Claim 1, we are in the regime where counterfactuals are useful.  But in what way are they useful?

By Corollary 1.2, we wish to build a useful model based upon our experiences.  A useful model is one that is as predictively accurate as possible while still being accessible (i.e. interpretable) by the human mind.  Given that: (1) a model is describable as data, (2) the most data can be stored in our brains in the form of long-term memory, and (3) the maximal predictive accuracy of a model is a non-decreasing function of its description length, then a maximally predictive model is one that is stored in our long-term memory.  However, human working memory is limited in capacity relative to long-term memory.

Claim 2.

The above are competing factors: A more descriptive (and predictive) model (represented by more data) may fit in long-term memory, but due to a limited working memory, it may be inaccessible (at least in a way that leverages its full capabilities).  Thus, attentional mechanisms are required to guide our retrieval of subcomponents of the full model to load into working memory.

Again, by Assumptions 1, 2, our models are approximate — both inaccurate and incomplete.  Thus, we wish to improve our models by integrating over our entire experiences.  This equates to computing the following posterior distribution:

$Pr(causalnetwork\mid experience)$

$$=\frac{Pr(experience\mid causalnetwork)\times Pr\left(causalnetwork\right)}{Pr\left(experience\right)}$$

By Assumption 3, humans cannot compute updates to their priors according to the above formula.

Claim 3.

Humans do something akin to MCMC sampling to approximate the above posterior.  Because MCMC methods (e.g., Gibbs sampling, Metropolis-Hastings) systematically explore the space of models in a local and incremental manner (e.g., by conditioning on all but one variable in Gibbs sampling, or by taking local steps in model space in Metropolis-Hastings) AND only require reasoning via likelihood ratios (Assumption 3), we can overcome the constraints imposed by our limited working memory and still manage to update models that fit in long-term memory but not entirely in working memory.

MCMC methods require initialization (i.e. a sample to start from).

Claim 4.

Counterfactuals provide this initialization.  Given that our model is built up entirely of true samples of the world, our aims is to interpolate between these samples.  (We don't really have a prior at birth on the ground-truth causal network on which the world operates.)  Thus, we can only trust our model with 100% credibility at observed samples.  Furthermore, by Assumption 2, we are pressured to minimize time to convergence of any MCMC method.  Hence, the best we can do is to begin the MCMC sampling procedure starting from a point that we know belongs in the support of the distribution (and likely in a region of high density). 

From the Metropolis-Hastings Wikipedia:

Although the Markov chain eventually converges to the desired distribution, the initial samples may follow a very different distribution, especially if the starting point is in a region of low density. As a result, a burn-in period is typically necessary.

Counterfactuals allow us to avoid the need for any costly burn-in phase.