Improving models continually with online methods

The Rosetta Stone is the Science

Interstellar Explained

Increasing bandwidth parameter

Optimizing model performance

• Tuning the model parameters—ML algorithms are configured with parameters specific to the underlying algorithm, and the optimal value of these parameters often depends on the type and structure of the data. The value of each parameter, or any of them combined, can have an impact on the performance of the model. We introduce various ways to find and select the best parameter values, and show how this can help in determining the best algorithm for the dataset in question.
• Selecting a subset of features—Many ML problems include a large number of features, and the noise from those features can sometimes make it hard for the algorithm to find the real signal in the data, even though they might still be informative on their own. For many ML problems, having a lot of data is a good thing; but it can sometimes be a curse. And because you don’t know beforehand when this will affect your model performance, you have to carefully determine the features that make up the most general and accurate model.
• Preprocessing the data—If you search the internet for machine-learning datasets, you’ll find easy-to-use datasets that many ML algorithms can be quickly applied to. Most real-world datasets, however, aren’t in such a clean state, and you’ll have to perform cleaning and processing, a process widely referred to as data munging or data wrangling. The dataset may include names that are spelled differently, although they refer to the same entity, or have missing or incorrect values, and these things can hurt the performance of the model. It may sound like edge cases, but you’ll be surprised how often this happens even in sophisticated, data-driven organizations.

Cosmos

Hypatia - The Last Light of Alexandria

Standard Deviation vs Standard Error

Varieties of regression analysis

O cérebro primitivo no controle da sobrevivência

• ritual
• agressão
• submissão
• territorialidade
• rejeição aos forasteiros
• hierarquia de dominância

Time dilation visualized with light clocks

Cosmos On Weed with Neil deGrasse Tyson

Data Preprocessing

• Data quality is defined in terms of accuracy, completeness, consistency, timeliness, believability, and interpretabilty. These qualities are assessed based on the intended use of the data.
• Data cleaning routines attempt to fill in missing values, smooth out noise while identifying outliers, and correct inconsistencies in the data. Data cleaning is usually performed as an iterative two-step process consisting of discrepancy detection and data transformation.
• Data integration combines data from multiple sources to form a coherent data store. The resolution of semantic heterogeneity, metadata, correlation analysis, tuple duplication detection, and data conflict detection contribute toward smooth data integration.
• Data reduction techniques obtain a reduced representation of the data while minimizing the loss of information content. These include methods of dimensionality reduction, numerosity reduction, and data compression. Dimensionality reduction reduces the number of random variables or attributes under consideration. Methods include wavelet transforms, principal components analysis, attribute subset selection, and attribute creation. Numerosity reduction methods use parametric or nonparatmetric models to obtain smaller representations of the original data. Parametric models store only the model parameters instead of the actual data. Examples include regression and log-linear models. Nonparamteric methods include histograms, clustering, sampling, and data cube aggregation. Data compression methods apply transformations to obtain a reduced or “compressed” representation of the original data. The data reduction is lossless if the original data can be reconstructed from the compressed data without any loss of information; otherwise, it is lossy. 
• Data transformation routines convert the data into appropriate forms for mining. For example, in normalization, attribute data are scaled so as to fall within a small range such as 0.0 to 1.0. Other examples are data discretization and concept hierarchy generation.
• Data discretization transforms numeric data by mapping values to interval or concept labels. Such methods can be used to automatically generate concept hierarchies for the data, which allows for mining at multiple levels of granularity. Discretization techniques include binning, histogram analysis, cluster analysis, decision-tree analysis, and correlation analysis. For nominal data, concept hierarchies may be generated based on schema definitions as well as the number of distinct values per attribute.
• Although numerous methods of data preprocessing have been developed, data preprocessing remains an active area of research, due to the huge amount of inconsistent or dirty data and the complexity of the problem.

Domains

Papiro Rhind

Em 1855, um advogado e antiquário escocês, A. H. Rhind (1833 - 1863), viajou, por razões de saúde, ao Egito em busca de um clima mais ameno, e lá começou a estudar objetos da Antigüidade. Em 1858, adquiriu um papiro que continha textos matemáticos.

O papiro Rhind ou Ahmes mede 5,5 m de comprimento por 0,32 m de largura, datado aproximadamente no ano 1650 a.C. onde encontramos um texto matemático na forma de manual prático que contém 85 problemas copiados em escrita hierática pelo escriba Ahmes de um trabalho mais antigo.

Uma parte do papiro Rhind. Depositado no Museu Britânico, Londres.

Bibliografia:
Boyer, Carl B., História da Matemática, Edgard Blücher, São Paulo, 1974.
Eves, Howard, Introdução à História da Matemática, Unicamp, Campinas, 1997.

Data analysis steps

Size is relative

Recommender algorithm evaluation

Method

Map & Reduce