Sunday, December 04, 2022
Sunday, October 23, 2022
The concept of probability for discrete variables can be extended to that of a probability density p(x) over a continuous variable x and is such that the probability of x lying in the interval (x, x+δx) is given by p(x)δx for δx → 0. The probability density can be expressed as the derivative of a cumulative distribution function P(x).
An illustration of a distribution over two variables, X, which takes 9 possible values, and Y , which takes two possible values. The top left figure shows a sample of 60 points drawn from a joint probability distribution over these variables. The remaining figures show histogram estimates of the marginal distributions p(X) and p(Y ), as well as the conditional distribution p(X|Y = 1) corresponding to the bottom row in the top left figure.
Thursday, October 20, 2022
Monday, September 05, 2022
Saturday, August 27, 2022
Monday, July 25, 2022
Saturday, July 16, 2022
A good scientific talk is a good scientific story
A typical plot diagram for an action movie can be similar to a the narrative structure of a scientific presentation. Just as a protagonist overcomes obstacles through a series of action scenes, a scientist pursues scientific objectives through a series of experiments to reach a goal. As a scientist comes closer and closer to reaching a goal, curiosity comes to a maximal point, the initial scientific question is answered, and the presentation reaches a sense of resolution.
Thursday, July 07, 2022
Top non-technical nuggets of wisdom of code
- Your code should work first; you can worry about optimization later.
- Your code needs to run on a machine, but it also needs to be read by a human.
- Keep away from boxes; you’re a developer.
- Write unit tests for your code!
- Knowledge is everywhere; learn how to tap into it and make the most of it.
- Side projects are fantastic and should not be a scary thing to tackle.
- Avoid companies that are like family.
- Unlimited time off can be a trap, but it can also be a wonderful perk; make sure you ask the right questions.
- Trivial perks that only adorn the job offering, such as free food and chill-out zones should be ignored.
- Flexible hours, paid parental leave, company gear, and other useful perks are what you should look for as part of a job offer.
- Plan first, code later. That way you have a blueprint to base your work on.
- Everyone in your team is as important as you, even those who don’t write code.
- Your ego can be the end of your career; you have to learn to keep it under control.
- Do not fear code reviews. They’re a perfect learning opportunity; learn to get the most out of them.
Sunday, June 26, 2022
Attention Rebellion
- Cause One: The Increase in Speed, Switching, and Filtering
- Cause Two: The Crippling of Our Flow States
- Cause Three: The Rise of Physical and Mental Exhaustion
- Cause Four: The Collapse of Sustained Reading
- Cause Five: The Disruption of Mind-Wandering
- Cause Six: The Rise of Technology That Can Track and Manipulate You
- Cause Seven: The Rise of Cruel Optimism
- Cause Eight: The Surge in Stress and How It Is Triggering Vigilance
- Causes Nine and Ten: Our Deteriorating Diets and Rising Pollution
- Cause Eleven: The Rise of ADHD and How We Are Responding to It
- Cause Twelve: The Confinement of Our Children, Both Physically and Psychologically
Friday, June 24, 2022
Complex Is Better Than Complicated
The Zen of Python—a collection of principles that summarize the core philosophy of the language—has crystal-clear points like these:
- Simple is better than complex.
- Complex is better than complicated.
- Flat is better than nested.
- Readability counts.
If the implementation is easy to explain, it may be a good idea.
DATA INTEGRITY
Data integrity refers to the physical characteristics of collected data that determine the reliability of the information. Data integrity is based on parameters such as completeness, uniqueness, timeliness, accuracy, and consistency.
Completeness
Data completeness refers to collecting all items necessary to the full description of the states of a considered object or process. A data item is considered complete if its digital description contains all attributes that are strictly required for human or machine comprehension. In other words, it may be acceptable to have missing pieces in the expected records (i.e., no contact information) as long as the remaining data is comprehensive enough for the domain.
For example, when a sensor (e.g., an IoT sensor) is involved, you might want it to sample data at a frequency of 10 minutes or even less if required or appropriate for the scenario. At the same time, you might want to be sure that the timeline is continuous with no gaps in between. If you plan to use that data to predict possible hardware failures, then you need be sure you can keep an eye close enough to the target event and not miss anything along the way.
Completeness results from having no gaps in the data from what was supposed to be collected and what is actually collected. In automatic data collection (i.e., IoT sensors), this aspect is also related to physical connectivity and data availability.
Uniqueness
When large chunks of data are collected and sampled for further use, there’s the concrete risk that some data items are duplicated. Depending on the business requirements, duplicates may or may not be an issue. Poor data uniqueness is an issue if, for example, it could lead to skewed results and inaccuracies.
Uniqueness is fairly easy to define mathematically. It is 100 percent if there are no duplicates. The definition of duplicates, however, depends on the context. For example, two records about Joseph Doe and Joe Doe are apparently unique but may refer to the same individual and then be duplicates that must be cleaned.
Timeliness
Data timeliness refers to the distribution of data records within an acceptable time frame. The definition of an acceptable time frame is also context-specific. It refers to the duration of the time frame and the appropriate timeline.
In predictive maintenance, for example, the timeline varies depending on the industry. Usually, a 10-minute timeline is more than acceptable but not for reliable fault predictions in wind turbines. In this case, a 5-minute interval is debated, and some experts suggest an even shorter rate of data collection.
Duration is the overall time interval for which data collection should occur to ensure reliable analysis of data and satisfactory results. In predictive maintenance, an acceptable duration is on the order of two years’ worth of data.
Accuracy
Data accuracy measures the degree to which the record correctly describes the observed real-world item. Accuracy is primarily about the correctness of the data acquired. The business requirements set the specifications of what would be a valid range of values for any expected data item.
When inaccuracies are detected, some policies should be applied to minimize the impact on decisions. Common practices are to replace out-of-range values with a default value or with the arithmetic mean of values detected in a realistic interval.
Consistency
Data consistency measures the difference between the values reported by data items that represent the same object. An example of inconsistency is a negative value of output when no other value reports failures of any kind. Definitions of data consistency are, however, also highly influenced by business requirements.
Sunday, June 19, 2022
Classifying Objects
The classification problem is about identifying the category an object belongs to. In this context, an object is a data item and is fully represented by an array of values (known as features). Each value refers to a measurable property that makes sense to consider in the scenario under analysis. It is key to note that classification can predict values only in a discrete, categorical set.
Variations of the Problem
The actual rules that govern the object-to-category mapping process lead to slightly different variations of the classification problem and subsequently different implementation tasks.
Binary Classification. The algorithm has to assign the processed object to one of only two possible categories. An example is deciding whether, based on a battery of tests for a particular disease, a patient should be placed in the “disease” or “no-disease” group.
Multiclass Classification. The algorithm has to assign the processed object to one of many possible categories. Each object can be assigned to one and only one category. For example, classifying the competency of a candidate, it can be any of poor/sufficient/good/great but not any two at the same time.
Multilabel Classification. The algorithm is expected to provide an array of categories (or labels) that the object belongs to. An example is how to classify a blog post. It can be about sports, technology, and perhaps politics at the same time.
Anomaly Detection. The algorithm aims to spot objects in the dataset whose property values are significantly different from the values of the majority of other objects. Those anomalies are also often referred to as outliers.
The Dev DataOps Agile cycle
For data science and development teams to work together (and along with domain experts), it is necessary that skills tend to merge: data scientists learning about programming aspects and user experience and, more importantly, developers learning about the intricacies and internal mechanics of machine learning.
Saturday, June 11, 2022
Algorithms are presented in seven groups or kingdoms distilled from the broader fields of study
- Stochastic Algorithms that focus on the introduction of randomness into heuristic methods.
- Evolutionary Algorithms inspired by evolution by means of natural selection.
- Physical Algorithms inspired by physical and social systems.
- Probabilistic Algorithms that focus on methods that build models and estimate distributions in search domains.
- Swarm Algorithms that focus on methods that exploit the properties of collective intelligence.
- Immune Algorithms inspired by the adaptive immune system of vertebrates.
- Neural Algorithms inspired by the plasticity and learning qualities of the human nervous system.
Sunday, June 05, 2022
Sunday, April 24, 2022
Sunday, April 17, 2022
A Pale Blue Dot
Sunday, April 10, 2022
* as a Service
SAAS - Software as a Service
PAAS - Plataform as a Service
IAAS - Infrastructure as a Service
CAAS - Containers as a Service
FAAS - Function as a Service
Wednesday, April 06, 2022
Tuesday, March 22, 2022
Saturday, March 19, 2022
Friday, March 18, 2022
Wednesday, March 09, 2022
Sunday, February 20, 2022
Sunday, February 13, 2022
Friday, January 07, 2022
Learning Rate
Arguably the most important hyperparameter, the learning rate, roughly speaking, controls how fast your neural net “learns”. So why don’t we just amp this up and live life on the fast lane?
Not that simple. Remember, in deep learning, our goal is to minimize a loss function. If the learning rate is too high, our loss will start jumping all over the place and never converge.
And if the learning rate is too small, the model will take way too long to converge, as illustrated above.
Monday, January 03, 2022
Saturday, January 01, 2022
Understanding the Consistency Models
Consistency defines the rules under which distributed data is available to users. What this means is that when new data is available (i.e., new, or updated data) in a distributed database, the consistency model determines when the data is available to users for reads. Besides strong and eventual, there are three additional consistency models. These are the bounded staleness, session, and consistent prefix.
Strong
Consistency Model
Guarantees that any read of an item (such as a customer record) will return the most recent version of such item.
Eventual
Consistency Model
When using the eventual consistency model, it is guaranteed that all the replicas will eventually converge to reflect the most recent write.
Bounded
Staleness Consistency Model
With bounded staleness, reads may lag writes by at most K operations or a t time interval. For an account with only one region, K must be between 10 and 1,000,000 operations, and between 100,000 and 1,000,000 operations if the account is globally distributed.
Session
Consistency Model
The session consistency model is named so because the consistency level is scoped at the client session. What this means is that any reads or writes are always current within the same session, and they are monotonic.
Consistent
Prefix Consistency Model
The last consistency model is consistent prefix. This model is like the eventual consistency model; however, it guarantees that reads never see out-of-order writes.
Consistency
for Queries
By default,
any user-defined resource would have the same consistency level for queries as
was defined for reads.
Sunday, December 26, 2021
Descriptive, Diagnostic, Predictive, Prescriptive, and Cognitive
- Descriptive: Descriptive analytics helps answer questions about what has happened, based on historical data.
- Diagnostic: Diagnostic analytics helps answer questions about why things happened.
- Predictive: Predictive analytics helps answer questions about what will happen in the future.
- Prescriptive: Prescriptive analytics helps answer questions about what actions should be taken to achieve a goal or target.
- Cognitive: Cognitive analytics attempts to draw inferences from existing data and patterns deriving conclusions.
Sunday, December 19, 2021
Thursday, December 09, 2021
Sunday, December 05, 2021
Saturday, November 20, 2021
17 Common Issues In Machine Learning: Simplified
INTRODUCTION
Machine Learning or ML is one of the most successful applications of Artificial intelligence which provides systems with automated learning without being constantly programmed. It has acquired a ton of noticeable quality lately due to its capacity to be applied across scores of ventures to tackle complex issues quickly and effectively. From Digital assistants that play your music to the products being recommended based on prior search, Machine Learning has taken over many aspects of our life. It is a skill in high demand as companies require software that can grasp data and provide accurate results. The core objective is to obtain optimal functions with less confusion.
1. WHAT IS MACHINE LEARNING?
Machine Learning is a segment that comes under Artificial Intelligence (AI) that increases the quality of applications by using previously assimilated data. It programs systems to learn and grasp data without having to feed a new code for every new similar activity. The aim is for the flow to be automated rather than continuously modified. Hence by experience and past intel, it improves the program by itself.
2. WHY MACHINE LEARNING?
The domain of Machine Learning is a continuously evolving field with high demand. Without human intervention, it delivers real-time results using the already existing and processed data. It generally helps analyze and assess large amounts of data with ease by developing data-driven models. As of today, Machine Learning has become a fast and efficient way for firms to build models and strategize plans.
3. ADVANTAGES OF MACHINE LEARNING
- Completely Automated ( Zero human intervention)
- Analyses large amounts of data
- More efficient than traditional data analytical methods
- Identifies trends and patterns with ease
- Reliable and efficient
- Less usage of workforce
- Handles a variety of data
- Accommodates for most forms of applications
4. COMMONLY USED ALGORITHMS IN MACHINE LEARNING
There are many different models in Machine Learning. Here are the most commonly used algorithms in the world today:
- Gradient Boosting algorithms dimensionality Reduction Algorithms
- Random Forest
- K-Means
- KNN
- Naive Bayes
- SVM
- Decision Tree
- Logistic Regression
- Linear Regression
Machine Learning or ML is slowly but steadily having a huge impact on data-driven business decisions across the globe. It has also helped organizations with the correct intel to make more informed, data-driven choices that are quicker than conventional methodologies. Yet, there are many issues in Machine Learning that cannot be overlooked in spite of its high productivity.
5. LIST OF COMMON PRACTICAL ISSUES IN MACHINE LEARNING
1) LACK OF QUALITY DATA
One of the main issues in Machine Learning is the absence of good data. While upgrading, algorithms tend to make developers exhaust most of their time on artificial intelligence. Data quality is fundamental for the algorithms to work as proposed. Incomplete data, unclean data, and noisy data are the quintessential foes of ideal ML. Different reasons for low data quality are-
Data can be noisy which will result in inaccurate predictions. This often leads to less accuracy in classification and low-quality results. It is noted as one of the most common errors faced in terms of data.
Incorrect or incomplete information can also lead to faulty programming through Machine Learning. Having less information will lead the program to analyze based on the minimal data present. Hence, decreasing the accuracy of the results.
For better future actions, the generalizing of input and output of past data is crucial. But a common issue that occurs is, the output data can become difficult to generalize.
2) FAULT IN CREDIT CARD FRAUD DETECTION
Although this AI-driven software helps to successfully detect credit card fraud, there are issues in Machine Learning that make the process redundant. It is tough for the system to spot anything without adequate amounts of data, hence making them blind to any illegal connections. Hence detecting fraud without possessing a significant amount of data is close to impossible.
3) GETTING BAD RECOMMENDATIONS
Proposal engines are quite regular today. While some might be dependable, others may not appear to provide the necessary results. Machine Learning algorithms tend to only impose what these proposal engines have suggested. So if there is any modification in the necessity of the result, then the recommendation will be of no use. Creating a complex algorithm, collecting large amounts of data, and implementing the algorithm, leading to nothing but incorrect results in case of changed priorities is one of the biggest issues with Machine Learning
4) TALENT DEFICIT
Albeit numerous individuals are pulled into the ML business, however, there are still not many experts who can take complete control of this innovation. It is quite rare to find a trained professional who is capable of comprehending the problems in Machine Learning and be able to reach out to a reliable software solution for the same.
5) IMPLEMENTATION
Organizations regularly have examination engines working with them when they decide to move up to ML. The usage of fresher ML strategies with existing procedures is a complicated errand. Keeping up legitimate documentation and interpretation need to go a long way to facilitating maximum usage. There are issues in Machine Learning when it comes to implementation-
Slow deployment – Although the models of Machine Learning are time efficient the creating process of the same is quite the opposite. As it is still a young innovation the implementation time is slow.
Data Security – Saving confidential data on ML servers is a risk as the model will not be able to differentiate between sensitive and insensitive data.
Lack of data is another key issue faced during the implementation of the model. Without adequate data, it is not possible to give out valuable intel.
6) MAKING THE WRONG ASSUMPTIONS
ML models can’t manage datasets containing missing data points. Thus, highlights that contain a huge part of missing data should be erased. On the other hand, if there are a couple of missing qualities in an element, rather than erasing it, we could fill those vacant cells. The most ideal approach to manage these issues in Machine Learning is to ensure that your data doesn’t accompany gaping holes and can convey a considerable measure of presumptions.
7) DEFICIENT INFRASTRUCTURE
ML requires a tremendous amount of data stirring abilities. Inheritance frameworks can’t deal with the responsibility and clasp under tension. You should check if your infrastructure can deal with issues in Machine Learning. If it can’t, you should hope to upgrade it completely with good hardware and adaptable storage.
8) HAVING ALGORITHMS BECOME OBSOLETE WHEN DATA GROWS
ML algorithms will consistently require a lot of data when being trained. Frequently, these ML algorithms will be trained over a specific data index and afterward used to foresee future data, a cycle which you can only expect with a significant amount of effort. The earlier “accurate” model over the data set may presently not be considered accurate, at a point where the arrangement of data changes.
9) ABSENCE OF SKILLED RESOURCES
The other issues in Machine Learning are that deep analytics and ML in their present structures are still new technologies. From the start code to the maintenance and monitoring of the process, Machine Learning experts are required to maintain the process. Artificial Intelligence and Machine Learning industries are still freshers to the market. Finding enough resources in the form of manpower is also difficult. Hence, there is a lack of talented representatives available to develop and manage scientific substances for ML. Data researchers regularly need a mix of space insight just as top to bottom knowledge of mathematics, technology, and science.
10) CUSTOMER SEGMENTATION
Let us consider the data of human behavior by a user during a time for testing and the relevant previous practices. All things considered, an algorithm is necessary to recognize those customers that will change over to the paid form of a product and those that won’t. A model with this choice issue would permit a program to trigger relevant recommendations for the user based on his catalog’s behavior.
The lists of supervised learning algorithms in ML are:
- Neural Networks
- Naive Bayesian Model
- Classification
- Support Vector Machines
- Regression
- Random Forest Model
11) COMPLEXITY
Although Machine Learning and Artificial Intelligence are booming, a majority of these sectors are still in their experimental phases, actively undergoing a trial and error method. From the setting up of the system to instilling complex data and even coding, the procedure is extremely complicated and quite tedious. It is a time-consuming and strenuous procedure that cannot accommodate any kinds of errors or mistakes.
12) SLOW RESULTS
Another one of the most common issues in Machine Learning is the slow-moving program. The Machine Learning Models are highly efficient bearing accurate results but the said results take time to be produced. Due to an overload of data and requirements, it takes longer than expected to provide results. This is mainly because of the complex algorithm that they work on and the time it takes to derive usable results. Another reason is that it requires constant monitoring at every step of the process.
13) MAINTENANCE
Requisite results for different actions are bound to change and hence the data needed for the same is different. This needs editing the code and more resources for monitoring the changes as well. As the outputs need to be generalized, regular monitoring and maintenance are necessary. Consistent maintenance is the key to keep the program up to date
14) CONCEPT DRIFT
This occurs when the target variable changes, resulting in the delivered results being inaccurate. This forces the decay of the models as changes cannot be easily accustomed to or upgraded. To eradicate this problem, a model that can adapt to a whole array of changes is necessary.
15) DATA BIAS
This occurs when certain aspects of a data set need more importance than others. Focusing on particular features within the database in order to generalize the outcomes is very common in Machine Learning Models. This leads to inaccurate results, low outcome levels, and other such errors.
16) HIGH CHANCES OF ERROR
Many algorithms will contain biased programming which will lead to biased datasets. It will not deliver the right output and produces irrelevant information. The usage of this can lead to bigger errors in the business models. This commonly occurs when the planning process is not done right. As you must have figured out by now, Machine Learning is all about concise algorithms. Before creating the model it is imperative that they identify the accurate problem statement and create a strategy. The major issues that arise in Machine Learning are from the errors in planning before the implementation.
17) LACK OF EXPLAINABILITY
Machine Learning is often termed a “Black box” as deciphering the outcomes from an algorithm is often complex and sometimes useless. This basically means the outputs cannot be easily comprehended as it is programmed in specific ways to deliver for certain conditions. This lack of explainability will make the reverse engineering of an algorithm nearly impossible, reducing the credibility of the algorithm.
CONCLUSION
Although there are many issues in Machine Learning, Today it is noted as one of the most evolving industries with advanced technological developments. Many of the biggest companies use Machine Learning to assist with large-grouped data analytics. From medical diagnosis and its developments to prediction and classification, ML has become an important and highly used process.
Differentiate Between Linear and Nonlinear Equations
Linear Equations | Non-Linear Equations |
A Linear equation can be defined as the equation having the maximum only one degree. | A Nonlinear equation can be defined as the equation having the maximum degree 2 or more than 2. |
A linear equation forms a straight line on the graph. | A nonlinear equation forms a curve on the graph. |
The general form of linear equation is, y = mx +c Where x and y are the variables, m is the slope of the line and c is a constant value. | The general form of nonlinear equations is, ax2 + by2 = c Where x and y are the variables and a,b and c are the constant values |
Examples:
| Examples:
|
Convolutional Neural Networks (CNNs)
Convolutional Neural Networks (CNNs) are neural networks that automatically extract useful features (without manual hand-tuning) from data-points like images to solve some given task like image classification or object detection.
Friday, November 19, 2021
Change in Entropy
Here the entropy of the system has reached its maximum value, so that there is no more change in entropy.
Monday, November 15, 2021
Binomial Distribution
The binomial distribution is a discrete probability distribution. It describes the outcome of n independent trials in an experiment. Each trial is assumed to have only two outcomes, either success or failure. If the probability of a successful trial is p, then the probability of having x successful outcomes in an experiment of n independent trials is as follows.
Problem
Suppose there are twelve multiple choice questions in an English class quiz. Each question has five possible answers, and only one of them is correct. Find the probability of having four or less correct answers if a student attempts to answer every question at random.
Solution
Since only one out of five possible answers is correct, the probability of answering a question correctly by random is 1/5=0.2. We can find the probability of having exactly 4 correct answers by random attempts as follows.
To find the probability of having four or less correct answers by random attempts, we apply the function dbinom with x = 0,…,4.
+ dbinom(1, size=12, prob=0.2) +
+ dbinom(2, size=12, prob=0.2) +
+ dbinom(3, size=12, prob=0.2) +
+ dbinom(4, size=12, prob=0.2)
[1] 0.9274
Alternatively, we can use the cumulative probability function for binomial distribution pbinom.
Answer
The probability of four or less questions answered correctly by random in a twelve question multiple choice quiz is 92.7%.























