Paradigms of recommender systems

Recommender system as a black box

A Taxonomy for Artificial and Computational Intelligence

Recommender Systems An Introduction

In this age of information overload, people use a variety of strategies to make choices about what to buy, how to spend their leisure time, and even whom to date. Recom- mender systems automate some of these strategies with the goal of providing afford- able, personal, and high-quality recommendations. This book offers an overview of approaches to developing state-of-the-art recommender systems.

Binning

Binning or discretization is the process of transforming numerical variables into categorical counterparts. An example is to bin values for Age into categories such as 20-39, 40-59, and 60-79. Numerical variables are usually discretized in the modeling methods based on frequency tables (e.g., decision trees). Moreover, binning may improve accuracy of the predictive models by reducing the noise or non-linearity. Finally, binning allows easy identification of outliers, invalid and missing values of numerical variables.

Down on the Farm

The (flawed) comparison between brain and computer at a glance

Terms to know

1. Motive: a need or desire that influences behavior
2. Speculative statement: a statement of something that could logically happen in the present circumstances.
3. Point of View: an attitude or feeling about an issue
4. Theory: an explanation that may grow out of an hypothesis; tries to explain a wide variety of circumstances
5. Bias: a slant in favor or against
6. Fact: something that can be proven
7. Precedent: examples of earlier actions or events that can be used as an example in a later action
8. Opinion: something that cannot be proven; usually involves ones feelings about a subject
9. Objection: a statement against a statement, theory, or action.
10. Belief: a thought held to be true no matter what the facts are
11. Consequences: those actions or reactions that happen as a result of something being done or said

Commons Ways to Use Propaganda

Assertion
Assertion consists of simply stating a debatable idea as a fact. It relies on the premise that people like to believe what they are told. Examples include: “Women are bad drivers,” and “Men never stop to ask for directions.” Assertions are also common in advertising claims. Sometimes facts are fabricated to support a claim.

Bandwagon
The “bandwagon effect” refers to any situation in which people attempt to be part of a successful or popular endeavor merely for the sake of its popularity. Peer pressure can come into play if “everyone else is doing it.” The problem is that you do not know how other people made their decision. It may be that everyone is simply going along with what is popular. It is a natural human desire to want to be part of a majority group or a winning team.

Card Stacking 
This is where an unfair advantage is given to one point of view, while weakening another. Information may be presented out of context and important facts can be obscured. Even if alternate views are presented, they tend to be presented last and with less skill. Censorship is an extreme form. (You see this in politics a lot. The way opposite sides present their arguments on current tax legislation is a good example.)

Glittering Generalities
These are colorful vague terms that are meant to appeal to the audience. They are often left to stand alone without context or definition. The idea is to gain unquestioning approval of whatever is said. They are common in political campaigns and advertising. Look for vague, positive words that are not explained. (Hope and change were often used in the last presidential campaign.)

False Dilemma
Also known as false choice, it involves reducing a complex argument to a small number of alternatives and concluding that only one option is appropriate. “Either you agree with us, or you are a fool.” In the real world most issues are not so simple. (The Democrats might tell you that if we don’t raise taxes on the rich, the government will go broke. The Republicans might say that if we raise taxes on the rich, there will be fewer jobs.)

The Lesser of Two Evils
This is a type of false dilemma that offers two “bad” alternatives. Here the propagandist is trying to get you to adopt a perspective you would be hesitant to accept. To do so, an even worse alternative is offered as the only option. The imperfect option is better than a horrendous one. (In an environment where all politicians have low approval rates, you might be asked to vote for someone who isn’t as bad as the alternative.)

Name Calling
Name-calling is the use of negative words to disparage an enemy or an opposing view. Insulting words take the place of logical agreements, appealing to emotions, rather than reason. This is the opposite of glittering generalities.(Politicians on both sides use the word extremist to describe someone who simply disagrees with them. Republicans use the term socialist to describe people on the left, while democrats like to accuse republicans of trying to give more money to billionaires as if being one is bad.)

Pinpointing the Enemy
Problems rarely stem from a single cause, but propagandists often oversimplify. When an enemy is blamed for something that is someone else’s fault, it is a form of this technique known as scapegoating. The Nazi’s blamed the jews for Germany’s economic problems prior to World War II. Even if the target is responsible for part of the problem, they are probably not the sole cause. (Doug: Both parties blame each other for recent economic problems.)

Plain Folk
Here the propagandist tries to paint himself as just like you. Problems are blamed on outsiders. The risk is that the person using this technique is exposed as an impostor. Politicians often paint themselves as plain folk rather than Washington insiders. (Doug: Politicians on both sides claim to be in touch with the grass roots or the blue collar crowd. They want you to think they support Main Street rather than Wall Street.)

Testimonials
Most testimonials are made by famous people who tell us how much they like a product or a politician. The propagandist hopes that your feelings about the famous person will transfer to the product or cause he or she endorses. In addition to celebrity testimonials, politicians use “plain folk” testimonials where someone like you tells you why you should vote for someone. Some testimonials are not propaganda. An expert opinion from someone who does not gain financially is an example. Keep in mind that you need faulty reasoning in order to have propaganda.

Transfer
This is a subtile technique also known as association or false connection. It is often done in a symbolic manner. If you pose with the American flag you are sending the message that you are a patriot. Such use of symbols only becomes propaganda when symbols are use to send an unspoken emotional message. (Doug: Politicians will picture an opponent with someone who they think is less popular. In more liberal districts, democrats will try to associate opponents with Rush Limbaugh. Republicans often try to connect their opponents with Nancy Pelosi from the liberal hotbed of SanFrancisco.)

Desires and Fears
Most actions are motivated by desires and fears, and a skilled propagandist can use either against you. Many fears and desires are two sides of the same coin. We desire love and fear rejection. Many ads imply romance will follow if you use their product. Others imply that the product will prevent rejection. We desire prosperity and control, and fear being powerless and lacking material wealth. We desire immortality and fear death. (Doug: People put up with invasive searches at airports due to fear of terrorists. Attractive celebrities do ads for products designed to help you live longer.)

---

Positives Uses
While the word has negative connotations, propaganda can be used for good. It can be used to end slavery, promote peace, or encourage people to improve the world or their local community. It can be used to encourage and inspire. It can also engender sympathy and generosity and promote civic minded behavior. Some people disapprove of using propaganda in this manner, while others believe that the ends justify the means.

Negative Uses
Negative propaganda often promotes fear and hostility. Propaganda that promotes fear, leaves people open to further propaganda. Such fear is often disproportionate to the actual danger. Anger is closely related and used to much the same effect. It is a common reaction to injustice, humiliation, or betrayal. It shuts down rational thought. Hatred is like fear in that it is a powerful motivator of human emotions. Negative feelings can be used to promote discrimination, violence, and property violation. They can also be used to dehumanize and violate human rights. At the opposite end, they can deify.

For more go to DrDougGreen.Com

What is propaganda?

Propaganda is always manipulative. It is intended to give someone else control over your thoughts and actions. This is why it is crucial the you be able to know it when you see it so you can make rational, conscious decisions. It comes in many forms, but you can recognize it by its persuasive function, sizable target audience, the representation of a specific group’s agenda, and the use of faulty reasoning and/or emotional appeals. It is not always harmful. In some situations it can be benign or even positive.

A3 Implementation

Polemic Subject: What’s an Employee Worth?

Os 7 desperdícios na manufatura e no escritório

The Lean Production System

Hypothetical Example of Missing Data

p Values

Data, Information, Knowledge, Wisdom, and Understanding

Partitioning Services into Layers

The Smart UI “Anti-Pattern”

Advantages

• Productivity is high and immediate for simple applications.
• Less capable developers can work this way with little training.
• Even deficiencies in requirements analysis can be overcome by releasing a prototype to users and then quickly changing the product to fit their requests.
• Applications are decoupled from each other, so that delivery schedules of small modules can be planned relatively accurately. Expanding the system with additional, simple behavior can be easy.
• Relational databases work well and provide integration at the data level.
• 4GL tools work well.
• When applications are handed off, maintenance programmers will be able to quickly redo portions they can't figure out, because the effects of the changes should be localized to each particular UI.

Disadvantages

• Integration of applications is difficult except through the database.
• There is no reuse of behavior and no abstraction of the business problem. Business rules have to be duplicated in each operation to which they apply.
• Rapid prototyping and iteration reach a natural limit because the lack of abstraction limits re- factoring options.
• Complexity buries you quickly, so the growth path is strictly toward additional simple applica- tions. There is no graceful path to richer behavior.

Layer Volatility

Layering requirements derived from requirements analysis using QFD

A process of architectural design from the layered requirements

Transatlantic - Bridge Across Forever (2001)

Escalas e Operações

The Scientific Method

Música de velho

Basic Deployment Pipeline

Difference Between "Agile" and Agile

Inovadores de inovações inovadoras

Barreiras para a implementação da cultura Lean

Os 14 princípios de Deming

1. Constância de propósitos;
2. Acredite nesta nova forma de trabalhar;
3. Não dependa de inspeção para atingir a qualidade;
4. Não aprove custos ou orçamentos com base no preço. Busque minimizar o custo total (Keiretsu: um fornecedor para cada item, relacionamento de longo prazo);
5. Melhore constantemente seus processos;
6. Treinamento no local de trabalho;
7. Institua liderança, com objetivo de ajudar as pessoas e os processos a executarem um trabalho melhor, e também a incentivar o trabalho coletivo;
8. Elimine o medo: todos devem procurar ser o melhor possível;
9. Elimine as barreiras entre os departamentos, todos são partes de um processo;
10. Elimine metas de qualidade e produtividade para a mão-de-obra: quem deve melhorar é o processo;
11. Elimine objetivos locais e substitua pelos globais. Incentive a administração por processos através do exemplo de líderes;
12. Dê direito do operário de orgulhar-se de seu desempenho, valorizando sempre o coletivo;
13. Implante um forte programa de educação e auto-aprimoramento;
14. Todos da empresa devem se comprometer no processo de realizar a transformação.

Objetivo do Lean: Eliminar Desperdício!

Lean Office

How to Manage the "7 Wastes" of Agile Software Development

Origins of the term waste

The concept of "waste" in business was coined in the manufacturing industry during late 1940s. It was used by Toyota. In those days, automobiles involved a huge amount of manufacturing, and companies had to charge their customers a high price. The only option for Toyota to reduce car prices was to find ways to reduce the manufacturing costs. As part of this exercise, they started identifying "waste," which meant the feature (or process step) that did not add value for the customer. Once they identified the waste, they created ways to eliminate that waste from the system.

Over time, "eliminate waste" became the fundamental concept of Lean, which was used in the manufacturing industry.

What is waste?
Waste is, in fact, the opposite of value (a capability delivered to the customer through which the customer attains a tangible or intangible benefit). So whatever feature or functionality or process step that neither adds value nor is used will be considered waste and should be eliminated from the system/product/process.

The seven wastes listed below were identified in manufacturing in the Toyota Production System (TPS) by Shigeo Shingo:

1. Inventory: Unfinished goods (also called as "work in progress," or WIP)
2. Overproduction: Producing more than the demand requires
3. Extra processing: Additional steps in the process that aren't really needed
4. Transportation: Shipping the goods from one place to the other
5. Waiting: Lag between process steps
6. Motion: Moving around within the process
7. Defects: Flaws in the deliverables that impact their features/functionality

Seven wastes of software development

Based on the seven wastes of manufacturing industry, Mary and Tom Poppendieck have defined seven wastes that are appropriate for software development. What follows provides information about the meaning of each of the seven wastes of software development, which can help us eliminate them (or at least reduce their impact) with respect to Agile software development. Generally, you will find these wastes during sprint planning or during sprint execution.

Waste #1: Partially done work

Meaning: It generally means that the work (story) that is not completely done as per your Definition of Done, and hence you cannot demo it or you cannot release it. This can include code that is not refactored, code that is not unit tested/integration tested, code that is not properly documented, code that is not deployable, etc.

Possible reasons:

1. Prioritizing a story without having complete information about that story from the product owner
2. Technical complexity that was not analyzed properly during the sprint planning
3. Wait time between the tasks that are identified to complete the story
4. Improper dependencies for this story based on other stories
5. Removing an existing story that is halfway done and getting a new story into the sprint
6. Incomplete/inadequate tasks identification

How can you eliminate it?

1. Try to have a detailed discussion with product owner to understand the functionality of the story, and also to understand what value it adds to the product. Then prioritize it for a sprint. This calls for great coordination and collaboration between the team and the product owner.
2. Try to evaluate the technical complexity of the story based on its functionality. If required, go for a technical spike before you arrive at the estimate. This will, at the least, help you assess how complex it is so that you can either pick it up directly or ask the product owner to split it -- or go with some other technical implementation approach.
3. Try to do tasks in parallel as much as possible. This should be the true Agile way and not a mini-Waterfall way. When you are stuck somewhere, look for help. Develop he cross-functional teams that can address this point effectively, because anyone can pick up any task at least to some extent.
4. Manage your product backlog in such a way that the dependencies among various stories are clearly identified before the stories are picked up for a sprint. Again, this needs lot of coordination and collaboration between the team and the product owner. Sometimes the product owner may not have understand the technical dependencies, so in such scenarios the team should help the product owner. Example: Log-in cannot be prioritized without registration.
5. When you have stories lined up (committed) for a sprint, do not try to manipulate them during the sprint. It will add unnecessary rework and will lead to a loss in velocity.
6. When you decompose the story into tasks in your sprint planning meeting, try to spend a good amount of time and involve all the team members. This will help you to come up with all the required tasks to complete the story. If you do not do this, you will have an insufficient list of tasks but, in reality, you will have to perform every task to complete a story. So the tasks that are not identified in your sprint planning may lead to additional effort that is not planned, so it may ultimately push your story to an "incomplete" state.

Waste #2: Extra features

Meaning: Providing more than what is being asked for. Remember the Pareto principle (80 percent of users use only 20 percent of the product features).

Possible reasons:

1. Lack of understanding of the product vision
2. Who is in your product's "target audience"?
3. Gold-plating by the development team
4. Wrong prioritization of product features

How can you eliminate it?

1. The product backlog is the baby of the product owner, who should map it to the product vision. He or she is also responsible for the product ROI. So the product owner is responsible for managing the product backlog according to the product vision. He should prioritize the stories as per the value.
2. Every product will have its own target audience, the primary users of the product. The product owner has to make sure to identify the target audience, and only then start creating the product backlog. If required, use personas to identify the features based on the target audience.
3. Gold-plating generally comes from the development team, and sometimes it can be unintentional. While prioritizing the stories for a sprint, the product owner and the development team have to come to a common agreement on the stories for that sprint, and they must stick to that list.
4. Right from the release planning, the product features have to be carefully prioritized. The keys to prioritizing these stories are value, cost, and risk.

Waste #3: Relearning

Meaning: Not using the knowledge that is available within the team members, trying to reinvent the wheel.

Possible reasons:

1. Lack of a proper knowledge-sharing process within the team
2. Lack of required documentation
3. Missing information radiation and osmatic communication
4. Distributed teams

How can you eliminate it?

1. The team should share knowledge continuously throughout the sprint. The best places for knowledge sharing are the daily Scrum meeting and the task board. All the team members should participate in all team meetings. This will enhance knowledge sharing.
2. The general myth about Agile is: "Agile is anti-documentation." This is not true. The team should prepare all the required documentation, but it should always be "just in time."
3. Try to keep all important information displayed in a team space so that everyone has equal access. Osmatic communication plays a vital role in information sharing and knowledge distribution. Make sure to document the "tacit knowledge" that will be repeatedly required. You can use tools such as SharePoint, Wiki, etc.
4. Co-location is the best approach for executing Agile projects. If this isn't possible, due to any practical reasons, use tools like videoconferencing so that knowledge sharing is easier.

Waste #4: Hand-offs

Meaning: Passing the work from person to another, after completing the first person's work.

Possible reasons:

1. The nature of the tasks required for a story
2. Teams working from different locations
3. Lack of visibility of the information

How can you eliminate it?

1. Sometimes it is unavoidable, and different tasks have to be done by different team members (e.g., analysis, UI, client-side coding, server-side coding, etc). It's good to have cross-functional teams so that, over a period of time, the handoffs can be reduced.
2. When teams are working from different locations (time zones), if the information is not properly handed over, then this will lead to waiting. So make sure there is a proper hand-off process, and obtain confirmation from the other side of the team. Also make sure to execute the tasks at one location as much as possible, as this will reduce hand-offs and waiting time.
3. Try to keep important flowcharts and wireframes visible and clear. This can also be helpful to reduce hand-off time. Your Kanban task board, with regular updates, will really help here, along with your Definition of Done.

Waste #5: Delays

Meaning: Anything that causes more time to deliver a value-added activity, or delays the beginning of the value-added activity.

Possible reasons:

1. Lack of required team members
2. Unwanted processes
3. Too many things in progress
4. External dependencies
5. Lack of "value" understanding
6. Assumptions/clarifications and impediments

How can you eliminate it?

1. Make sure you have all the required skill sets assigned to your project. If you start a sprint without having the required team members with the proper skill sets, this will lead to delays.
2. Identify only mandatory processes at the beginning of the sprint. For example, if you do not need to get the code review for all of the code, then categorize the code into low complex, medium complex, and high complex, and send only medium and high complex code for review. This will reduce the cycle time and increase efficiency.
3. Keep only what you can handle on your plate and leave the rest open. This will help other team members pick up the work that's based on their skill sets and bandwidth.
4. Make sure to have all required external assistance available in time for your work. For example, if you need an external architect review, plan for it up front. Similarly, if you're working with multiple teams from different locations, then encourage stub-based development. I have personally observed that when we have videoconferencing calls with other teams working from different locations, usually the videos won't work, you have to waste time to fix the problem, if it isn't fixed in a certain amount of time you need to make alternate arrangements, etc. (One way to tackle the delays in such calls is for the ScrumMasters from both teams go on the call two minutes beforehand and ensure that everything is working properly. Also, always keep a conference bridge number as backup, and after a fixed amount of time -- say, five minutes -- spent trying to fix any problems, move to the conference bridge.)
5. Always try to do value stream mapping and see what is value-added time and what is not. This will give you an idea of what your current process efficiency is. Based on the current state of the value stream map, and by applying Lean practices, you can improve the cycle time and thereby reduce delays. Also, automate the test cases wherever possible so that you can reduce a considerable amount of time when you have to run them recursively.
6. Make sure you get clarity on your assumptions and inputs for your clarifications at the right time. Raise the red flag before you get into the danger zone, and get proper attention from respective stakeholders. Track your impediments effectively; the ScrumMaster is responsible for resolving the team's impediments.

Waste #6: Task Switching

Meaning: Team members moving from one task to another without completing the first task properly.

Possible reasons:

1. Interruptions in the ongoing tasks
2. Lack of ground-level analysis of the tasks required for the stories
3. A shared team working on more than one project at a time
4. Lack of proper coordination between the product owner and development team

How can you eliminate it?

1. Interruptions can be due to many reasons, including lack of complete information about the task, hardware-related dependencies, etc. Make sure the tasks are detailed enough in the sprint planning meeting and identify any external dependencies up front. Raise them as impediments and let the ScrumMaster work on them.
2. Make sure to spend enough time when you're splitting the stories into tasks during your sprint planning meeting. You need to identify all the possible tasks, and also identify the order of those tasks, so that you can work on one story until it's completely done without any task switching.
3. Ideally, Scrum teams should be dedicated teams. If you have shared teams, then naturally that leads to task switching.
4. If the product owner does not provide the priority of the stories during the sprint planning, this may lead to juggling between stories during the sprint. This, in turn, leads to task switching. So try to have the stories prioritized by the product owner. For this the value, cost, and risk are the key factors, as mentioned above.

Waste #7: Defects

Meaning: Erroneous functionality that produces wrong output.

Possible reasons:

1. Lack of understanding on the story
2. The story does not satisfy the INVEST principle
3. Lack of engineering practices such as TDD, refactoring
4. Missing acceptance criteria
5. Lack of technical skill sets for team members
6. Late involvement of testers
7. Inattention to the automation testing

How can you eliminate it?

1. Don't jump in to story development until you have a complete understanding of the story. Refer to the tenth Agile principle -- simplicity, or maximizing the amount of work undone. Wherever more complexity is involved, then go with the "specification by example" approach.
2. The product owner has to ensure that the story follows the INVEST principle. If not, then try to split it into more granular stories.
3. Be strict about making test-driven development, pair programming, and refactoring part of your development plan. These are powerful engineering practices.
4. The product owner must attach the acceptance test criteria to each story that is prioritized for a sprint. This will help the developer understand the story in more detail from the implementation point of view.
5. Make sure you have right skills on the team and also have suitable coding standards and guidelines in place up front.
6. It is always good to have the testers involved right from the sprint planning stage. It will help them engage in the largest possible test coverage, which reduces defects.
7. Make sure to have an automated test suite created from the very first sprint. For any project, a certain number of test cases (sanity tests) will have to run more than once, so automating them will save time and also reduce defects.

Lean on Software Development

Waste #1 - Partially Done Work
Waste #2 - Extra Features
Waste #3 - Relearning
Waste #4 - Handoffs
Waste #5 - Delays
Waste #6 - Task Switching
Waste #7 - Defects

Command Query Responsibility Segregation (CQRS)

CRUD mode

CQRS mode

Microservices Characteristics

• Componentization via services
• Organized around business capabilities
• Products not projects
• Smart endpoints and dumb pipes
• Decentralized governance
• Decentralized data management
• Infrastructure automation
• Design for failure
• Evolutionary design

Big O notation

Recomendações aos cientistas, Karl Popper

"A tarefa mais importante de um cientista é certamente contribuir para o avanço de sua área de conhecimento. A segunda tarefa mais importante é escapar da visão estreita de uma especialização excessiva, interessando-se ativamente por outros campos em busca do aperfeiçoamento pelo saber que é a missão cultural da ciência. A terceira tarefa é estender aos demais a compreensão de seus conhecimentos, reduzindo ao mínimo o jargão científico, do qual muitos de nós temos orgulho. Um orgulho desse tipo é compreensível. Mas ele é um erro. Deveria ser nosso orgulho ensinar a nós mesmos, da melhor forma possível, a sempre falar tão simplesmente, claramente e despretensiosamente quanto possível, evitando como uma praga a sugestão de que estamos de posse de um conhecimento que é muito profundo para ser expresso de maneira clara e simples.

Esta, é, eu acredito, uma das maiores e mais urgentes responsabilidades sociais dos cientistas. Talvez a maior. Porque esta tarefa está intimamente ligada à sobrevivência da sociedade aberta e da democracia.

Uma sociedade aberta (isto é, uma sociedade baseada na idéia de não apenas tolerar opiniões dissidentes mas de respeitá-las) e uma democracia (isto é, uma forma de governo devotado à proteção de uma sociedade aberta) não podem florescer se a ciência torna-se a propriedade exclusiva de um conjunto fechado de cientistas.

Eu acredito que o hábito de sempre declarar tão claramente quanto possível nosso problema, assim como o estado atual de discussão desse problema, faria muito em favor da tarefa importante de fazer a ciência -- isto é, as idéias científicas -- ser melhor e mais amplamente compreendida."

Karl R. Popper: The Myth of the Framework (in defence of science and rationality). Edited by M. A. Notturno. (London: Routledge, 1994), p. 109.

Enterprise Architecture Context

Enterprise Architecture

Enterprise Integration Patterns

ARCHITECTURE DEFINITION, REQUIREMENTS ANALYSIS, AND SOFTWARE DESIGN

TYPICAL VIEW AND PERSPECTIVE APPLICABILITY

The Database Bottleneck

Techniques for Change

Cyclomatic Complexity

Basic Deployment Pipeline

The "Agile" Team

the relative ease of making changes to speculative development versus YAGNI code bases

What are you doing?

Software Engineer Mindset

Fallacies of Distributed Computing

Essentially everyone, when they first build a distributed application, makes the following eight assumptions. All prove to be false in the long run and all cause big trouble and painful learning experiences.

• The network is reliable
• Latency is zero
• Band width is infinite
• The network is secure
• Topology doesn‘t change
• There is one administrator
• Transport cost is zero
• The network is homogeneous

Context, not controle

ACO

An ant arriving in city i chooses the next city to move to as a function of the pheromone values tij and of the heuristic values hij on the arcs connecting city i to the cities j the ant has not visited yet. 

Basic structure of a neural expert system

Comparison of expert systems (ES), fuzzy systems (FS), neural networks (NN) and genetic algorithms (GA)

The main players of the expert system development team

Difference Between Abstract and Summary

Structure of an extended multipopulation evolutionary algorithm

Problem solution using evolutionary algorithms

An intelligent agent as a black box