In most of our decisions, we are not betting against another person. Rather, we are betting against all the future versions of ourselves that we are not choosing.

- Annie Duke

Introduction

In an increasingly complex and uncertain world, making well-informed decisions often requires navigating through incomplete information and noise from information overflow. Assumptions play a pivotal role in navigating complexity in decision making. Properly using assumptions helps simplify problems by filling in the missing information, thus enabling us to tackle complex issues more easily. They also encourage us to consider alternative perspectives, leading to more objective decisions.

Assumptions are statements accepted as true without definitive proof, serving as starting points for reasoning or action. They are essential tools that help us formulate strategies, plan for the future, and adapt to changing circumstances. However, unchecked or unexamined assumptions can lead to misunderstandings, flawed strategies, and unintended consequences. Therefore, it's important that when using assumptions that you approach them with a balance of confidence and skepticism.

This article delves into the role of assumptions in decision-making—their benefits, pitfalls, and how to effectively incorporate them into your decision-making process. We'll explore techniques for formulating clear assumptions, assigning confidence levels to them, validating them through evidence and show how to incorporate principles like Bayesian probability and counterfactuals to superpower our assumptions.

Assumptions: The Good and the Bad

Advantages of Using Assumptions in Your Decision-Making

1. Simplification of Complex Problems: Assumptions reduce complexity by filling in the gaps of information, they help make difficult and problems that contain high levels of uncertainty more manageable.

2. Efficiency in Decision-Making: They allow for quicker decisions by enabling progress without waiting for all data.
   

3. Encouragement of Creative Thinking: Assumptions open up possibilities beyond the known facts, fostering innovation and alternative solutions.
   

4. Preparation for Uncertainty: By considering different assumptions, you can plan for various scenarios, enhancing adaptability.
   

Disadvantages of Using Assumptions in Your Decision-Making

1. Risk of Inaccuracy: Incorrect assumptions can lead to flawed strategies and poor outcomes.
   

2. Overlooking Important Data: Relying too heavily on assumptions may cause neglect of crucial information.
   

3. Reinforcement of Biases: Personal biases can influence assumptions, leading to subjective decisions. Make sure that you consider biases that might be creeping in to your assumptions, and ultimately into your decision-making process.
   

4. Resistance to New Information: Fixed assumptions may hinder the acceptance of new evidence that contradicts them.
   

5. False Sense of Security: Unquestioned assumptions might create unwarranted confidence in uncertain situations.

It’s essential to hold your assumptions loosely, viewing them as temporary placeholders that might change with the acquisition of new information (See Bayesian probability).

Using Assumptions in Your Decision-Making Process

Formulate Assumptions

• Identify Gaps: Recognize where information is missing and consciously decide on assumptions to fill those gaps.

• Be Explicit: Clearly state your assumptions to yourself and to your team to ensure everyone is on the same page.

• Rate Confidence Levels: Assess how confident you are in each assumption. Are they based on solid data, past experience ( see definition: base rates), or are they speculative? Assigning confidence levels helps gauge the reliability of your assumptions and helps keep you honest.

• Incorporate Counterfactual Thinking: For each assumption, consider counterfactuals by asking "what if" questions about alternative scenarios. This helps identify potential risks and opportunities.

Example:

• Assumption: Our new product will meet high market demand.

• Counterfactual: What if the market demand is lower than expected due to emerging competitors?

Validate Assumptions

• Question Regularly: Continually challenge your assumptions by asking critical questions. What evidence supports this assumption? What if it isn’t true?

• Seek Evidence: Look for data, feedback, or expert opinions that can confirm or refute your assumptions.

• Consider Alternatives: Explore different scenarios where your assumptions might not hold. This prepares you for unexpected outcomes.

Example:

• Assumption Validation: Conduct market research to test the demand for your product.

• Counterfactual Exploration: Analyze the impact on your business if a key competitor releases a similar product.

Iterate, Iterate, Iterate

• Feedback Loop: Establish mechanisms to collect feedback on the outcomes of decisions based on your assumptions.

• Measure Against Expectations: Compare actual outcomes with what you expected if your assumptions were correct.

• Adjust Accordingly: Use new information and results to refine your assumptions. Be willing to revise or abandon assumptions that no longer serve you.

• Explore Counterfactual Scenarios: Analyze how different outcomes could have occurred if alternative assumptions were made. Ask yourself what factors led to the actual result versus the expected one.

• Learn and Adapt: Use discrepancies between expected and actual outcomes to refine your assumptions and decision-making processes.
  
  Example:

• Assumption: A marketing campaign will increase sales by 20%.

• Outcome: Sales increased by only 10%.

• Counterfactual Analysis: What if the campaign had targeted a different demographic? What if a competitor hadn't launched a promotion simultaneously?

• Action: Adjust future campaigns based on these insights.

Apply Bayesian Probability

Regularly Update Your Beliefs and Assumptions Based on New Information

Incorporate principles from Bayesian Probability, which provides a mathematical framework for updating beliefs in light of new evidence. Bayesian thinking emphasizes that our initial beliefs (prior probabilities) should be adjusted (posterior probabilities) as we receive new data. The main takeaway is not the mathematical calculations, but the concept of rethinking your assumptions whenever there is new information or evidence. This approach leads to more accurate and reliable decisions over time, as it systematically incorporates new information. A perfect example of using assumptions as temporary guideposts.

Conclusion

Instead of trying to predict a single future outcome, prepare for a range of possible futures based on different assumptions and their counterfactuals. This approach acknowledges uncertainty and equips you to adapt as situations evolve. Embracing flexibility, continuously updating your beliefs, and being willing to adjust your assumptions will lead to more resilient and effective decision-making.

Definitions

Assumptions: Ideas or statements accepted as true without definitive proof, used as a starting point for further reasoning or action. In decision-making, assumptions fill informational gaps and provide a foundation for developing strategies.

Base Rate: The general probability of an event occurring within a given context, based on historical data or general knowledge.

Bayesian Probability is a framework for interpreting probability as a measure of belief or confidence, rather than as a long-term frequency of events. It allows for the updating of probabilities as new information becomes available, using Bayes' Theorem.

Beliefs: Convictions or acceptances that certain things are true or real, often held without complete evidence. Beliefs are shaped by personal experiences, cultural contexts, and values, influencing how we interpret information and make decisions.

Bets: Decisions made with incomplete information, based on the best available evidence and assumptions. Betting in decision-making involves weighing the probabilities and potential outcomes to make informed choices under uncertainty.

Counterfactuals: Hypothetical scenarios that consider what might have happened if past events had occurred differently. They involve thinking about alternative realities and are useful for understanding causality, exploring different outcomes, and improving future decision-making by learning from past possibilities.

We are going to go walk through a process of creating a document corpus to use with LLMs via Retrieval-Augmented Generation (RAG) in the context of a City Building Department.

Why Create a Document Corpus?

A well-organized document corpus serves as the backbone of any AI system designed to provide accurate, context-aware responses to queries. For municipal governments, where the complexity of regulations, policies, and procedures can be daunting, having an accessible and well-structured corpus can vastly improve efficiency and public service delivery.

Imagine a city’s building department, where staff members, contractors, and even citizens frequently need to access detailed information on building codes, permits, inspections, and zoning regulations. A document corpus tailored for AI can help the department quickly retrieve relevant information, answer queries, and make informed decisions.

Step 1: Define Objectives and Scope

Start by clarifying the goals of your document corpus. For the building department, the primary objective might be to improve the speed and accuracy of responses to queries about building codes and permit processes. The target audience would include department staff, contractors, and the public.

The scope should include all relevant documents, such as building codes, inspection reports, permit applications, zoning regulations, and internal memos.

Step 2: Document Collection

Identify and gather all relevant documents within the building department. These might include:

· Building Codes: The full text of local building codes, updated regularly.

· Permit Applications: Forms and guidelines for various types of permits.

· Inspection Reports: Historical and recent reports on building inspections.

· Zoning Regulations: Documents outlining zoning laws and restrictions.

· Internal Memos: Communications regarding policy changes or new procedures.

Collect these documents in their existing formats, whether they are PDFs, Word documents, or spreadsheets. For now, focus on gathering the content rather than worrying about format consistency. The use of web scrapers can be used to speed up the collection of digital documents.

Step 3: Data Cleaning and Preprocessing

Next, clean and preprocess the documents to ensure they are ready for AI integration. This step involves:

· Deduplication: Remove duplicate documents to ensure the corpus is concise.

· Text Cleaning: Standardize text formatting and remove irrelevant content like disclaimers or redundant headers.

· Document Structuring: Break down large documents into smaller, logically coherent sections or chunks (e.g., chapters, headings). For example, a lengthy building code document might be divided by sections or chapters or subheadings.

Step 4: Content Chunking

Chunking involves dividing documents into manageable pieces that can be effectively processed by the AI. In our building department example, you might chunk the building codes by individual code sections, ensuring each chunk covers a specific regulation or guideline.

Keep chunks between 200-500 words, ensuring that each chunk is semantically coherent. This makes it easier for the AI to retrieve relevant information based on user queries.

Segment the document by using tools like nltk or spaCy (Python libraries) to split text by sentences or paragraphs. Create a script that identifies logical breakpoints in the document (e.g., headings) and splits the content accordingly.

Step 5: Document Categorization and Tagging

Organize the chunks into categories such as “Building Codes,” “Permit Applications,” and “Inspection Reports.” Apply relevant tags to each chunk, like “residential,” “commercial,” “zoning,” or “electrical,” to facilitate search and retrieval.

For example, a section of the building code dealing with residential fire safety could be tagged with “residential,” “fire safety,” and “building codes.” Assign a unique ID to each chunk and store it in a database or spreadsheet along with metadata (e.g., document ID, section title, chunk ID).

Step 6: Embedding and Indexing

Convert the text chunks into vector representations using a pre-trained embedding model. Store these embeddings in a vector database to enable efficient retrieval.

For instance, you might use an embedding model like BERT to convert the text into vectors and store them in a database like Pinecone. This setup allows the AI to quickly retrieve and reference the most relevant chunks when answering queries.

Step 7: Quality Assurance

Perform a manual review of a sample of the processed documents to ensure they meet quality standards. For example, check that a search for “residential zoning regulations” correctly pulls up the relevant sections of the zoning code.

Test the RAG system by posing sample queries and evaluating the accuracy and relevance of the responses. Make iterative adjustments as needed.

Step 8: Deployment and Integration

Integrate the corpus with your AI system, configuring it to use RAG for enhanced retrieval. Train users on how to interact with the system, emphasizing best practices for querying.

For example, a staff member might use the system to quickly find the exact code requirements for installing a commercial elevator, ensuring they provide accurate information to a contractor.

Step 9: Documentation

Document the entire process, including the tools used and any challenges encountered. Create user guides that are easy to follow, helping staff and other stakeholders effectively use the AI system.

Example: Implementing the Process in a City’s Building Department

Let’s bring this to life with a specific example. Suppose the City of Miami’s Building Department wants to create a document corpus to help streamline their response to building code inquiries. Here’s how they could do it:

1. Objective: The goal is to reduce the time it takes for staff to respond to queries about building codes and permit processes.

2. Document Collection: Gather all relevant building codes, permit forms, zoning regulations, and historical inspection reports.

3. Data Cleaning: Remove outdated or duplicate documents, standardize text formatting, and break large documents into logical chunks.

4. Content Chunking: Divide the building code into chunks by section, each covering a specific regulation.

5. Categorization and Tagging: Organize the chunks into categories like “Building Codes” and “Zoning Regulations,” applying tags such as “residential” and “commercial.”

6. Embedding and Indexing: Convert text chunks into vectors using a model like BERT and store them in a vector database.

7. Quality Assurance: Test the system with common queries to ensure accurate and relevant responses.

8. Deployment: Integrate the corpus with the department’s AI system, train staff on its use, and continuously monitor performance.

By following this streamlined process, the City’s Building Department can significantly improve its efficiency, reduce response times, and provide more accurate information to both staff and the public.

Conclusion

Creating a document corpus for AI doesn’t have to be a daunting task. By following a structured process, municipal departments like the City of Miami’s Building Department can leverage AI to enhance their operations, improve public service, and make more informed decisions. As cities continue to embrace digital transformation, building a solid foundation with a well-organized document corpus will be key to unlocking the full potential of AI.

Steps to Develop Your Own Mental Model

1. In which domains are you a subject matter expert?

Identify 2-3 domains where you feel most confident in your expertise. These will form the foundation of your personal mental model.

Consider the areas where you have deep knowledge and experience. These could be:

• Your professional field (e.g., software engineering, finance, marketing)

• A hobby or personal interest you've extensively pursued (e.g., chess, photography, history)

• Life experiences that have given you unique insights (e.g., living in multiple cultures, overcoming specific challenges)

2. Which mental models can complement your domain knowledge?

While your expertise forms the core, incorporating established mental models can enhance your framework. Here are some fundamental models that can benefit anyone's toolkit:

• First Principles Thinking: Breaking down complex problems into their most basic, foundational elements and then reasoning up from there. First principles thinking helps in developing novel solutions and avoiding assumptions.

• Second-Order Thinking: Considering the long-term consequences of decisions, not just immediate outcomes. This model helps in anticipating unintended consequences and making more robust decisions.

• Inversion: Approaching problems backward by focusing on what you want to avoid rather than what you want to achieve.

• Probabilistic Thinking: Understanding that most outcomes are uncertain and thinking in terms of probabilities rather than absolutes. Crucial for decision-making under uncertainty.

• Systems Thinking: Viewing problems as parts of an overall system rather than in isolation. This approach helps in understanding complex interactions and finding holistic solutions.

3. Create your own mental model framework

To combine your domain expertise with these general mental models:

1. Apply first principles thinking to break down the problem into its core components.

2. Use your domain knowledge to understand these components deeply.

3. Apply second-order thinking to consider long-term and indirect effects.

4. Use inversion to identify potential pitfalls or undesired outcomes.

5. Apply probabilistic thinking to assess the likelihood of different outcomes.

6. Use systems thinking to understand how this problem or decision fits into the larger context.

7. Make your decision

This process allows you to leverage both your specialized knowledge and the best mental models to make the best decisions possible.

4. Test and Refine

Track your decisions in a decision journal. Are you making better decisions by applying your mental model framework? How can you improve your decision making? Iterate based on results and feedback.

Conclusion

Creating your own mental models is an ongoing process of observation, analysis, and refinement. It's a powerful way to develop your critical thinking skills and gain unique insights into the world around you. By combining your specialized knowledge with established cognitive tools, you create a decision-making framework that is both powerful and uniquely suited to your strengths.

While there are hundreds of mental models you can apply, you need to start small. Ideally, with domains of knowledge that you are familiar with, and then adding a handful of the best mental models. From this starting point you then add complementary mental models to your framework. The goal is to make them your own, so that you can apply them to make the best decisions possible.

This personalized approach to mental models can lead to breakthrough insights in your field, help you communicate complex ideas more effectively, and give you a competitive edge in problem-solving. Remember, the goal isn't to create a perfect, unchanging model, but rather to develop a flexible, evolving toolkit that grows with your experience and knowledge.

As you continue to refine your mental model framework, you'll likely find that it not only improves your decision-making but also enhances your ability to learn and adapt in a rapidly changing world. Embrace the process, stay curious, and don't be afraid to challenge and revise your models as you encounter new information and situations.

By leveraging large language models (LLMs) you can have a tireless tutor, that can adapt to your learning style, assess your knowledge gaps, and patiently guide you through complex topics.

In this article, we will explore a learning workflow that uses AI to customize and superpower your learning experience. From assessing your current knowledge to reinforcing your learning through adaptive quizzes, we will see how AI is going to revolutionize education.

The AI Learning Workflow

Creating a personalized learning experience with AI involves several stages, from assessing your current knowledge to curriculum creation and continuous assessment. Here’s a detailed process flow:

1. Assess your current knowledge on a subject

Identify your starting point on subject, this is the first step to help the AI tailor your curriculum and content by knowing your initial understanding of a subject.

Prompt
Generate an self-assessment quiz for [subject]. Include multiple-choice questions, short answer questions, and complex problems to evaluate the learner's current understanding comprehensively.

2. Create a personalized curriculum with resources

Have the AI generate a structured curriculum tailored to individual learning goals, covering a that covers your current knowledge level. Have the AI suggest textbooks, online courses, videos, articles, and practical projects. You can dynamically update the curriculum every few months based on your current level.

Prompt:

Create a {timeframe} curriculum on {subject} for a {knowledge level} student. Include textbooks, online courses, videos, articles and practical projects.

3. Generate and curate content specific to your learning needs

While the curriculum provides you with the scaffolding of what you need to learn, this is where you want to build on the content that you are going to learn. You can use the sections in the curriculum to have the LLM generate content specific to your learning needs.

Based on the initial assessment, LLMs can generate content that targets specific areas a learner needs to focus on. This includes creating detailed explanations, examples, and case studies tailored to the learner's interests and needs.

Prompt:

As {well-known mentor} create content for {week 1 of curriculum}. Include detailed explanations, summaries, and examples for a {knowledge-level}

3b. Leverage enhanced coding environments (optional)

If you are learning a programming language or math you can have the LLM generate content using Jupyter Notebooks to allow for interactive and hands-on learning experiences.

Prompt:

Integrate learning materials for [topic] with Python and Jupyter notebooks. Create interactive and hands-on learning experiences where learners can run code, visualize data, and experiment with concepts in real-time.

4. Have it act as a mentor

Have the LLM act as a mentor so that it may guide you through complex subjects, provide explanations, and answer questions real-time. This can help learners overcome obstacles and understand difficult concepts more efficiently.

Prompt:

Explain {knowledge gap} to a 12th grader, provide detailed explanations and examples.

5. Generate Exercises, examples, and practice problems

Develop practical exercises and examples that reinforce the concepts you’re learning. This helps in applying theoretical knowledge to practical scenarios. You can have it adapt the difficulty of the questions it asks based on your previous response.

Prompt:

Generate {exercises / practical examples} for {topic}. Include step-by-step solutions and explanations for each exercise.

6. Interactive quizzes and feedback loop

Implement quizzes to test your understanding and get immediate feedback. This continuous assessment helps in tracking progress and identifying areas that need further review.

Prompt:

Create a quiz to gauge my understanding on {topic} and provide feedback and suggestions on areas needing improvement.

7. Track progress and adjust learning paths

Have the AI keep you on track and modify the learning approach as needed. Based on the feedback, you can always go back and adjust the curriculum as needed. You can even have the AI review the curriculum and have it adjust it based on your quiz results.

Prompt:

Help me track my progress as I learn [specific topic]. Create a system for logging completed tasks, quizzes, and exercises. Based on my performance and feedback, suggest adjustments to my learning path to ensure continuous improvement and mastery of the topic

Conclusion

As we have seen throughout this article, the integration of LLMs into the learning process offers several key advantages:

1. Personalization delivery of educational content: LLMs can tailor content and pacing to the students needs, making education more accessible and effective for different types of learners.

2. Immediate feedback and adaptive learning: The ability to provide instant, contextual feedback allows for rapid iteration and improvement in the learning process.

4. Interactive and engaging learning experiences: Through generated exercises, real-world examples, and conversational interactions, AI makes learning more dynamic and engaging.

5. Flexibility and accessibility: With 24/7 availability of a high quality teaching resource, LLMs democratize access to high quality education.

AI used alongside human teachers will help fill in the education gaps we are currently experiencing. This is the missing tool that enhances our learning experience and help us achieve new levels of self-improvement.

If you are like me, even thought the world is filled with problems, you are blind to these opportunities and their potential startup implications. Some people can identify these problems, but most of us need a framework to aid them in identifying and selecting these potential startup opportunities.

The issue is not just identifying a problem, but identifying one that you will be passionate about and willing to spend months, years or a lifetime working on. The goal is to make sure that the problem is worth solving, and that we are capable and willing to solve it, and that this quest will bring you joy.

We will be exploring a three-step framework designed to help you identify, analyze, and select problems to solve that are meaningful to you. By leveraging tools like the 21-day solution journal, RICE prioritization, and the Ikigai model, this guide will equip you with the strategies to discover and choose opportunities that are not only worth solving, but that will bring you happiness in doing so.

The Finding a Problem to Solve Framework

Problem Identification: The 21 Day Solution Journal

The 21 Day Solution Journal is the tool to help you identify problems as you encounter them in your everyday life. For the next three weeks commit to noting down every problem you encounter, no matter how big or small, or whether it’s at home, work or in your community. By the end of the 21 days, you will have a list of problems to analyze further and hopefully you have the problem that will bring meaning to your life.

How it works

1. For 21 days*, write down 5-10 problems you encounter each day. These could be things that annoy you, inefficiencies that you notice, or challenges that you face.

2. By the end of the 21 days, review your list and look for patterns or recurring themes. and move on to the problem analysis phase.

The idea is that by consistently observing and recording problems, you'll train yourself to be more attuned to opportunities for improvement. It's a way to generate a large quantity of potential ideas that are grounded in real problems that impact you.

*you can adjust the amount of time to more or less days, but you should keep a minimum of 7 and a maximum of 30 days in your journal.

Solution Journal Template

Entry Title: [Short description of the challenge/opportunity]
Date: [YYYY-MM-DD]

Category: [e.g., Technology, Health, Education, etc.]
Scope: [Scope, home, community]
Type: [e.g., Challenge, Opportunity]
Context: [Brief description of the situation or environment]

Description:

• Details: [Detailed description of the challenge or opportunity]

• Impact: [How does this challenge/opportunity affect you or others?]

• Possible Solutions: [Initial thoughts on potential solutions]

RICE Measures:

• Reach: [How many people are affected by this?]

• Impact: [What is the potential positive impact if solved?]

• Confidence: [How confident are you in your understanding of this challenge/opportunity (50% if you not sure, 80% if you are somewhat sure, 100% if you are certain?]

• Effort: [How much effort would it take to address this?]

Ikigai Measures:

• What You Love: (L = )[Do you feel passionate about this challenge/opportunity?]

• What You're Good At: (G= ) [Do you have skills that could address this challenge/opportunity?]

• What The World Needs: (W = )[Is this something that the world needs or could benefit from?]

• What You Can Be Paid For: (P = ) [Is there economic viability in solving this challenge/opportunity?]

Reflection:

• Reflection Notes: [Any other thoughts or insights]

Scoring:

RICE Score Calculation:

RICE Score=Reach × Impact × Confidence / Effort

Ikigai Score Calculation:

Ikigai Score = L + G + W + P

Total Score Calculation:

Total Score = RICE Score + Ikigai Score

Problem Analysis

The solution journal will have walked you through the process of identifying and scoring your potential candidates using RICE prioritization and Ikigai criteria and provided you with a total score to help you rank the problems that matter the most to you.

Analyzing Your List

1. Identify High Scoring Problems: Look for problems that score high in both RICE and Ikigai. These are likely to be impactful, feasible, and personally meaningful.

2. Look for Patterns: See if there are any common themes or categories among the high-scoring problems. This can help you focus your efforts on a specific area.

3. Consider Feasibility: Ensure that the problems you are most interested in are also feasible given your resources, skills, and time.

4. Assess Passion and Joy: Reflect on whether these problems excite you and bring you joy. Long-term projects require sustained motivation.

Problem Selection: RICE and Ikigai

In this phase your are going to select the top 3 problems based on the total score from the Solution Journal.

Things to think about while analyzing:

• Which are the 2-3 problems that rank highest using the RICE prioritization criteria?

• Which are the 2-3 problems that rank highest using the Ikigai criteria?

• Do you see yourself working on this project 6 months, or +1 years from now?

Selecting the problems to solve with the highest total score, ensures that the problems you choose are not only worth solving but that they also align with your passions and strengths and will ultimately bring your joy. Narrowing down your focus will allow you to channel your energy and resources effectively, increasing your chances of success.

An Example : from problems to solutions

Problem Identification: During the 21-day challenge, you noted a recurring issue with public transportation in your city being unreliable and causing delays. This affects thousands of commuters daily, leading to lost productivity and frustration.

Analysis: You scored this problem using the RICE and Ikigai measures:

RICE Measures:

• Reach: High (9) - This problem affects thousands of commuters daily.

• Impact: High (8) - Improving transportation can significantly enhance daily lives.

• Confidence: Moderate (7) - You have some understanding of the transportation system.

• Effort: High (5) - Requires significant resources and coordination.

RICE Score Calculation:

RICE Score = Reach × Impact × Confidence / Effort
RICE Score = 9 × 8 × 7 / 5 = 100.8

Ikigai Measures:

• What You Love (L): High (9) - Passionate about urban development and technology.

• What You're Good At (G): High (8) - Background in software development and engineering.

• What The World Needs (W): High (8) - Efficient transportation is crucial for city life.

• What You Can Be Paid For (P): Moderate (6) - Potential for funding and support through a tech startup.

Ikigai Score Calculation:
Ikigai Score = 𝐿 + 𝐺 + 𝑊 + 𝑃
Ikigai Score = 9 + 8 + 8 + 6 = 31

Total Score Calculation:

Total Score = RICE Score + Ikigai Score
Total Score = 100.8 + 31 = 131.8

Conclusion

Finding the right problem to solve is a journey that requires introspection, research, and strategic analysis. By following the steps outlined in this guide—from identification and analysis to selection and practical application—you can ensure that the problems you choose to tackle are not only impactful but also aligned with your passions and strengths. Embrace the process, and you'll be well on your way to making a meaningful difference in the world.

The Japanese concept of ikigai, meaning "a reason for being," offers a framework to help you find the intersection of what you love, what you are good at, what the world needs, and what you can be paid for. We will explore a practical scoring system based on ikigai to help you prioritize potential startup projects, ensuring that they align with your passions, skills, and market opportunities.

Ikigai Scoring System Framework

1. Passion (What You Love): How much do you love this problem or area?

2. Vocation (What You Are Good At) : How good are you at solving this problem? Do you have relevant experience in this area?

3. Mission (What the World Needs) : How important is this problem for the world? Is there a real need?

4. Profession (What You Can Be Paid For): How viable is this problem as a business? Can it generate revenue?

Each project will be scored on these four criteria. We can use a 0-10 scale for each criterion, with 0 being the lowest and 10 being the highest. Additionally, we can assign weights to each criterion based on your personal priorities.

An example of weights for each criteria:

- Passion: 25%

- Vocation: 25%

- Mission: 30%

- Profession: 20%

Calculate Total Score

Use the following formula to calculate the weighted score for each project:

For Project A:

Compare and Prioritize Projects

Repeat the scoring process for all potential projects and compare the total scores. Higher scores indicate better alignment with your ikigai.

Example Projects Comparison

In this example, Project B has the highest total score and might be the best fit for you to pursue.

This scoring system is flexible and can be tailored to match your specific goals and needs by adjusting the weights.

Conclusion

Choosing the right problem to solve can help you find the right passion project or startup idea and it is crucial for long-term success and personal fulfillment. By applying the ikigai-based scoring system, you can evaluate potential projects and ensure they align with your passions, skills, and business needs. This approach not only increases the likelihood of your startup's success but also ensures that your work is meaningful and rewarding and most importantly that it ultimately brings you joy.

Before rushing in to dismantle what we perceive to be outdated or inefficient, it is important to pause and consider why those systems are in place.

Understanding the Principle

Chesterton's Fence, is named after the English writer G.K. Chesterton, and it states that before removing a "fence" (or any established system or idea), we must first understand why it was built. The parable goes like this: If you come across a fence in a field and don't know its purpose, it would be foolish to tear it down without first investigating why it's there. The fence might be serving a vital function that isn't immediately apparent.

This principle applies to the fences in our everyday lives such as policies, processes, and beliefs that shape our organizations. Before we advocate for change, we must take the time to comprehend the reasoning behind the existing structures. We need to prevent the negative unintended consequences of making changes without thought of the impacts those changes bring.

Applying Chesterton's Fence in Practice

When evaluating potential changes, Chesterton's Fence encourages us to dig deeper and seek a more comprehensive understanding of the status quo. Here are some ways to apply this mental model:

1. Policy Changes: Before lobbying for a new policy, research the history and intent of the current one. What problem was it designed to solve? What unintended consequences might arise from its removal?

2. Process Improvements: When streamlining a process, investigate why each step exists. Some steps that seem inefficient might be serving a crucial purpose, such as ensuring quality or compliance.

3. Belief Systems: Before dismissing a long-held belief, try to understand its origins and the value it provides to those who hold it. This empathy can lead to more constructive dialogues and mutually beneficial solutions.

The Value of Thoughtful Progress

Chesterton's Fence reminds us that not all change is progress. By taking the time to understand the reasoning behind existing structures, we can avoid reckless innovation and unintended consequences.

This mental model encourages us to ask questions, seek diverse perspectives, and challenge our assumptions. By doing so, we can identify which "fences" are truly unnecessary and which ones still serve a valuable purpose.

Sometimes things that seem inefficient are in place for a reason, understanding these reasons will lead us to building better solutions. Understanding the past is crucial for meaningful change.

Mental models are frameworks, concepts, and heuristics that help us make sense of the world and make better decisions. They are simplified representations of how things work, and they are derived from many disciplines including economics, psychology, physics, biology, and philosophy.

Some examples of well-known mental models:

• Occam's Razor: The simplest explanation is usually the correct one

• Pareto Principle: 80% of outcomes come from 20% of causes

• Confirmation Bias: We tend to search for and focus on information that confirms our preexisting beliefs

• Incentives: Understanding motivations and incentives is key to analyzing systems and behaviors

Mental models on Gatherthink:

• The Pugh Matrix

• Design Patterns Inspired by the Art of War

• The Dunning-Kruger Effect

• Navigating Probability, Uncertainty, and Randomness

Why are mental models useful?

Mental models give us powerful tools to:

1. Quickly grasp the key drivers of a situation

2. Spot patterns and connect ideas across domains

3. Make more rational, less biased decisions

4. Predict likely outcomes and unintended consequences

5. Communicate and reason more effectively

The more mental models we have in our toolkit, the better we think and the better decisions we make. By combining models, we gain insights we would not see through a single lens.

Applying Mental Models to Aid Decision Making

The primary focus of Gatherthink will be to explore how mental models can be practically applied to improve our thinking and decision making. In upcoming posts, we'll dive deep into specific mental models, providing real-world examples and case studies of how they can help us:

• Avoid cognitive biases and logical fallacies

• Analyze complex systems and situations

• Make better decisions under uncertainty

• Create frameworks for better thinking

Resources

• Farnam Street - Shane Parrish's blog dedicated to mental models and better thinking

• James Clear's Mental Models - A curated list of mental models

• Untools - A collection of thinking tools and frameworks

What mental models are you most excited to learn about? What decisions or areas of your life could you apply them to? Let me know in the comments!

How the Pugh Matrix works

1. Define Criteria:

• Start by determining the essential factors or requirements that your solution needs to fulfill (e.g., cost, ease of use, safety, features, etc.).

• Assign weights to each criterion if some are more critical than others.

2. List Alternatives:

• Clearly identify the different possible solutions or options you are considering.

3. Establish a Baseline:

• Select one of your solutions to be the "baseline" or reference point. This is often the current situation or a standard option.

4. Create the Matrix:

• Draw a table with columns for each of your criteria and rows for your alternatives. Include a row for the baseline option.

5. Compare and Score:

• For each alternative and criterion, compare it to the baseline using symbols:

  • + (Better): If the alternative is superior to the baseline for that criterion. Add 1 point

  • S (Same): If the alternative is equivalent to the baseline.

  • - (Worse): If the alternative is inferior to the baseline. Subtract 1 point

6. Calculate Totals:

• Count the number of +, S, and - symbols for each alternative.

• Multiply the scores by the weights (if you assigned any) and sum them to get the total weighted score for each option.

7. Analyze and Decide:

• The alternative(s) with the highest total score(s) represent the most promising solutions based on your criteria.

Pugh Matrix Example

Calculation

Using these steps, let's calculate the scores for Options A, B (Baseline), and C:

• Option A:

  • Cost: -1 (negative) * 3 (weight) = -3

  • Features: +1 (positive) * 4 (weight) = +4

  • Usability: +1 (positive) * 2 (weight) = +2

  • Total Score for Option A: -3 + 4 + 2 = 3

• Option B (Baseline):

  • Since Option B is the baseline, all its scores will be zero (0), making its total score 0 by definition.

• Option C:

  • Cost: +1 (positive) * 3 (weight) = +3

  • Features: +1 (positive) * 4 (weight) = +4

  • Usability: 0 (same as baseline) * 2 (weight) = 0

  • Total Score for Option C: 3 + 4 + 0 = 7

Analysis

Option C scored highest with 7 points. Based on this, Option C might be favored because it is less expensive than Option A and has more features than the baseline B.

Benefits of using a Pugh Matrix

• Clarity: It provides a structured approach for making decisions.

• Objectivity: Helps reduce biases by focusing on defined criteria.

• Collaboration: Can be used as a team exercise to gain different perspectives.

• Documentation: Provides a record of how you reached your decision.

The Pugh Matrix offers a powerful tool to simplify and streamline the decision-making process, allowing you to evaluate multiple options objectively and systematically.

What is Role Prompting?

Role prompting means explicitly telling the AI what role you want it to play in the conversation. This helps scope and focus the AI's responses. Some examples of roles you can assign to the AI include:

• A subject matter expert (e.g. a historian, scientist, lawyer, doctor)

• A writing assistant (e.g. to help brainstorm ideas or edit your work)

• A specific character or persona (e.g. a pirate, talk show host, or famous person)

• An objective, neutral analyst to summarize information

• A creative story teller or worldbuilder

Example role prompt

I want you to act as a life coach. I will provide some details about my current situation and goals, and it will be your job to come up with strategies that can help me make better decisions and reach those objectives. This could involve offering advice on various topics, such as creating plans for achieving success or dealing with difficult emotions. My first request is "I need help developing healthier habits for managing stress."

By specifying the AI's role up front, it knows what perspective and tone to take in its responses. The AI can then tailor its language and draw upon the most relevant parts of its knowledge base and give advice as if it was the expert you requested in your prompt.

150 role prompting examples

This is the first in a series of posts on implementing an information diet composed of impactful, high value content. The idea is to categorize information sources by their depth, breadth, and flow of content using the analogy of bodies of water.

The goal is to spend 80% of your time consuming content from the Ocean, 10% from Lakes and the remaining 10% split between content from rivers and streams.

Content sources from the least important to most important:

Streams

Streams are made up of social media, and blog content. Composed of content that is short, and fast-flowing. The reliability of the information in streams can be a concern. Streams are useful for picking up emerging trends or getting a sense of the "pulse" of a topic. Stream content needs a lot of filtering, and the time spent here should be limited.

• X: Great to keep abreast of news and trends especially from accounts you trust. Maximize your use of X by leveraging lists on topics and accounts that you trust and provide value.

• Blogs

Rivers

News websites and popular podcasts – A wider, more organized flow. May have a discernible direction/agenda. reliability and quality vary based on the source. Upstream rivers have high quality content but downstream rivers may have reliability issue and biases.

• Magazines - Choose high quality content like The Economist.

• Newspapers Provide a daily flow of diverse and current information.

• Podcasts

Lakes

Lakes are a contained body of knowledge, and are made up of non- classical books, trade publications, TED Talks, and some magazines. Content here can be deep and specialized, or offer a broad survey of a field. Lakes can become stagnant and might not pass the test of time.

• Books (Non-Classics)

• TED Talks

• Documentaries

• Trade publications

Oceans

Oceans are wide and vast, and are composed of foundational texts, classical books, and peer-reviewed journals. The content here has passed the test of time, and this is where you should spend the majority of your time (80%). Content in this category have met the lindy effect test.

• Books (Classic): foundational texts and Classic books, with their enduring relevance and depth.

• Scientific Papers / Peer reviewed journals

There was fear that when AI would play chess better than humans, it would mean the end of chess as a human sport, but the opposite has happened. Chess has become even more popular, and training and learning from chess engines has propelled an improvement in human play.

There is an intangible beauty in human imperfection, the swashbuckling tactic played by a human player may not be algorithmically correct, but in it there is perhaps more beauty, than a combination derived from an computer chess engine. While there is much to be concerned about, the upcoming interaction between human and machine will lead to exiting and creative times.

Gaining an advantage

Early adaptors of AI, will be able to gain a competitive advantage against those that shun AI. Artificial intelligence is a great levelling tool for those in the middle of the bell curve. Top creatives in art, writing, and programming will be negatively impacted as AI can cause the mediocre to rise and become competitive with the most talented.

Revolutionizing the way we learn and improve based on chess:

• Chess players review games with chess engines. By reviewing past decision and providing feedback, AI can serve as an insightful mentor.

• It can help us ask critical 'what if' questions, promoting analytical thinking.

• It can also aid in building upon an idea, such as developing a chess opening.

What we don't know

Even at this early stage, there is much we don’t know. Things are muddy even when trying to forecast the current and near current state of of Large Language Models (LLMs) at the level of ChatGPT 4 . The implications of Artificial General Intelligence (AGI) would be impactful and impossible to forecast.

While AI's triumph in chess offers significant insights, it also raises many questions. The absence of an economic facet in chess compared to business contexts is one key difference to consider.

Some unknowns include:

• The emergence of new careers and occupations.

• The rise of mediocrity as a potential consequence of AI.

• Identifying the group likely to suffer adverse effects due to AI in the near future, known as the AI dispossessed.

• What the top 1% need to do to maintain their competitive advantage. Fighting back against LLM training using their content and art styles is one example.

• The ethical use of the technology, one negative in chess has been the rise of cheating by using chess engines to play online chess.

Balancing the promises and challenges of AI

Overall, the rise of AI in chess offers a microcosm for understanding potential AI developments in other domains. It highlights both the promises and challenges that AI presents, and underscores the need for thoughtful, ethical, and inclusive AI governance. While there may be negative impacts, it is important that you are able to reap any advantages that this new technology will offer you, since this will be the differentiating factor that will propel and keep you relevant in the years to come.

Specific foreseeable impacts include:

• Top 1% of creatives will feel an impact as they will have more competition from the rest of us that are utilizing AI to artificially upskill.

• A new way to learn and grow. The educational system needs to adapt and incorporate AI into the curriculum and teaching practices.

• The value of imperfection: craftsmen will be valued and their products will appreciate in value.

• Short-term employment impact when corporations replace human employees with AI. However, current limitations in AI could render some of these changes temporary, with human employees being reinstated when AI proves less cost-effective. This will also be offset by the creation of new careers and an expansion of machine learning related jobs.

Ultimately, humans will find novel and interesting ways to adapt, creating something new and beautiful that we cannot foresee.

The problem with the world is that the intelligent people are full of doubts, while the stupid ones are full of confidence.” —Charles Bukowski

The Dunning-Kruger effect is a cognitive bias where individuals with low ability at a task overestimate their ability, while individuals with high ability at a task underestimate their ability. It is named after David Dunning and Justin Kruger, the two social psychologists who first investigated the bias.

According to Dunning and Kruger's research, this phenomenon occurs for a couple of reasons:

1. Incompetence prevents the recognition of incompetence: This means that people with low ability at a task are not able to recognize their own lack of skill. They don't know what they don't know, so they tend to believe they're more competent than they really are.

2. Experts are more aware of the complexity and depth of a subject: On the other end of the spectrum, people with a high level of skill or knowledge are more aware of what they don't know and can see the complexity and depth of a subject. Therefore, they may underestimate their abilities compared to others, thinking that if it's difficult for them, it must be even more difficult for others. This is sometimes referred to as the "curse of knowledge."

The Dunning-Kruger effect has been observed in a variety of areas, including logical reasoning, financial knowledge, medical knowledge, driving skills, and more.

One interesting note is that as people gain more experience and knowledge in a subject, they usually pass through a sort of "U-shaped" confidence curve. They start out overconfident, then as they learn more, their confidence drops. Then, as they become truly competent, their confidence rises again, but now it's based on a more realistic self-assessment.

Leveraging the Dunning-Kruger Effect

Using the Dunning-Kruger effect as a mental model can be very useful in a number of ways, from personal growth and self-improvement to better managing people in a work environment. Here are a few ways you can leverage it:

1. Self-Awareness and Humility: Being aware of the Dunning-Kruger effect can help you approach tasks and challenges with a healthy dose of humility. Knowing that you might overestimate your skills when you're a novice at something can prompt you to seek additional training, feedback, and advice.

2. Continuous Learning: Understanding the Dunning-Kruger effect can serve as a reminder of the importance of lifelong learning and continuous self-improvement. By acknowledging that true mastery of a subject involves recognizing the complexity and nuances that aren't visible at the outset, you can foster a mindset of constant learning and improvement.

3. Better Communication: In contexts where you are the expert, you can use this effect to better communicate with non-experts. It reminds you that things you find obvious might not be so for others and that you need to adjust your language and explanations accordingly.

4. Critical Decision Making: When making important decisions or judgments, consider the possibility that the Dunning-Kruger effect may be influencing your perceptions. Seek out the opinions and expertise of others to ensure you're not being overly confident in your own abilities or understanding.

The key to leveraging the Dunning-Kruger effect is recognizing it both in yourself and others, which can lead to improved self-awareness, communication, and decision-making.

In the next five years, we can expect AI to continue its rapid advancement, becoming even more integrated into our industries, services, and daily lives. This will invariably lead to a significant transformation in the job landscape, the economy, and in turn, our societies.

The Changing Job Landscape

As AI becomes more sophisticated, it will automate many routine and repetitive tasks. This doesn't necessarily mean widespread job loss, but rather a shift in the nature of jobs available. Jobs requiring creativity, critical thinking, and human touch will become more prevalent. On the other hand, the demand for purely technical skills or those involving routine tasks will decrease.

To stay relevant in this changing landscape, people need to invest in lifelong learning. Core skills such as critical thinking, creativity, complex problem-solving, and emotional intelligence will become even more vital. These are areas that AI, in its current state, struggles to replicate. Learning to work alongside AI will also be essential, understanding its strengths and weaknesses and leveraging it as a tool rather than seeing it as a threat.

Government and AI

Now, let's move on to the role of governments in this transition. Government has a crucial role in ensuring a just and equitable transition to an AI-driven economy. This includes investing in education and re-skilling initiatives to prepare the workforce for the shift. Social safety nets need to be strengthened and potentially reimagined to support those displaced during the transition. Policies need to be formulated to protect privacy and prevent misuse of AI. Governments also need to invest in infrastructure that enables AI integration, from high-speed internet to data centers.

Perhaps most importantly, we need to address the economic implications. AI has the potential to drastically increase productivity and wealth, but it also runs the risk of exacerbating inequality. If the benefits of AI are concentrated in the hands of a few, we risk creating a society of haves and have-nots. Governments need to ensure that the benefits of AI are broadly shared, which might involve innovative economic policies such as wealth redistribution or universal basic income.

Towards a New Paradigm

Lastly, we need a philosophical shift. We need to redefine our understanding of work and its role in our lives. If AI and automation take over a significant portion of tasks currently performed by humans, we may find ourselves with more leisure time. The ancient Greeks saw leisure as the pinnacle of human existence, the time for self-improvement, civic participation, and philosophical pursuits. Perhaps it's time we revisited this idea.

The future is always uncertain, but what's certain is that AI is a transformative force. Our task is not to resist this change but to manage it in a way that serves the common good. This requires foresight, adaptability, and a commitment to learning and fairness.

The AI-driven economy is not just about technology; it's about people. It's about our ability to adapt, our resilience, our creativity, and our capacity for empathy. If we remember this, we will not only survive the transition but thrive in it.

As Artificial Intelligence (AI) and large language models (LLMs) like GPT-4 become increasingly prevalent in various applications, including content generation, question-answering, and sentiment analysis, it is crucial to ensure the reliability and accuracy of the information provided. Users must be able to trust the outputs of these AI systems, especially when making critical decisions based on the generated content. However, evaluating the trustworthiness of LLM outputs can be challenging due to their ability to produce plausible-sounding but not always accurate responses a term referred to as hallucinations.

Hallucinations: In the context of Large Language Models (LLMs), the term "hallucinations" refers to generated outputs that may appear coherent and plausible but are factually incorrect, irrelevant, or inconsistent with the input provided. Hallucinations occur when the model produces information that isn't grounded in the data it has been trained on or when it extrapolates beyond its training data in an incorrect or nonsensical way. Techniques such as prompt engineering, model fine-tuning, and incorporating confidence scores can help mitigate the risk of hallucinations and improve the reliability of LLM outputs.

An LLM Confidence Score System, can help evaluate and quantify the reliability of LLM outputs. The system assesses the output's quality based on multiple criteria, including the relevance and authoritativeness of sources, recency of information, consistency among sources, and the language model's inherent uncertainty. By calculating a confidence score on a scale of 0 to 100%, users can gain insight into the trustworthiness of the AI-generated content, allowing them to make more informed decisions.

How it works

Creating a confidence score for results returned by a large language model (LLM) like GPT-4 can be an effective way to gauge the reliability of the output. Although there isn't a universally accepted method for this, I can suggest an approach that considers various criteria and provides a confidence score on a 100% scale.

Here's a list of criteria you might consider:

1. Relevance of sources (Rs): The degree to which the sources are directly related to the query.

2. Authoritativeness of sources (As): The credibility and expertise of the sources in the subject matter.

3. Recency of sources (Rc): The freshness of the information, with more recent sources having higher value.

4. Consistency of information (Ci): The level of agreement among sources on the given information.

5. Language model's uncertainty (Lu): The inherent uncertainty in the model's output, which can be determined by analyzing the model's softmax probabilities (lower uncertainty corresponds to higher confidence).

You can create a weighted formula using these criteria. Assign a weight (w) to each criterion and normalize the values for each of them on a scale of 0 to 1. Then, calculate the confidence score using the following formula:

Confidence Score = 100 (w1 Rs + w2 As + w3 Rc + w4 Ci + w5 (1 - Lu))

Note that the weights should sum up to 1 (w1 + w2 + w3 + w4 + w5 = 1). You can adjust the weights according to the importance you assign to each criterion.

An example

As a starting point, we are assigning equal weights to each criterion in the confidence score formula. However, please keep in mind that these weights are subjective, and you may need to adjust them based on your specific use case, preferences, and observations.

Here are the initial weights:

1. Relevance of sources (Rs) - Weight (w1): 0.2

2. Authoritativeness of sources (As) - Weight (w2): 0.2

3. Recency of sources (Rc) - Weight (w3): 0.2

4. Consistency of information (Ci) - Weight (w4): 0.2

5. Language model's uncertainty (Lu) - Weight (w5): 0.2

Using these weights, the confidence score formula is:

Confidence Score = 100 (w1 Rs + w2 As + w3 Rc + w4 Ci + w5 (1 - Lu))

Sample calculation:

Let's assume you've normalized the values for each criterion on a scale of 0 to 1, and you have the following values:

Rs = 0.9 As = 0.85 Rc = 0.75 Ci = 0.8 Lu = 0.1

Plugging these values into the formula:

Confidence Score = 100 (0.2 0.9 + 0.2 0.85 + 0.2 0.75 + 0.2 0.8 + 0.2 (1 - 0.1)) Confidence Score = 100 (0.18 + 0.17 + 0.15 + 0.16 + 0.18) Confidence Score = 100 0.84 Confidence Score = 84%

In this example, the calculated confidence score is 84%, which indicates a high level of confidence in the LLM output.

Confidence Score Expectations

Here's a potential scale with expectations for confidence scores of LLM results. The scale is divided into five tiers, representing varying levels of confidence in the information:

1. <50% - Low confidence: Results in this tier might not be reliable, and there could be significant issues with the relevance, authoritativeness, or recency of sources. The information may be inconsistent, and the language model's uncertainty could be high. Users should be cautious and seek additional sources to verify the information.

2. 50-70% - Moderate confidence: Results in this range may be useful, but some concerns remain. There might be some inconsistencies among sources or a mix of reputable and less credible sources. While the information may be relevant to the query, it's recommended to cross-check with other sources.

3. 70-80% - Fairly high confidence: Results in this tier are likely to be reliable, with mostly reputable sources, and the information is generally consistent. However, there might be a few minor issues with recency or authoritativeness. The language model's uncertainty could be moderate. It's still a good idea to confirm the information with additional sources.

4. 80-90% - High confidence: Results in this range are considered highly reliable. The information comes from relevant and authoritative sources, with a high degree of consistency. The language model's uncertainty is low, but it's always a good practice to verify the information if it's critical.

5. 90-100% - Very high confidence: This tier represents the highest level of confidence in the results. The information comes from highly reputable sources, and there is strong agreement among them. The language model's uncertainty is very low. While the information is expected to be accurate, it's still essential to exercise caution and verify the information when necessary.

Regardless of the confidence score, it is always a good practice to cross-check information, especially for critical or sensitive topics.

Optimizing the Confidence Scoring System

Optimizing the confidence score system for LLM outputs involves several steps to improve its effectiveness and reliability. Here are some suggestions:

1. Fine-tune the weights: Test different weight combinations for the criteria (relevance, authoritativeness, recency, consistency, and language model's uncertainty) to find the optimal balance that best represents your desired level of confidence. This may require a trial-and-error approach and evaluating the results against real-world examples.

2. Include additional criteria: Consider incorporating other relevant factors that might impact the confidence score, such as sentiment analysis (to identify possible bias in the sources) or diversity of sources (to ensure a variety of perspectives).

3. Develop a training dataset: Create a labeled dataset with real-world examples of LLM outputs and their corresponding confidence scores (as judged by human experts). This dataset can be used to train a machine learning model to predict confidence scores automatically.

4. Train a model for scoring: Using the labeled dataset, train a machine learning model (e.g., a neural network, decision tree, or support vector machine) to predict confidence scores based on the input features (relevance, authoritativeness, recency, consistency, and language model's uncertainty).

5. Evaluate and iterate: Continuously evaluate the performance of the confidence score system using various evaluation metrics (e.g., accuracy, F1 score, precision, and recall) and real-world examples. Refine the model and the criteria weights based on the feedback and insights obtained from this evaluation.

6. Incorporate user feedback: Create a mechanism for users to provide feedback on the perceived confidence and accuracy of the LLM output. This information can be valuable for improving the confidence score system.

7. Periodic updates: As language models and information sources evolve, ensure that the confidence score system is regularly updated to maintain its effectiveness and accuracy.

By following these steps, you can optimize the confidence score system to be more effective in gauging the reliability of LLM outputs. Remember that the process will likely require iterative improvements and ongoing evaluation to ensure the system remains accurate and relevant over time.

A confidence score is sorely needed to gauge the AI generated content landscape. As AI technology continues to evolve, addressing challenges like hallucinations and ensuring the reliability of LLM outputs will remain paramount in fostering trust between AI systems and their users.

Brier Scores, developed by Glenn W. Brier in 1950, provide a numerical evaluation of the accuracy of probabilistic predictions. They measure the mean squared difference between the predicted probabilities and actual outcomes, making it easy to see how well your predictions align with reality.

By tracking your forecasts and calculating your Brier Scores, you can identify patterns and biases in your thinking, ultimately leading to better decision-making.

Calculating Brier Scores

Formula

brier score = (probability - outcome)^2 + (probability - outcome)^2 / N

Where outcome = 1 if win / correct or 0 f loss / incorrect
N = number of items

Example calculation

For example an 81% win forecast versus a 19% loss where the underdog wins

(win probability – 0)^2 + (loss probability – 1)^2
(.81 – 0)^2 + (.19-1)^2 / 2 = .66

If the result would have been a win for the favorite, then your brier score would be:
(.81 – 1)^2 + (.19-0)^2 / 2 = .036

A lower Brier Score indicates more accurate predictions, while a higher score suggests room for improvement.

A Decision Making Journal

A prerequisite to tracking and improving your decisions is to start using a decision-making journal. The journal will help you keep track of your decisions and their outcomes and by incorporating Brier Scores into your journal, you can quantify the accuracy of your forecasts.

Here's an example of what a decision making journal could look like:

1. Date: Record the date of the prediction.

2. Prediction: Describe the event or decision you're forecasting.

3. Probability: Assign a probability to your prediction (from 0 to 1).

4. Reasoning: Briefly explain the rationale behind your prediction.

5. Outcome: Record the actual outcome of the event or decision (1 for true, 0 for false).

6. Brier Score: Calculate the squared difference between the predicted probability and the actual outcome.

7. Brier Score Avg: After accumulating multiple predictions, calculate the mean of all squared differences to get your average Brier Score.

An example entry in a decision-making journal could look like this:

By consistently updating your decision-making journal and calculating your Brier Scores, you can identify patterns, biases, and areas for improvement in your decision-making process. Over time, you'll become more aware of your forecasting strengths and weaknesses, leading to better-informed decisions and more accurate predictions.

Design patterns are like reusable solutions or templates that help solve common problems in a specific context. Think of them as a set of best practices or guidelines that can be applied to different situations.

Design patterns help people save time and effort by using proven solutions instead of starting from scratch every time they face a similar problem.

Sun Tzu's "The Art of War" contains strategic principles and tactics that can be applied to various domains beyond military strategy, including business, management, and even software development.

Here are are design patterns that were inspired by Sun Tzu’s the Art of War to help you solve common problems in a time tested and specific context.

1. Know Yourself and Your Enemy: Understand the strengths and weaknesses of your own system and those of competing systems or adversaries. In software development, this could involve analyzing the capabilities and limitations of your tools, libraries, and frameworks, as well as being aware of competing technologies and potential security threats.
   

   Pattern: Conduct regular assessments of your technology stack, monitor competitors, and stay informed about security vulnerabilities.
   

2. Adapt and Evolve: Embrace change and be prepared to adjust your strategies and tactics in response to shifting conditions. In software development, this might involve adopting new technologies, refactoring code, or pivoting to address changing user needs.
   

   Pattern: Implement agile development practices, promote a culture of continuous learning and improvement, and be open to change.
   

3. Divide and Conquer: Break down complex problems into smaller, more manageable pieces. In software development, this can involve modularizing code, creating microservices, or using design patterns that promote separation of concerns.
   

   Pattern: Use modular programming, microservices, and appropriate design patterns to manage complexity and improve maintainability.
   

4. Plan and Prepare: Develop a clear strategy and make thorough preparations before taking action. In software development, this could include requirements gathering, project planning, designing system architecture, and setting up development and deployment environments.
   

   Pattern: Follow a structured software development life cycle (SDLC) process, and use project management tools and methodologies to ensure proper planning and preparation.
   

5. Speed and Efficiency: Strive for speed and efficiency in execution to gain a competitive advantage. In software development, this might involve optimizing code performance, automating repetitive tasks, and streamlining workflows.
   

   Pattern: Use performance optimization techniques, embrace automation, and continuously refine development processes to improve efficiency.
   

6. Unity and Collaboration: Foster a spirit of unity and collaboration within your team, and work together to achieve common goals. In software development, this could involve creating a supportive team culture, using collaborative tools and practices, and promoting knowledge sharing.
   

   Pattern: Encourage open communication, use collaborative tools and practices (e.g., version control, pair programming, code reviews), and promote a culture of knowledge sharing and mutual support.
   

7. Leverage Terrain: Sun Tzu emphasizes the importance of using the terrain to your advantage in warfare. In software development or business, this could involve capitalizing on your unique positioning, resources, or expertise.
   

   Pattern: Identify and leverage your team's unique strengths, resources, and knowledge to gain a competitive edge.
   

8. Deception and Surprise: Sun Tzu advises using deception and surprise to outwit opponents. In business or technology, this could involve developing innovative, unexpected solutions or launching disruptive products that catch competitors off guard.
   

   Pattern: Encourage creative thinking and risk-taking to foster innovation and develop disruptive solutions that give your organization a competitive advantage.
   

9. Avoid Prolonged Conflict: Prolonged conflict can lead to resource depletion and diminished morale. In software development, this can relate to avoiding projects that drag on indefinitely, addressing technical debt, and managing scope creep.
   

   Pattern: Monitor project timelines, prioritize addressing technical debt, and manage scope to prevent resource exhaustion and maintain team morale. End projects that are not going anywhere and that are creating a drain on the team.
   

10. Conservation of Resources: Sun Tzu advises conserving resources and using them wisely. In software development or business, this can involve optimizing resource allocation, reducing waste, and maximizing return on investment (ROI).
    

    Pattern: Implement resource management practices, minimize waste through process improvement, and focus on high-ROI projects and initiatives.

By translating ancient principles like Sun Tzu's "The Art of War" into design patterns, you can gain valuable insights and guidance for ancient wisdom to overcome challenges, and improve your competitive position.

Emotions in Decision Making

Emotions can significantly impact our decision-making processes. They serve as an internal guidance system, helping us evaluate situations and choose appropriate actions based on our values and priorities. Emotions can be both beneficial and detrimental to decision making, depending on their appropriateness to the situation.

Positive Impact of Emotions

Emotions can facilitate decision making in several ways. For example, positive emotions like happiness, contentment, and excitement can increase our cognitive flexibility and creativity, allowing us to consider alternative solutions and make more informed decisions. Additionally, emotional intelligence - the ability to recognize, understand, and manage our emotions and those of others - can enhance decision-making skills by promoting empathy, effective communication, and teamwork.

Negative Impact of Emotions

On the other hand, emotions can also hinder decision making. Intense emotions, such as anger, fear, or sadness, can impair our ability to think clearly and rationally. When emotions overwhelm our cognitive processes, we may make impulsive decisions or fail to consider all relevant factors. Moreover, emotions can create decision-making biases, leading us to favor options that align with our current emotional state or confirm pre-existing beliefs.

Bias in Decision Making

Cognitive biases are systematic errors in thinking that occur when our brain processes information using mental shortcuts or heuristics. While these shortcuts can be efficient, they often lead to irrational decisions and distorted perceptions of reality. Some common biases that can impact decision making include:

Confirmation Bias

Confirmation bias is the tendency to search for, interpret, and recall information in a way that confirms our pre-existing beliefs or hypotheses. This bias can lead us to selectively focus on evidence that supports our views while ignoring or discounting contradictory information. Consequently, confirmation bias can result in poor decision making by preventing us from considering alternative perspectives or recognizing the flaws in our reasoning.

Anchoring Bias

Anchoring bias refers to the tendency to rely too heavily on the first piece of information encountered when making decisions. For example, when negotiating a price, the initial offer often serves as an anchor that influences subsequent counteroffers and valuations. Anchoring bias can lead to suboptimal decisions by causing us to place disproportionate weight on initial information and insufficient weight on new, relevant data.

Overconfidence Bias

Overconfidence bias occurs when we overestimate our abilities, knowledge, or the accuracy of our predictions. This bias can result in poor decision making by causing us to underestimate risks, overlook potential problems, or fail to seek external input. Overconfidence can also lead to confirmation bias, as our belief in the correctness of our views may make us more likely to dismiss contradictory evidence.

Availability Heuristic

The availability heuristic is a mental shortcut that involves basing judgments on the ease with which examples come to mind. This can lead to decision-making biases when easily recalled information is not representative of the broader context or when more recent events unduly influence our perceptions.

Conclusion

Emotions and biases are an inherent part of human decision making, and their influence can lead to both positive and negative outcomes. By recognizing the impact of emotions and biases on our decisions, we can develop strategies to manage their effects and make better choices. Knowing when bias and emotions are impacting our decision making process is the first step in weeding them out and replacing them with logic and reason to make the best decisions that we can.

Decision: A decision is a choice made after considering alternatives and assessing the potential consequences of each alternative.

Probability: Is a measure of the likelihood of an event occurring. Expressed as a number between 0 and 1, where 0 indicates the event is impossible, and 1 indicates that the event is certain to occur. The probability of an event can be calculated by dividing the number of ways the event can occur by the total possible outcomes. It is a way of quantifying the uncertainty associated with an event or outcome.

Chance: Is a way of describing the degree of randomness and unpredictability associated with a given situation or event.

Uncertainty: Lack of knowledge or information about a particular situation or event. It can be caused by incomplete data, complex systems or unpredictable events.

Randomness: Refers to the property of lacking a pattern or predictability. In a random system, outcomes are not determined by any prior events or factors, and are not influenced by any external factors or causes. It is critical for making accurate predictions and decisions in the face of uncertainty.

Risk: Refers to the potential loss, harm or negative consequences associated with a particular decision, action, or event. Risk can be measured by the likelihood of a negative outcome occurring, as well as the potential severity of that outcome. It is used to inform decision making and manage uncertainty.

Skill / Luck Continuum: It is a concept used to describe the balance between skill and luck involved in determining outcomes of various activities or events. The continuum ranges from activities that are pure skill, such as chess, to activities that are pure luck, such as playing the lottery. Most situations fall somewhere along the continuum.

Bias: Refers to a tendency to favor one option or outcome over others. Bias can arise from personal beliefs, experiences, or preferences, and can affect the decision-making process.

Consequence: A consequence is the outcome or result of a decision. Consequences can be positive or negative, and decision makers must consider the potential consequences of each alternative when making a decision.

Complex Systems: Systems composed of many interconnected parts that interact with each other in multiple ways, often resulting in behavior that is difficult to predict or control.

The basic principle of spaced repetition is that information is more effectively retained when it is revisited at the optimal time. Instead of cramming information into your brain all at once, you can review it in smaller, more manageable chunks over a longer period of time. This allows you to remember the information more easily and for a longer period of time.

Spaced repetition can be applied to a variety of learning activities, including studying for exams, learning new languages, and memorizing important facts and figures. There are several software tools available that use spaced repetition algorithms to help you remember information more effectively, such as Anki and SuperMemo.

The Forgetting Curve

The forgetting curve shows that we tend to forget information very quickly after learning it, but the rate of forgetting slows down over time. By reviewing information just before we forget it, we can reinforce it in our memory and prevent it from being lost.

The forgetting curve typically looks like a steep decline in memory retention soon after learning, followed by a more gradual decline over time. The exact shape of the curve can vary depending on a number of factors, such as the complexity of the information being learned, the individual's prior knowledge of the topic, and the amount of reinforcement the information receives.

By reviewing information just before we're about to forget it, we can reinforce it in our memory and make it more likely that we'll remember it in the long term. The exact timing of spaced repetition intervals will vary depending on the individual and the specific information being learned, but in general, the intervals should be longer for information that is more easily remembered, and shorter for information that is more difficult to remember.

Spaced Repetition Algorithms

1. SuperMemo algorithm: This is one of the oldest and most well-known spaced repetition algorithms. It calculates the optimal time to review information based on how well you've remembered it in the past. The algorithm uses a formula that takes into account how long you've known the information, how often you've reviewed it, and how difficult it is to remember.

2. SM-2 algorithm: This is a modified version of the SuperMemo algorithm that is commonly used in flashcard software, such as Anki. The algorithm adjusts the intervals based on how difficult the information is to remember, and it takes into account how long it has been since you last reviewed the information.

3. Leitner system: This is a simple algorithm that is based on the use of a series of boxes, typically used with flashcards. Information starts in Box 1, and as you review it correctly, it moves to Box 2, then Box 3, and so on. If you get a flashcard wrong, it moves back to Box 1. The algorithm uses the boxes to determine how often to review the information.

4. Mnemosyne algorithm: This algorithm is similar to the SuperMemo algorithm, but it includes a number of additional features, such as the ability to adjust the intervals based on how confident you are in your answer.

SuperMemo Algorithm Explained

The SuperMemo algorithm is was one of the earliest algorithms developed specifically for spaced repetition learning, and it has been used extensively in research on memory and learning.

The SuperMemo algorithm has also been incorporated into a number of popular spaced repetition software tools, including Anki and Mnemosyne. These tools use modified versions of the SuperMemo algorithm to optimize the timing of spaced repetition intervals based on your individual learning needs.

Here's an example of how the algorithm might work:

Let's say you're trying to memorize a list of vocabulary words for your Spanish class. You learn the words for the first time on Monday, and you get them all correct.

According to the SuperMemo algorithm, you should review the words again the next day, Tuesday. If you get them all correct again, you should review them again on Thursday. If you get them all correct again on Thursday, you should review them again the following Monday.

Each time you review the words and get them all correct, the interval between reviews gets longer. If you make a mistake, the algorithm will adjust the interval accordingly, so you review the word more frequently.

To calculate the optimal review interval, the SuperMemo algorithm uses a formula that takes into account how long you've known the information, how often you've reviewed it, and how difficult it is to remember. The formula looks like this:

Interval = Previous interval * Difficulty factor

The difficulty factor is a number between 0 and 1 that represents how difficult the information is to remember. The more difficult the information, the smaller the difficulty factor.

For example, let's say the interval between your first and second review is 1 day, and the difficulty factor for the vocabulary words is 0.8. To calculate the interval between your second and third review, you would multiply the previous interval (1 day) by the difficulty factor (0.8):

Interval = 1 day * 0.8 = 0.8 days (rounded up to 1 day)

So in this case, you would review the vocabulary words again on Thursday, which is 1 day after your second review.

The Future of Spaced Repetition

The following are Neural Network based spaced repetition algorithms that have been developed in the last few years:

1. SRS Transformer: This is a neural network-based algorithm that uses a transformer architecture to optimize spaced repetition intervals. The algorithm takes into account factors such as the difficulty of the material and the learner's performance history to determine the optimal review interval.

2. Deep Spaced Repetition: This is another neural network-based algorithm that uses deep learning to optimize spaced repetition intervals. The algorithm uses a combination of convolutional neural networks and recurrent neural networks to model the forgetting curve and predict the optimal review time for each item.

3. Neural-SR: This is a neural network-based algorithm that uses a deep neural network to model the relationship between spaced repetition intervals and long-term memory retention. The algorithm uses reinforcement learning to determine the optimal review interval for each item based on the learner's performance history.

These neural network-based algorithms are designed to improve upon traditional spaced repetition algorithms by taking into account more complex factors such as the learner's individual characteristics and performance history. By using machine learning techniques to optimize spaced repetition intervals, these algorithms can provide more personalized and effective learning experiences. However, they are also more complex and computationally intensive than traditional algorithms, and may require more data to train effectively.

Conclusion

Spaced repetition has been shown to be effective for a wide range of learning activities, including studying for exams, learning new languages, and memorizing important facts and figures. There are several software tools available that use spaced repetition algorithms to help individuals remember information more effectively. By incorporating spaced repetition into your learning process you can improve your ability to remember and recall information and it can help make you a super learner.