Since my last post about Decision Stories. posted here, I received some feedback about the usefulness of Decision Stories as a product. The main critique is about the ease of use and adoption of this tool. First, it’s not easy to use. As in people need to invest too much effort in thinking about how they made decisions before. And sometimes it’s not easy for the person to recall how they made the decision. They might not be aware the process even if they try to recall. So there needs to be processes to guide for the user to think about how they make those decisions. I will expand these ideas in a later post.
What I want to talk in this post is about is the second point and possible pivots. And that is not a lot of people are making life changing decisions in their personal life. If not many people are using the product on a daily basis, it’s even harder for them to justify the initial investment of effort to record their decisions.
One of the suggestions is to use this tool to help mid-level managers to make everyday decisions, so they can spend more time on other tasks such as personal and employee development. There are potential benefits to this approach. Most decisions made by mid-level managers involve technical decisions that have data to back it up. These decisions are more quantifiable and involve less “gut feeling”. It’s easier for the model to pick up quantifiable features and make reliable predictions.
The type of decisions made by mid-level managers are more homogenous. For example, a customer service call center manger generally worries about the average handle time of calls, and don’t worry about marketing strategies of product lines. Whereas an executive manager needs to think about how the change in product line will impact the type of customer service changes in call centers branch of the customer service organization. And since a consistent decision type will more likely generate accurate decision predictions, this is more likely to work for mid-level managers and not executive levels.
A bonus for the mid-level managers to have consistent decisions is clarity in management expectations. If the managers are comfortable to share the predictions, it could be used to manage expectations within their department. For example, if the employees knows exact how their manager is going to react to a specific scenario, they could adjust their expectations accordingly and synchronize efforts to maximize results. This of course needs trust building between the manager and their direct reports prior to deployment. Ironically, well trusted teams probably already have clear expectations between team members. It’s the new team members that need help the most.
And of course we have to mention that any model prediction involving human decisions requires sanity checks. Blindly trusting a model decision without thinking about consequences invites disasters. Before people start using this more or less automatically, both the manager and their direct reports needs to understand the risk. And if they don’t feel comfortable using the tool, they should be compelled to use it.
I’m interested in what other pros and cons about this application people can think of. I would love to hear any suggestions!
About a month ago, I introduced an idea I had about retaining the intelligence of a human. Rather than building an AI to mimic humans, I want to capture how a human would make decisions. Because I think the decisions we made say a lot about who we are as a person.
So I went after the idea and developed a prototype. I wrote some decision stories or scenarios in life where one needs to make a decision. The decisions have the potential to shape who we are as a person. Each decision story has 4 choices one can choose from. They are not black-or-white decisions. Each decision could be a valid choice, depending on how the person chooses to live their life. Once the person made enough choices from various decision stories, the algorithm could help the person make a choice on a never seen scenario. This could help us make decisions in the event we are either unsure or don’t want emotion to influence our decisions. Or if we recorded the decisions of our role models and would like to know what decisions would they make in the same situation. You can actually ask, “What would X do?”. This is still a work in progress, so I’m just piecing together many parts as we go.
The backend uses a model to summarize how the person made decisions before. And come up with a generalization of the person’s profile. Then use the profile to guide future decisions.
The steps to do this are as follows:
First, go to this page to log in. https://www.panzoto.com/login/. This is used to track the decisions you made based on premade stories.
Then go to this page to make as many decisions as possible. https://www.panzoto.com/add-decision/. I recommend at least 10 decisions, but it will work with just one decision if making decisions is inherently difficult.
Then go to this page to type in the decision you would like to make. You can either provide several choices or simply ask a question. The model will return the choice number, or make up some reasonable choices based on the decisions you made before. https://www.panzoto.com/make-a-decision/.
If you like to submit more stories to help to expand the decision story repository. Please submit those on this page. https://www.panzoto.com/submit-your-decision-story/. Keep in mind to use gender-neutral references when building your story. Refrain from including personal detail in the stories. Make the stories as generic as you can. The stories will be approved by a human.
DISCLAIMER: By using this service, you assume the potential risk of asking a computer model to make suggestions to you. This is meant to be helpful and not inflict harm. It should be used in the case where you seriously considered the pros and cons of at least 2 decisions, but were still unable to make the decision because of your emotion. Please don’t ask for ethically questionable decisions. The model has filters for harmful language and self-inflicting harm. In general, please don’t ask questions that will get this service shut down.
I was always fascinated with artificial general intelligence. In particular, I’m constantly thinking about how to generate a computer equivalent of how humans think and act. It has been difficult to make such systems because the current deep learning techniques is focusing on detection techniques like vision and NLP. Not much is focused on how humans interpret information and make decisions. So I thought about what things can be useful to the masses, yet still reflect how the human brain works.
The project I come up with is in the form of presenting a short story to the user and asking how they make decisions based on the story. When the user needs to make similar decisions in the future, there is a repository of similar stories and decisions made previously by the user, and the repository can be used to mimic the closest decision the user would have made. Although not exactly replicating the intelligence of the user, if we can make similar decisions as the user, then the end result is similar in terms of impact on their life.
Our adult lives are filled with decisions we have to make. As much as we like to, we don’t normally change how we make decisions, unless there is a life-altering event, such as a near-death experience or the loss of a family member. If we can capture how people make decisions, and able to replicate those decisions, then it’s as if we replicated the person’s behavior and values.
It may also be useful to the individual, in exposing the way that they make decisions, and asking if they like to make changes to how they are making those decisions. Pointing out the consequences of the decision-making sometimes can trigger the changes they need, without going through the life-altering experiences
Speech transcription is the process of converting speech audio into text. The text becomes searchable and there is variety of Natural Language Processing (NLP) tools that can make sense of it. Traditionally this is done by humans. Early technology are less accurate (<70%) so the NLP tools does not work affectively. Machine Learning made great strides and increased the accuracy to more than 90%. However this technology is largely inaccessible to an average person or app developer. Training your own model require technical knowledge, and cloud solutions like Google, AWS, or Microsoft Azure is relatively expensive for large quantities.
With Whisper, developers can use their own GPU capable hardware to make mass amount of speech transcriptions. Theoretically, this will enable more exciting solutions that utilize speech transcription technology. I personally like to see some competition in personal assistant field, on wearable technology.
I recently did a journal club explaining Google’s LaMDA model, Language Models for Dialog Applications. Specifically, I liked how they added the external Information Retrieval approach (aka. Google Search) to the existing large language model to ground the dialog. This helps the model to be more ground and believable to a human because it conforms to the human assumptions, 1) agrees with what was said before, and 2) agrees with human common sense (fact check).
After working in the industry for a while, and applying the machine learning knowledge in what works well for business, I have come to the conclusion that human much prefer precision than recall. First thing first, what are precision and recall. The technical definitions for precision is (true positive) / (true positive + false positive), and recall is (true positive) / (true positive + false negative). That formula may be hard to understand, but the intuitive sense is that using precision as the metric will lower the number of false positives, while using recall as the metric will lower the number of false negatives.
Further explaining the technical words, false positives are the number of time where you thought it was true, but it turns out to be false. This is what I traditionally think about mistakes. False negatives are the number of times where you thought it was false, but it is actually true. I think of this as surprises. Taking from the book, Thinking Fast and Slow, people tend to make up reasons and logic to make sure whatever their conclusion is, it is the correct one. Base on that human tendency, false positives may be harder to accept than false negatives. This is because false positives directly violated our believes, assume we are reasonable people, whereas false negative just let ourselves know we learned something new, and it’s much easier to accept.
Given that false positives are harder to take for human beings, we try really hard to minimize the chance of that happening. Before prediction models exists, people have rules to do things. And if the rules help us solve a problem our find a thing, we keep the rules. When false positives happen, we will fix the rules to minimize false positives, but we don’t really try to find where the rule fails to catch the things we want, which are false negatives. And in a sense, before big data and fast predictive models happen, regular people that doesn’t have statistical trainings doesn’t care that much about false negatives. I think we are more comfortable finding false positives than false negatives.
The reality is that I find myself making more models that favors precision than recall when I don’t have the ability to maximize both. Obviously I like to use F1, the harmonic mean of precision and recall, to measure model success. But it’s not always the case given the amount of labeled data I have. In most cases I rather tell my client that I was not able to find all the things you want, than tell my client I may tell you something that’s wrong. The traditional rule approach tend to find things that have high precision and doesn’t cover much ground. And combine a bunch of these rules to get the desired results. I find myself doing similar things, just with prediction models. I keep advertising the high recall models, especially for discovering phases, where my client can find new directions for research, but they have been slow adapting that approach. Maybe they also have a tougher time telling their customers that they are not 100 percent positive about their findings.
Outside of research, I don’t know if I will ever be able to ask my clients to use recall as a metric they truly care about. Maybe it’s because my industry value precision more. Or maybe the word precision sound more positive than recall :). I would love to find out some business examples, where people are more open minded about using recall to solve their problem. And the people who make decisions understand its value.
So I was thinking about using Wikipedia data to make a knowledge base and practice some NLP techniques on it. The first step is to import the English portion of Wikipedia into a mysql database so I query it as needed.
I first tried to download the already made sql, but those SQL script available to download doesn’t actually include the text we see on Wikipedia. So I have go to the XML files, and follow instructions provided by this source.
Basically, we need to use a tool called MWDumper, that will convert XML into SQL scripts. We can download the compile java here, with the instructions here.
This code provided by the blog are mostly correct, except table page have one more column. All we need to do is to add the column like this:
ALTER TABLE page
ADD COLUMN page_counter INT AFTER page_restrictions;
Another change is that one of the column in revision is too small, so we need to change the field property.
ALTER TABLE `revision`
CHANGE `rev_comment` `rev_comment` blob NOT NULL AFTER `rev_text_id`;
There are also duplicate page_titles in page, so make sure they are not set to UNIQUE
ALTER TABLE `page`
ADD INDEX `page_name_title` (`page_namespace`, `page_title`),
ADD INDEX `name_title` (`page_namespace`, `page_title`),
DROP INDEX `page_name_title`,
DROP INDEX `name_title`;
After that it should just be a waiting game until everything is done. My slow server took about 2 days. The final size is about 126 GB on database. Happy NLPing!
Recently Github starting to send out invites for Copilot. It’s a AI assisted code generator for several different languages. For python, it will generate efficient code according to docstring the programmer wrote. For other languages, it will infer from the function declaration. I tested on Leetcode, and the time and space complexity is quite good. Although it struggle with some of the hardest tasks, it’s fulfills the promise it claims to do.
Should you use it though? The way the model is trained, it uses docstring and public available code. There is the obvious licensing issues. Can you use someone’s code, if they did not explicitly state it’s open source, even if it’s a public repository. There have been cases already discovered that have personal info in the comments of code, or in the embedded HTML. That makes people think twice about using it if they might sued later.
Another point is should you use it even if it’s legal. For now, it only generate a single function. I haven’t seen it write a complete class or generate scripts with folder structure. When the program gets more complicated, a lot of the higher CS concepts like cohesion, coupling, and usage of design patterns are more import than writing an efficient function. Therefore, I would put this as a tool for beginner to learn programming than an actual tool for advanced programmer to deploy. I have been learning and debating about when to use object oriented programming and when to use functional programming. I found the following resources to be helpful. For now, I’m still in the camp of learning better structure than blindly using Copilot to generate programs.
#ArjanCodes channel on Youtube: https://www.youtube.com/watch?v=ZsvftkbbrR0&list=PLC0nd42SBTaNuP4iB4L6SJlMaHE71FG6N&index=7
The traditional hypothesis testing I learned in school involves this:
Stating with null hypothesis, which assumes no effects, and
Alternative hypothesis, which state there is an effect.
So we can make the null hypothesis that “Earth is flat”, and the alternative hypothesis that Earth is not flat.
For humans, we can explore the physical world and design experiments to disapprove the null hypothesis. But for a AI, assuming these is no way for the AI to interact with the world through physics. How we can we truly test the hypothesis?
One way to go about this is find a general consensus amongst human knowledge. For example, using Google search result to see whether human agree on the hypothesis. This of course doesn’t mean that human consensus is the truth. In the early times, the consensus was that “Earth is flat”, and the consensus only changed recently if we consider the whole span of human consensus. It is very likely that the human consensus will change over time.
So what then? Do we believe that if AI cannot interact with the physical world, then it cannot prove or disapprove a hypothesis? Is there any alternative? Is there even an approximation? Is it possible that we can deduce from logic and evidence, that upon the collection of enough evidence and applying proper logic that we can disapprove the null hypothesis without physically interact with the world? Sometimes theoretical physics prove the existence of certain things before experiment verification.