You’ve hit GPT’s token limit!

All about tokens - AI’s default language!

Okay, how many times have you got this message about ChatGPTs tokens being exhausted? Maybe you

- gave it a 300-400 page report to summarise, or

- spent hours trying to get the right output from it, or

- were just flirting with it for hours..

(I seriously hope it’s not the last one :p)

Jokes apart, a lot of times, ChatGPT says you’ve hit the “token” limit. Today, we’ll explore what this “token” means, and how AI processes words.

So, what’s a token?

A token is simply a chunk of text. AI doesn’t read sentences the way we humans do. It breaks down sentences into tokens, each of which could be a whole word, part of a word, or even a punctuation. For example:

• The word “chat” is one token

• “Chatting” would break into two tokens: "chat" + "ting"

• A punctuation mark like a full stop (.) is also a token

Some examples:

• The sentence "I love AI" has 3 tokens ["I", "love", "AI"

• The sentence “I really love AI and how it’s evolving.” has 10 tokens ["I", "really", "love", "AI", "and", "how", "it", "’s", "evolving", "."]

The logic of tokenization is a bit complex (and frankly even I haven’t understood all of it), so let’s not get into that. Let’s just understand that different LLMs use different tokenization models.

But if you want to see more examples of how a sentence is broken down into tokens in GPT, you can play around with OpenAI’s tokenizer HERE.

Why Do Tokens Matter?

Understanding tokens is crucial because the number of tokens directly impacts how the model processes and responds to inputs. Most LLMs, like GPT, have a token limit. If you go beyond this limit, the input is truncated or might not be processed as intended.

Additionally, token count also impacts the cost of processing requests. And since API calls are priced based on token usage, knowing how tokens work can help optimize efficiency and reduce costs.

Essentially, knowing what tokens are and how they work, forms one of the fundamentals of mastering AI.

But it’s cool..since you’re subscribed to this newsletter, you know it now 😉

Something cool I explored this week in AI

So I was in Goa last week, and (obviously) we clicked a lot of pictures! Now, Goa being a tourist spot, (obviously) a lot of pictures had unwanted people in the background.

So I used this tool called Magic eraser to just remove unwanted people from my life..sorry, my pictures.

Here’s the output I got, in literally 2 minutes! (Ignore the quality of the “after”image - that’s all you get in the free version 🥲)

But isn’t it sooo freaking smooth? All unwanted humans, just poof..gone!

If only we had a magic eraser to remove unwanted people from our lives, zindagi ka safar would be so much easier! 😛

But until then, you can just make do with pictures.

By the way, if you have Google Photos on your phone, this is an in-built feature there too. Give it a try and let me know how it works out?

---

Well, that’s all for today! Hope you liked this piece! If you did, and if you want me to continue writing, why don’t you just hit reply and tell me how you found it?

I’d love to read your responses! 🙂

Couple of other things I’d like to ask (and tell)

1. As you know, this is a pretty new newsletter (and this is only the fourth email). So I’d love if you could tell me what you’re looking for, from this newsletter. It’ll help me write relevant content that will eventually help both of us. Please let me know HERE - Just 3 mandatory questions, won’t take more than a minute!

2. I also write a newsletter simplifying business and money management. It has more than 10,400 subscribers as of today. If you’d like to read more about startups, the economy or money management, subscribe to it here

3. I write smaller, more frequent content on LinkedIn - You can follow me there for byte-sized stuff

4. If you liked this piece, please share it with your friends (SHARE button below). It’ll help spread awareness faster :)