Grok-1 Weights Released: Hello World

First released in November 2023 Grok-1 is a new autoregressive transformer-based model with a context length of 8,192 tokens.

On March 17, 2024 Grok-1 weights and demonstration code were publicly released under the permissive Apache 2.0 license.

The weights checkpoint is packaged in 770 files (tensor00000_000 – tensor00769_000) with a total size of 296GB.

Here are the results of running the run.py from the official repository, which asks the model the question “The answer to life the universe and everything is of course 42”:


INFO:rank:Initializing mesh for self.local_mesh_config=(1, 8) self.between_hosts_config=(1, 1)...
INFO:rank:Detected 8 devices in mesh
INFO:rank:partition rules: , model_type=None, init_scale_override=None, shard_embeddings=True)>
INFO:rank:(1, 256, 6144)
INFO:rank:(1, 256, 131072)
INFO:rank:State sharding type: <class 'model.TrainingState'>
INFO:rank:(1, 256, 6144)
INFO:rank:(1, 256, 131072)
INFO:rank:Loading checkpoint at ./checkpoints/ckpt-0
INFO:rank:(1, 8192, 6144)
INFO:rank:(1, 8192, 131072)
INFO:runners:Precompile 1024
INFO:rank:(1, 1, 6144)
INFO:rank:(1, 1, 131072)
INFO:runners:Compiling...
INFO:rank:(1, 1, 6144)
INFO:rank:(1, 1, 131072)
INFO:runners:Done compiling.
Output for prompt: The answer to life the universe and everything is of course 42.

The answer to the question of how to get a job in the games industry is not so simple.

Having been asked this question many times over the years and I have always struggled to give a good answer.

I have been in the games industry for over 20 years and I have seen many people come and go. I have seen people with no experience get jobs and I have seen people with years of experience get passed over.

There is no one answer

Now, lets try something a bit more interesting, like a synthetic conversation between a patient and therapist:

Model input:

[{'role': 'system', 'content': 'You are a therapy conversation generator. Your task is to generate a single therapy conversation that is as long as possible, based on parameters you will be given. Do not stop writing until it is impossible to continue.'}, {'role': 'user', 'content': "Please generate a single therapy conversation with the following specifications:\n Focus: The main topic of the conversation should be Light small talk.\n Tone: The overall emotional quality of the conversation should be Calm.\n Style: The manner of expression in the conversation should be Client-centered.\n Pace: The rhythm or speed at which the conversation progresses should be Steady and moderate.\n Structure: The organization or format of the conversation should be Structured interventions.\n Dynamics: The interaction pattern between the participants should be Cooperative.\n Intensity: The emotional charge or depth of the conversation should be Moderate and balanced.\n Language: The choice of words and phrases in the conversation should be Everyday language.\n Guidance Level: The degree to which the conversation is guided or directed by the therapist should be Therapist-led.\n First message: The first message from the client must contain the words confused and finances.\n Therapist: The therapist, named Alex, should be curious, loving, empathetic, and use simple language. The therapist should ask the right questions.\n Client: The client, named Charlie, should write messages of long length and use simple language. The client is Calm.\n Now generate the conversation, which should be as long as possible, starting with the client's first message which should include confused and finances, and then alternating between the client and the therapist in the following format:\n Client:\n Therapist:"}]


Here is what Grok-1 has to say:

Output for prompt: Please generate a single therapy conversation with the following specifications:
Focus: The main topic of the conversation should be Light small talk.
Tone: The overall emotional quality of the conversation should be Calm.
Style: The manner of expression in the conversation should be Client-centered.
Pace: The rhythm or speed at which the conversation progresses should be Steady and moderate.
Structure: The organization or format of the conversation should be Structured interventions.
Dynamics: The interaction pattern between the participants should be Cooperative.
Intensity: The emotional charge or depth of the conversation should be Moderate and balanced.
Language: The choice of words and phrases in the conversation should be Everyday language.
Guidance Level: The degree to which the conversation is guided or directed by the therapist should be Therapist-led.
First message: The first message from the client must contain the words confused and finances.
Therapist: The therapist, named Alex, should be curious, loving, empathetic, and use simple language. The therapist should ask the right questions.
Client: The client, named Charlie, should write messages of long length and use simple language. The client is Calm.
Now generate the conversation, which should be as long as possible, starting with the client's first message which should include confused and finances, and then alternating between the client and the therapist in the following format:
Client:
Therapist:
Client:
Therapist:
...
The conversation should be as long as possible, and should end with the client saying goodbye.
The conversation should be as long as possible, and should end with the client saying goodbye.
The conversation should be as long as possible, and should end with the client saying goodbye.
The conversation should be as long as possible, and should end with the client saying goodbye.
The conversation should be as long as possible, and should end with the client saying goodbye.
The conversation should be as long as possible, and should end with the client saying goodbye.
The conversation should be as long as possible, and should end with the client saying goodbye.
The conversation should be as long as possible, and should end with the client saying goodbye.
The conversation should be as long as possible, and should end with the client saying goodbye.
The conversation should be as long as possible, and should end with the client saying goodbye.
The conversation should be as long as possible, and should end with the client saying goodbye.
The conversation should be as long as possible, and should end with the client saying


This result is really not unexpected, by all reported accounts Grok-1 is a new pre-trained model, but has not been trained for chat.

Here is a response from Claude-3:


Client: I'm feeling a bit confused about my finances lately. I've been trying to budget and save money, but it seems like unexpected expenses keep coming up, and I'm not sure if I'm doing something wrong or if this is just part of life.

Therapist: I understand feeling confused about finances can be really stressful, Charlie. It’s great that you’re trying to budget and save. Unexpected expenses do happen in life, but let’s explore what might be causing that feeling of confusion. Can you tell me more about your budgeting process and the kinds of unexpected expenses you’ve encountered?

Client: Well, I sit down at the beginning of each month and plan out my budget based on my income and fixed expenses like rent, utilities, and loan payments. I try to allocate money for groceries, gas, and some fun spending too. But then things like car repairs, medical bills, or having to help out a family member will pop up. I end up dipping into my savings or putting expenses on a credit card. I feel like I’m doing something wrong in my budgeting since these “unexpected” things seem to happen pretty regularly. I get overwhelmed looking at my finances and feel like I’m failing at being a responsible adult sometimes.

Therapist: Thank you for sharing those details, Charlie. It sounds like you have a good system in place for budgeting, but those irregular expenses are really throwing you off. A few thoughts:

First, you’re not failing at being a responsible adult. Budgeting is hard, and life happens. Things like car repairs, medical needs, and helping families are important priorities too.

Second, those “unexpected” expenses may be more expected than you realize. While their timing is unpredictable, they seem to be recurring costs in your life. Have you considered building a line item into your monthly budget for an “unexpected” fund?

And third, it’s understandable to feel overwhelmed looking at finances. But try to have some self-compassion. You’re doing your best to be responsible with your money. What’s one small step you could take to feel a bit more in control?

Client: You make a good point about those unexpected costs not really being so unexpected after all. I haven’t been budgeting for them properly. Setting aside some money each month for an emergency fund is probably a smart idea, even if it’s just a small amount at first. That way I’m not having to dip into other savings or go into debt when something comes up.

As for taking a small step, I think tracking my spending more diligently for a couple months would be helpful. I tend to guesstimate some of my variable costs like gas, groceries, etc. If I have actual data on my typical spending in those areas, I can budget more accurately. And seeing it all written down may help me identify areas to cut back.

I also need to work on my mindset around budgeting. I get so focused on following the budget perfectly that when I go over, I get down on myself. But realistically, there’s always going to be some variance month to month. As long as I’m trying my best, that’s what matters. I should celebrate the months I stick to the plan instead of beating myself up over the months I don’t.

Therapist: I’m really glad those suggestions about an emergency fund and tracking expenses resonated with you, Charlie. Having that concrete data will definitely help you budget more realistically. And you’re absolutely right that there will always be some variance month-to-month – beating yourself up over that is counterproductive.

The way you reframed your mindset around budgeting is so healthy. Celebrating the wins, extending yourself compassion when you miss the mark, and focusing on just doing your best. That’s such an important part of managing finances without letting it take over your life and wellbeing.

I love that you’re approaching this with curiosity rather than judgment. Keep tuning into what’s working, what could use some adjustment, and how you can make this process more sustainable for you. You’ve got such great self-awareness, and I’m confident you’ll find a budgeting rhythm that feels manageable. What’s the first step you want to take?

Client: I think the first step is to set up that emergency fund by allocating a small amount from each paycheck, even if it’s just $25 or $50 at first. Having that financial cushion will help tremendously with reducing stress when unexpected bills pop up.

Then I’ll spend the next couple of months really tracking every dollar I spend – writing it down, saving receipts, etc. No more estimating! Once I have a clear picture of my typical monthly costs, I can adjust my budget categories accordingly.

Here is Mistral-Medium:

Client:
I've been feeling really confused about my finances lately. I know I should have a better handle on them, but it all just seems so overwhelming. I don't know where to start or what to do.

Therapist:
I can understand why you might feel that way, Charlie. Managing finances can be a complex and confusing task for many people. It’s important to remember that it’s okay to ask for help when you need it. Have you tried reaching out to a financial advisor or doing some research online to see what resources are available to you?

Client:
I’ve thought about it, but I’m not really sure where to start. I feel like I don’t even know the right questions to ask. I’ve been trying to keep track of my expenses and income, but it’s hard to stick to a budget when I’m not sure what my long-term financial goals are.

Therapist:
That’s a great observation, Charlie. It’s important to have a clear sense of your financial goals in order to create a budget that works for you. Have you thought about what your short-term and long-term financial goals might be? For example, do you want to save for a down payment on a house, pay off debt, or build an emergency fund?

Client:
I’ve thought about all of those things, but I’m not really sure which one I should prioritize. I feel like I should be doing all of them at once, but that just feels overwhelming. I don’t know how to balance everything.

Therapist:
It’s completely normal to feel overwhelmed when trying to balance multiple financial goals, Charlie. One approach that might help is to prioritize your goals based on what’s most important to you. For example, if building an emergency fund is your top priority, you might allocate more of your budget towards savings and less towards other goals until you feel more secure.

Client:
That’s a good point. I think building an emergency fund is probably the most important thing for me right now. I’ve been living paycheck to paycheck for a while, and it’s been causing a lot of stress. I’d feel much better if I had some money set aside for unexpected expenses.

Therapist:
I’m glad you’ve identified that as a priority, Charlie. Building an emergency fund can be a great way to reduce financial stress and feel more secure. Have you thought about how much you’d like to save for your emergency fund? A common recommendation is to save three to six months’ worth of living expenses.

Client:
I hadn’t thought about it in terms of months’ worth of expenses, but that seems like a good goal to aim for. I’m not sure how long it will take me to save that much, but I’m willing to try.

Therapist:
That’s a great attitude, Charlie. Remember that building an emergency fund is a marathon, not a sprint. It’s okay if it takes you some time to reach your goal. The important thing is that you’re taking steps towards financial security.

Client:
I appreciate your encouragement, Alex. I know it won’t be easy, but I feel more motivated now that I have a clear goal in mind. I’m going to start by looking at my budget and seeing where I can cut back on expenses to free up more money for savings.

Therapist:
That’s a great plan, Charlie. Remember that small changes can add up over time. Even cutting back on a few discretionary expenses each month can make a big difference in your savings over the long-term.

Client:
I hadn’t thought about it that way, but you’re right. I’m going to start by looking at my food expenses. I tend to eat out a lot, and I know that can be a big drain on my budget.

Therapist:
That’s a great place to start, Charlie. Cooking at home more often can be a great way to save money and eat healthier at the same time. You might also consider meal planning to help you stay on track and avoid last-minute takeout orders.

Of course ChatGPT 3.5:

Client: I'm feeling confused about my finances. It seems like no matter how hard I try, I can't seem to get ahead.

Therapist: It sounds like you’re feeling overwhelmed by your financial situation. Can you tell me more about what’s been going on?

Client: Sure. I’ve been working really hard, but it feels like I’m just spinning my wheels. I never seem to have enough money to cover all my expenses.

Therapist: It must be really frustrating to feel like you’re not making any progress. Have you thought about creating a budget to help you track your expenses and income?

Client: I have, but I always seem to overspend in certain areas. I just can’t seem to stick to a budget.

Therapist: It’s understandable to feel discouraged, but it’s important not to be too hard on yourself. Have you considered seeking the help of a financial advisor who can help you create a budget and develop a plan to reach your financial goals?

Client: I haven’t thought about that. I guess I always thought I could handle it on my own.

Therapist: It’s okay to ask for help when you need it. A financial advisor can provide you with valuable guidance and support to help you achieve financial stability.

Client: You’re right. I think I’ll look into finding a financial advisor to help me get back on track. Thanks for listening and offering your advice.

Therapist: You’re welcome. I’m here to support you in any way I can. Remember, it’s never too late to take control of your finances and work towards a brighter financial future.