Method

Task set 2.1, 16 October 2026. Everything needed to repeat a measurement is published with the results.

Task set 2.1

40 tasks in 8 groups: small daily tasks, large multi-step tasks, code, browser tasks, routines, research, writing and data. Each task runs at least once a week. The 5 most popular tasks (T01, T02, T04, T07, T08) run twice a week, and the 3 daily routines (T03, T29, T32) every night.

Some tasks measure the cost around a task rather than the task itself: opening a session, connectors that are loaded but not used, resuming a chat after the cache has expired, a handoff to a new chat, and polling.

The tasks page has every prompt, input and checklist. The answer keys are not published while the task set is in use, so they cannot end up in a run.

All inputs are synthetic: fictional people, companies and pages, generated with a fixed seed. No real personal data is used.

How usage is measured

• Source. The usage event list in the Grok Bot account (grok.com/dashboard/usage), read through its own export. Each event has a token count and a UTC time to the minute. Nothing is scraped.
• Unit. Tokens per task: the sum of all Grok Bot events in the window of one run. Plus the share of the weekly pool: tokens divided by the pool of the plan.
• Dollars. Only when a run actually uses on-demand usage. Grok Bot publishes no price per credit, so there is no conversion and no estimate.
• Validity rule. A run only counts when the window from 2 minutes before its start to 2 minutes after its end contains no other events. Otherwise it is logged as invalid: overlap and run again later.
• Quiet window. Every night 03:00-06:00 Amsterdam time, never outside it. The routines start at 03:00, 03:05 and 03:10, then 6 task slots from 03:20. All times are logged in UTC.
• Setup. Each task runs as its own routine in a separate, bare measurement bot: no memory, no skills, the default model. Two exceptions: T11 runs in a copy of that bot with 5 connectors and skills turned on, and T32 gets the public X search tools. Runs are at least 4 minutes apart. The model and version are recorded per run.
• Runs. Each result is the median, min and max of the last 3 valid runs of that task. Fewer than 3 valid runs is marked provisional. Week 1 is a trial round with every task once.

How success is scored

Each task has a fixed checklist with a pass threshold. A run passes when it meets that threshold. Failed runs are reported too, with their token cost.

Test pages

Tasks that need the web use test pages on a separate test site, agent-meter-lab.pages.dev/test/. The routines T03 and T29 read pages that change every day at 00:05 UTC on a fixed 7-day schedule. T32 reads public posts on X. Forms and the login send to test endpoints on that site that only log what they receive.

What is published with each result

• The task set version, all prompts, inputs and checklists
• The plan, the model and its version, and the measurement date
• The number of valid runs and the runs that were thrown out
• The raw usage export and a screenshot of the event list for each window

Limitations

• The usage list has no breakdown per bot. That is why runs only happen in a quiet window, with the validity rule above.
• One task can produce several events. We always take the sum in the window.
• The weekly pool size depends on the plan and is read from the account, not from a public price list.
• Results apply to the plan, model and date they were measured on. Prices and limits can change.

Versions

• 1.0, 16 October 2026: first task set, 10 tasks.
• 2.0, 16 October 2026: 40 tasks, measured every night.
• 2.1, 16 October 2026: 10 tasks replaced (T11, T12, T13, T15, T17, T30, T32, T33, T34, T38). Answer keys are no longer published.

Any change to the task set gets a new version number and is listed here.