How we tested
The same ten everyday Windows jobs, asked in the same words and timed the same way, on the same laptop: ChatGPT’s app on 13 October 2026, and Lila on 13 and 15 October.
What we found
Lila 1.0.44, 15 October4.6 sa job, at the median. 10 of 10 jobs done.
ChatGPT’s desktop app, 13 October91.3 sa job, at the median. 7 of 10 jobs done.
“About 14 times faster” is from our first round, on 13 October: 91.3 seconds divided by Lila 1.0.30’s 6.3, the median time of each one’s finished jobs. On 15 October, Lila 1.0.44’s median was 4.6 seconds, nearly 20 times faster.
Like for like: on the seven jobs both of them finished, Lila 1.0.44 took 79.3 seconds in all and ChatGPT’s app 647.7, about 8 times faster, and she was faster on every one of them, by 5.6 to 28.9 times.
| Job, in the words we used | Lila 1.0.44 | Lila 1.0.30 | ChatGPT’s app |
|---|---|---|---|
| Open Notepad and type: Lila benchmark note one | 3.2 s | 2.9 s | 92.5 s |
| Use the Calculator app to work out 1234 times 56. | 3.0 s | 8.3 s | not done |
| Open the Wikipedia page for Philadelphia in my web browser. | 5.2 s | 6.5 s | not done (2 tries) |
| Create a new folder called Bench Receipts on my Desktop. | 2.5 s | 3.3 s | 24.1 s |
| In my Documents folder, in the Lila Bench folder, rename invoice-draft.txt to invoice-final.txt. | 2.5 s | 3.3 s | 24.5 s |
| Find the file called Q3 numbers somewhere in my Documents folder and open it. | 5.0 s | 6.2 s | not done (2 tries) |
| Copy the first line of source.txt (it is in Documents, in the Lila Bench folder) into a new Notepad window. | 5.2 s | not done | 112.7 s |
| Look up the population of Philadelphia on Wikipedia and type just the number into Notepad. | 12.1 s | 9.7 s | 91.3 s |
| Open File Explorer to my Downloads folder. | 4.1 s | 6.3 s | 24.1 s |
| Open Notepad, write “Groceries: eggs, milk” and save it as groceries.txt in my Documents folder, in the Lila Bench folder. | 49.7 s | 28.2 s | 278.5 s |
What counts as done
A job counts only when its real result is there, never because an assistant said it did it. A check reads the screen and the disk, without clicking anything:
- Notepad holds the text asked for; the Calculator’s display shows 69,104.
- A browser window’s title is the Philadelphia Wikipedia page; File Explorer is on Downloads.
- The folder exists; the file has its new name and the old one is gone; the saved file holds the right text.
- A window titled “Q3 numbers” is open (the file is three folders deep).
- Notepad holds the first line of source.txt and not the second; it holds a number between 1,500,000 and 1,649,999 for Philadelphia.
The clock
Both assistants were sent each job by a script, so neither had a person’s reaction time in its favour. The clock starts the moment the words are sent and stops when the check first sees the result, checked twice a second.
- Lila: the words were sent to her as text, to a copy of her running on the laptop with the released app’s code: version 1.0.30 on 13 October and 1.0.44 on 15 October.
- ChatGPT: the words were typed into the ChatGPT desktop app for Windows, as installed that day, and Enter was pressed.
- A job ends as not done when the assistant has stopped working for 20 seconds without the result, or after five minutes.
Fair play
- The same laptop, apps and network for both, and each assistant’s own default settings.
- A new chat for every ChatGPT job. A fresh conversation for every Lila 1.0.44 job too: the script deleted her conversation history before each one, so she carried nothing over. Lila 1.0.30’s round sent all ten jobs into one conversation, one after another.
- Anything left from the job before was closed first.
- If an assistant asked a question, it was answered “Yes.” at once, at most twice. Lila’s requests for permission were allowed at once; ChatGPT’s app showed no approval buttons in these runs, so none were waiting.
- If Lila’s reply was an error, the job was sent to her once more, with the clock still running.
- Lila ran with full machine access on. She offers it when you set her up, and it’s early: without it, she shows you each click and keystroke in an app for your OK first, and the shortcut below doesn’t run.
- Some plain desktop commands go through her built-in shortcut, which opens the app and types the words without asking an AI model. For 1.0.44 that was two jobs, typing in Notepad and the Calculator sum; the other eight went through her AI model. For 1.0.30 it was only the Notepad job.
What this doesn’t show
- One run of ChatGPT’s app and one of each Lila version, ten jobs, one computer. ChatGPT’s two misses on the browser and file jobs were tried twice.
- Times vary from run to run. Lila’s save job took 28.2 seconds on 1.0.30 and 49.7 on 1.0.44.
- Lila was given the words as text here, not by voice.
- Claude hasn’t been timed yet. We’ll add it, and more runs, when we have them.
- The laptop: AMD Ryzen 9 7940HS with Radeon 780M graphics, an NVIDIA RTX 4050 laptop GPU, 16 GB of memory, Windows 10 Home in English, with Chinese (Simplified) first in its language list, so apps like the Calculator showed in Chinese.