{"id":24,"date":"2026-07-21T15:40:00","date_gmt":"2026-07-21T15:40:00","guid":{"rendered":"https:\/\/demos.canvaswp.com\/ai-coding\/2026\/07\/21\/evaluating-a-code-assistant-honestly\/"},"modified":"2026-07-21T15:40:00","modified_gmt":"2026-07-21T15:40:00","slug":"evaluating-a-code-assistant-honestly","status":"publish","type":"post","link":"https:\/\/demos.canvaswp.com\/ai-coding\/2026\/07\/21\/evaluating-a-code-assistant-honestly\/","title":{"rendered":"Evaluating a Code Assistant Honestly"},"content":{"rendered":"<p class=\"wp-block-paragraph\">Benchmarks in this field have a credibility problem, and it is our own fault as an industry. A score on a public set says how well a system does on that set, which is a different question from whether it helps the person typing.<\/p>\n<h2 class=\"wp-block-heading\">What we measure<\/h2>\n<p class=\"wp-block-paragraph\">Three things, all on private repositories with consent: how often a suggestion is accepted unedited, how often an accepted suggestion survives to the merge commit, and how much time a task takes end to end against the same task without the assistant.<\/p>\n<h2 class=\"wp-block-heading\">What we refuse to measure<\/h2>\n<p class=\"wp-block-paragraph\">Lines generated. It is the easiest number to move and the least connected to whether anything got better. A tool that writes more lines than you needed has cost you a review, not saved you a keystroke.<\/p>\n<p class=\"wp-block-paragraph\">We publish the method, the cohort size and the task list. We do not publish a single headline number, because a single number would be the thing everybody optimised and nobody trusted.<\/p>","protected":false},"excerpt":{"rendered":"<p>Every assistant claims a benchmark. Most benchmarks measure the benchmark. This is the evaluation set we use internally and why we publish the method rather than the score.<\/p>\n","protected":false},"author":56,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[9,7],"class_list":["post-24","post","type-post","status-publish","format-standard","hentry","category-research","tag-benchmarks","tag-evaluation"],"_links":{"self":[{"href":"https:\/\/demos.canvaswp.com\/ai-coding\/wp-json\/wp\/v2\/posts\/24","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/demos.canvaswp.com\/ai-coding\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/demos.canvaswp.com\/ai-coding\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/demos.canvaswp.com\/ai-coding\/wp-json\/wp\/v2\/users\/56"}],"replies":[{"embeddable":true,"href":"https:\/\/demos.canvaswp.com\/ai-coding\/wp-json\/wp\/v2\/comments?post=24"}],"version-history":[{"count":0,"href":"https:\/\/demos.canvaswp.com\/ai-coding\/wp-json\/wp\/v2\/posts\/24\/revisions"}],"wp:attachment":[{"href":"https:\/\/demos.canvaswp.com\/ai-coding\/wp-json\/wp\/v2\/media?parent=24"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/demos.canvaswp.com\/ai-coding\/wp-json\/wp\/v2\/categories?post=24"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/demos.canvaswp.com\/ai-coding\/wp-json\/wp\/v2\/tags?post=24"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}