Add Java, Kotlin, Rust, Ruby, C#, Dart, Lua, and Perl highlighting

.changeset/more-languages.md

@@ -0,0 +1,5 @@
+---
+"@tanstack/highlight": minor
+---
+
+Add isolated Java, Kotlin, Rust, Ruby, C#, Dart, Lua, and Perl language definitions. Core and existing selective bundles are unchanged.

README.md

@@ -195,7 +195,7 @@ Available themes: Aurora X, Dracula, GitHub Dark, GitHub Light, Gruvbox Dark, Gr
 
 ## Languages
 
-`apache`, `cmake`, `cpp`, `css`, `diff`, `dockerfile`, `ejs`, `env`, `go`, `html`, `http`, `js`, `json`, `jsx`, `markdown`, `mermaid`, `nginx`, `php`, `plaintext`, `python`, `scheme`, `shell`, `sql`, `svelte`, `toml`, `ts`, `tsrx`, `tsx`, `vue`, and `yaml`.
+`apache`, `cmake`, `cpp`, `csharp`, `css`, `dart`, `diff`, `dockerfile`, `ejs`, `env`, `go`, `html`, `http`, `java`, `js`, `json`, `jsx`, `kotlin`, `lua`, `markdown`, `mermaid`, `nginx`, `perl`, `php`, `plaintext`, `python`, `ruby`, `rust`, `scheme`, `shell`, `sql`, `svelte`, `toml`, `ts`, `tsrx`, `tsx`, `vue`, and `yaml`.
 
 Each language is available from `@tanstack/highlight/languages/<name>`. The aggregate `@tanstack/highlight/languages` entry can tree-shake, while direct subpaths make isolation explicit. Importing only core helpers from the root entry also removes unused language registrations in a compatible bundler.
 
@@ -218,7 +218,7 @@ Local browser bundles, minified with esbuild and compressed independently. KB us
 | Core + TSX | 10.11 KB | 4.26 KB | 3.90 KB |
 | Octane MDX + TypeScript | 13.86 KB | 5.59 KB | 5.17 KB |
 | Nine-language docs set | 16.04 KB | 6.20 KB | 5.66 KB |
-| All 30 languages | 30.72 KB | 10.77 KB | 9.79 KB |
+| All 38 languages | 49.99 KB | 16.82 KB | 15.19 KB |
 
 The following comparison was measured before the 1.0 property-context correction. Re-run the comparison commands below for current timings.
 

docs/guides/performance.md

@@ -8,15 +8,15 @@ CI measures selective browser bundles and highlighting performance on real docum
 
 ## Bundle profiles
 
-`pnpm run size` builds seventeen browser profiles with esbuild and measures minified, gzip, and Brotli bytes independently. It also checks that helper, adapter, and selective language imports retain only the requested modules.
+`pnpm run size` builds twenty-five browser profiles with esbuild and measures minified, gzip, and Brotli bytes independently. It also checks that helper, adapter, and selective language imports retain only the requested modules.
 
 | Profile | Registered languages | Current gzip | CI budget |
 | --- | --- | ---: | ---: |
 | Core | None | 1.82 KB | 2.0 KB |
 | TSX | TSX | 4.26 KB | 4.35 KB |
 | Octane | TypeScript plus Octane MDX adapter | 5.59 KB | 5.7 KB |
 | Docs | CSS, HTML, JS, JSON, JSX, Markdown, Shell, TS, TSX | 6.20 KB | 6.3 KB |
-| All | All 30 definitions | 10.77 KB | 10.9 KB |
+| All | All 38 definitions | 16.82 KB | 17.1 KB |
 
 KB uses 1,000 bytes. Core helpers imported from the root tree-shake to the same engine size. The standalone theme helper is 695 gzip bytes.
 
@@ -26,7 +26,7 @@ Selective profiles are the primary metric. The all-language profile exists to pr
 
 The committed corpus contains 334 real code fences sampled from TanStack documentation, with up to twenty samples per normalized language.
 
-`pnpm run bench` measures tokenization, HTML, Markdown, HAST, line numbers, long decorated blocks, and dedicated C++, CMake, and PHP samples. Each profile reports the median of three samples after two warmup passes, with a 1.2 second CI budget. The main highlighting profile processes at least 10,000 blocks.
+`pnpm run bench` measures tokenization, HTML, Markdown, HAST, line numbers, long decorated blocks, and dedicated C#, C++, CMake, Dart, Java, Kotlin, Lua, Perl, PHP, Ruby, and Rust samples. Each profile reports the median of three samples after two warmup passes, with a 1.2 second CI budget. The main highlighting profile processes at least 10,000 blocks.
 
 A local before-and-after review used the same minified bundle settings, fixtures, and benchmark harness on macOS arm64 with Node 24.15.0:
 
@@ -76,4 +76,4 @@ Context-aware fixes are welcome when they solve common docs code. A change shoul
 
 The correct response to a crossed budget is to inspect the behavior and architecture. Budgets can move when a measured quality improvement justifies the bytes, but the tradeoff must be explicit.
 
-The 1.0 property-context correction adds roughly 230 gzip bytes to the TSX profile without changing core. Local Node 26 gzip results differ slightly from CI compression: the CI docs profile is 6,217 bytes and all languages is 10,838 bytes. Their budgets are 6,300 and 10,900 bytes respectively, retaining a small explicit margin.
+The 1.0 property-context correction adds roughly 230 gzip bytes to the TSX profile without changing core. Local Node 26 gzip results differ slightly from CI compression: the CI docs profile is 6,217 bytes against a 6,300-byte budget, and the all-language budget keeps a comparable allowance above its local measurement.

docs/installation.md

@@ -60,7 +60,7 @@ export const highlighter = createHighlighter({
 })
 ```
 
-The root entry is useful for prototypes, server-only scripts, or sites where the roughly 11 KB gzip all-language build is acceptable:
+The root entry is useful for prototypes, server-only scripts, or sites where the roughly 17 KB gzip all-language build is acceptable:
 
 ```ts
 import { highlight } from '@tanstack/highlight'

docs/language-support.md

@@ -13,23 +13,31 @@ Every language is an isolated definition imported from `@tanstack/highlight/lang
 | Apache | `apache` | - | Directives, tags, comments |
 | CMake | `cmake` | - | Bracket strings/comments, nested variables, generator expressions |
 | C++ | `cpp` | `c++`, `cc`, `cxx`, `hpp`, `hxx` | Raw and prefixed strings, character literals, preprocessor directives, digit separators |
+| C# | `csharp` | `c#`, `cs` | Verbatim, interpolated, and raw strings with nested quotes, character literals, preprocessor directives, line-leading and parameter attributes, contextual and LINQ keywords |
 | CSS | `css` | - | Strings and comments protect inner syntax |
+| Dart | `dart` | - | Interpolated strings with nested quotes and braces, triple-quoted and raw strings, nested block comments, annotations, named-argument labels, contextual keywords |
 | Diff | `diff` | `patch` | Metadata, inserted, and deleted lines |
 | Dockerfile | `dockerfile` | `docker` | Common directives, variables, commands |
 | EJS | `ejs` | - | HTML plus optional JavaScript delegation |
 | Env | `env` | `dotenv` | Properties, values, comments |
 | Go | `go` | `golang` | Raw strings, runes, comments, declarations |
 | HTML | `html` | `htm`, `xml`, `angular-html` | Optional JavaScript/TypeScript and CSS delegation |
 | HTTP | `http` | - | Methods, headers, protocol, paths |
+| Java | `java` | - | Text blocks, character literals, annotations, contextual keywords, digit separators and hex floats |
 | JavaScript | `js` | `javascript`, `mjs`, `cjs`, `js-vue` | Templates, interpolation, regex literals |
 | JSON | `json` | `jsonc`, `json5` | Properties, comments, strings, literals |
 | JSX | `jsx` | - | JavaScript plus contextual JSX tags |
+| Kotlin | `kotlin` | `kt`, `kts` | String templates with nested quotes and braces, raw strings, nested block comments, annotations vs labels, backticked names, contextual soft keywords and accessors |
+| Lua | `lua` | - | Level-matched long-bracket strings and comments, string escapes, Luau backtick strings, shebang lines, paren-less calls, LuaJIT number suffixes |
 | Markdown | `markdown` | `md` | Optional fenced-language delegation |
 | Mermaid | `mermaid` | - | Common diagram declarations and arrows |
 | Nginx | `nginx` | - | Directives, variables, URLs, comments |
+| Perl | `perl` | `pl` | Sigil and special variables, quote-like operators with nested delimiters, regex vs division, heredocs, POD blocks |
 | PHP | `php` | - | PHP tags, attributes, quoted strings, heredoc/nowdoc, optional HTML delegation |
 | Plaintext | `plaintext` | `text`, `txt`, `-->` | Escaping only |
 | Python | `python` | `py` | Triple strings, prefixes, decorators, comments |
+| Ruby | `ruby` | `rb` | Interpolated strings with nested quotes, percent literals, heredocs, regex vs division, symbols and hash keys, block comments |
+| Rust | `rust` | `rs` | Nested block comments, raw strings with hash counts, byte and C strings, lifetimes vs character literals, attributes, macros |
 | Scheme | `scheme` | `scm`, `racket` | Comments, strings, forms, literals |
 | Shell | `shell` | `bash`, `sh`, `zsh`, `cmd`, `console` | Heredocs, parameter expansion, comment boundaries |
 | SQL | `sql` | - | Strings, comments, common SQL clauses |

docs/reference/default-entry.md

@@ -43,7 +43,7 @@ Returns the canonical registered language name for a name or alias. Names are tr
 function listLanguages(): Array<HighlightLanguage>
 ```
 
-Returns the 30 canonical language names registered in `defaultHighlighter`.
+Returns the 38 canonical language names registered in `defaultHighlighter`.
 
 ### `tokenize`
 

docs/reference/languages.md

@@ -24,23 +24,31 @@ const highlighter = createHighlighter({
 | `apache` | `@tanstack/highlight/languages/apache` | None |
 | `cmake` | `@tanstack/highlight/languages/cmake` | None |
 | `cpp` | `@tanstack/highlight/languages/cpp` | `c++`, `cc`, `cxx`, `hpp`, `hxx` |
+| `csharp` | `@tanstack/highlight/languages/csharp` | `c#`, `cs` |
 | `css` | `@tanstack/highlight/languages/css` | None |
+| `dart` | `@tanstack/highlight/languages/dart` | None |
 | `diff` | `@tanstack/highlight/languages/diff` | `patch` |
 | `dockerfile` | `@tanstack/highlight/languages/dockerfile` | `docker` |
 | `ejs` | `@tanstack/highlight/languages/ejs` | None |
 | `env` | `@tanstack/highlight/languages/env` | `dotenv` |
 | `go` | `@tanstack/highlight/languages/go` | `golang` |
 | `html` | `@tanstack/highlight/languages/html` | `htm`, `xml`, `angular-html` |
 | `http` | `@tanstack/highlight/languages/http` | None |
+| `java` | `@tanstack/highlight/languages/java` | None |
 | `js` | `@tanstack/highlight/languages/js` | `javascript`, `mjs`, `cjs`, `js-vue` |
 | `json` | `@tanstack/highlight/languages/json` | `jsonc`, `json5` |
 | `jsx` | `@tanstack/highlight/languages/jsx` | None |
+| `kotlin` | `@tanstack/highlight/languages/kotlin` | `kt`, `kts` |
+| `lua` | `@tanstack/highlight/languages/lua` | None |
 | `markdown` | `@tanstack/highlight/languages/markdown` | `md` |
 | `mermaid` | `@tanstack/highlight/languages/mermaid` | None |
 | `nginx` | `@tanstack/highlight/languages/nginx` | None |
+| `perl` | `@tanstack/highlight/languages/perl` | `pl` |
 | `php` | `@tanstack/highlight/languages/php` | None |
 | `plaintext` | `@tanstack/highlight/languages/plaintext` | `text`, `txt`, `-->` |
 | `python` | `@tanstack/highlight/languages/python` | `py` |
+| `ruby` | `@tanstack/highlight/languages/ruby` | `rb` |
+| `rust` | `@tanstack/highlight/languages/rust` | `rs` |
 | `scheme` | `@tanstack/highlight/languages/scheme` | `scm`, `racket` |
 | `shell` | `@tanstack/highlight/languages/shell` | `bash`, `sh`, `zsh`, `cmd`, `console` |
 | `sql` | `@tanstack/highlight/languages/sql` | None |
@@ -52,6 +60,6 @@ const highlighter = createHighlighter({
 | `vue` | `@tanstack/highlight/languages/vue` | None |
 | `yaml` | `@tanstack/highlight/languages/yaml` | `yml` |
 
-`@tanstack/highlight/languages` re-exports `apache`, `cmake`, `cpp`, `css`, `diff`, `dockerfile`, `ejs`, `env`, `go`, `html`, `http`, `js`, `json`, `jsx`, `markdown`, `mermaid`, `nginx`, `php`, `plaintext`, `python`, `scheme`, `shell`, `sql`, `svelte`, `toml`, `ts`, `tsrx`, `tsx`, `vue`, and `yaml`. The barrel is convenient but individual subpaths make bundle intent explicit.
+`@tanstack/highlight/languages` re-exports `apache`, `cmake`, `cpp`, `csharp`, `css`, `dart`, `diff`, `dockerfile`, `ejs`, `env`, `go`, `html`, `http`, `java`, `js`, `json`, `jsx`, `kotlin`, `lua`, `markdown`, `mermaid`, `nginx`, `perl`, `php`, `plaintext`, `python`, `ruby`, `rust`, `scheme`, `shell`, `sql`, `svelte`, `toml`, `ts`, `tsrx`, `tsx`, `vue`, and `yaml`. The barrel is convenient but individual subpaths make bundle intent explicit.
 
 See the [language support matrix](../language-support) for the context-aware behavior and current scope of each registration.

docs/test-strategy.md

@@ -19,21 +19,21 @@ The suite protects the package's actual product boundary: valid code commonly pu
 
 ## Size Profiles
 
-`pnpm run size` checks seventeen independent browser profiles, including root helpers, language barrel imports, adapters, and themes. Each has minified, gzip, and Brotli budgets. The main highlighter profiles are:
+`pnpm run size` checks twenty-five independent browser profiles, including root helpers, language barrel imports, adapters, and themes. Each has minified, gzip, and Brotli budgets. The main highlighter profiles are:
 
 | Profile | Languages | Gzip budget |
 | --- | --- | ---: |
 | Core | None | 2.0 KB |
 | TSX | TSX | 4.35 KB |
 | Octane | TypeScript plus Octane MDX adapter | 5.7 KB |
 | Docs | CSS, HTML, JS, JSON, JSX, Markdown, Shell, TS, TSX | 6.3 KB |
-| All | All 30 definitions | 10.9 KB |
+| All | All 38 definitions | 17.1 KB |
 
 The selective profiles are the primary product metric. The all-language profile protects the convenience entry from unbounded growth. Bundle graphs reject unexpected language or theme code. Package tests repeat isolation checks through public exports after building.
 
 ## Throughput
 
-`pnpm run bench` measures highlighting, tokenization, Markdown, HAST, line numbers, long numbered blocks, long decorated blocks, and dedicated C++, CMake, and PHP samples. Timings use the median of three samples after two warmup passes. Each profile has a 1.2 second CI budget; the main highlighting profile processes at least 10,000 blocks.
+`pnpm run bench` measures highlighting, tokenization, Markdown, HAST, line numbers, long numbered blocks, long decorated blocks, and dedicated C#, C++, CMake, Dart, Java, Kotlin, Lua, Perl, PHP, Ruby, and Rust samples. Timings use the median of three samples after two warmup passes. Each profile has a 1.2 second CI budget; the main highlighting profile processes at least 10,000 blocks.
 
 `pnpm run compare:sugar-high` compares the overlapping JS/TS/JSX/TSX use case. `pnpm run compare:shiki` compares all supported fixtures. These are directional measurements, not claims of equivalent grammar depth.
 

scripts/bench.mjs

@@ -68,6 +68,54 @@ try {
       observe: (result) => result.html.length,
       targetBlocks: 10_000,
     },
+    csharp: {
+      fixtures: [{ rawLang: 'csharp', code: '#nullable enable\n[Obsolete]\nvar path = @"C:\\\\temp""x""";\nvar text = $"Hi {user.Name} {(ok ? "yes" : "no")}";\nvar query = from item in items where item.Count > 1_000 select item;\nchar c = \'\\n\'; /* done */' }],
+      run: (fixture) => highlight(fixture.code, { lang: fixture.rawLang }),
+      observe: (result) => result.html.length,
+      targetBlocks: 10_000,
+    },
+    dart: {
+      fixtures: [{ rawLang: 'dart', code: '@immutable\nclass Point {\n  final greeting = \'Hi ${user.name} ${map[\'key\']}\';\n  final raw = r\'C:\\path\\$name\';\n  /* outer /* nested */ comment */\n  void show() => print(\'\'\'multi\nline\'\'\', count: 0x1F);\n}' }],
+      run: (fixture) => highlight(fixture.code, { lang: fixture.rawLang }),
+      observe: (result) => result.html.length,
+      targetBlocks: 10_000,
+    },
+    java: {
+      fixtures: [{ rawLang: 'java', code: '@Override\npublic record Point(int x, int y) {}\nString text = """\n    Hello "world"\n    """;\nchar c = \'\\u0041\';\nvar mask = 0x7fff_ffff; double d = 0x1.8p1; // done' }],
+      run: (fixture) => highlight(fixture.code, { lang: fixture.rawLang }),
+      observe: (result) => result.html.length,
+      targetBlocks: 10_000,
+    },
+    kotlin: {
+      fixtures: [{ rawLang: 'kotlin', code: '@file:JvmName("Main")\nval greeting = "Hi ${user.name.let { "[$it]" }} $count"\nval raw = """C:\\path ${\'$\'}x"""\n/* outer /* nested */ comment */\nfun `my test`() = loop@ for (i in 0..10) break@loop' }],
+      run: (fixture) => highlight(fixture.code, { lang: fixture.rawLang }),
+      observe: (result) => result.html.length,
+      targetBlocks: 10_000,
+    },
+    lua: {
+      fixtures: [{ rawLang: 'lua', code: '#!/usr/bin/env lua\nlocal text = [==[ long ]] string ]==]\n--[[ block\ncomment ]]\nlocal n = 0x10ULL + 1.5e3\nrequire "module"\nprint("tab\\t" .. text)' }],
+      run: (fixture) => highlight(fixture.code, { lang: fixture.rawLang }),
+      observe: (result) => result.html.length,
+      targetBlocks: 10_000,
+    },
+    perl: {
+      fixtures: [{ rawLang: 'perl', code: 'my %h = (key => $ARGV[0], list => [@_]);\nmy $q = qw{a {b} c};\n$x =~ s/foo/bar/g; my $y = $a / $b;\nprint <<"END";\nHello $name\nEND\n=pod\nDocs\n=cut' }],
+      run: (fixture) => highlight(fixture.code, { lang: fixture.rawLang }),
+      observe: (result) => result.html.length,
+      targetBlocks: 10_000,
+    },
+    ruby: {
+      fixtures: [{ rawLang: 'ruby', code: 'greeting = "Hi #{user.name} #{h["key"]}"\nwords = %w[alpha beta]\ntext = <<~SQL\n  SELECT * FROM t\nSQL\nmatch = value =~ /a+b/; ratio = a / b\nopts = { key: :value, "other" => 1 }\n=begin\nnote\n=end' }],
+      run: (fixture) => highlight(fixture.code, { lang: fixture.rawLang }),
+      observe: (result) => result.html.length,
+      targetBlocks: 10_000,
+    },
+    rust: {
+      fixtures: [{ rawLang: 'rust', code: '#[derive(Debug)]\nstruct Wrapper<\'a> { text: &\'a str }\n/* outer /* nested */ comment */\nlet raw = r#"quote "inside""#;\nlet bytes = b"data"; let c = c"text"; let ch = \'x\';\nprintln!("{}", vec![1u8, 0xff]);' }],
+      run: (fixture) => highlight(fixture.code, { lang: fixture.rawLang }),
+      observe: (result) => result.html.length,
+      targetBlocks: 10_000,
+    },
     tokenize: {
       fixtures,
       run: (fixture) => tokenize(fixture.code, { lang: fixture.rawLang }),

scripts/language-utils.mjs

@@ -5,23 +5,31 @@ export const supportedLanguages = [
   'apache',
   'cmake',
   'cpp',
+  'csharp',
   'css',
+  'dart',
   'diff',
   'dockerfile',
   'ejs',
   'env',
   'go',
   'html',
   'http',
+  'java',
   'js',
   'json',
   'jsx',
+  'kotlin',
+  'lua',
   'markdown',
   'mermaid',
   'nginx',
+  'perl',
   'php',
   'plaintext',
   'python',
+  'ruby',
+  'rust',
   'scheme',
   'shell',
   'sql',
@@ -40,6 +48,13 @@ const aliases = {
   cxx: 'cpp',
   hpp: 'cpp',
   hxx: 'cpp',
+  'c#': 'csharp',
+  cs: 'csharp',
+  kt: 'kotlin',
+  kts: 'kotlin',
+  pl: 'perl',
+  rb: 'ruby',
+  rs: 'rust',
   '-->': 'plaintext',
   'angular-html': 'html',
   'angular-ts': 'ts',