effect-io-ai

Package: @effect/ai
Module: Tokenizer

Tokenizer.Tokenizer

The Tokenizer service tag for dependency injection.

This tag provides access to tokenization functionality throughout your application, enabling token counting and prompt truncation capabilities.

Example

import { Tokenizer } from "@effect/ai"
import { Effect } from "effect"

const useTokenizer = Effect.gen(function* () {
  const tokenizer = yield* Tokenizer.Tokenizer
  const tokens = yield* tokenizer.tokenize("Hello, world!")
  return tokens.length
})

Signature

declare class Tokenizer

Source

Since v1.0.0