Class NativeAPI.Session

Namespace

Uralstech.UAI.LiteRTLM.Native

public static class NativeAPI.Session

Inheritance

object

NativeAPI.Session

Methods

litert_lm_session_cancel_process(nint)

Cancels the current processing in the session.

public static extern void litert_lm_session_cancel_process(nint session)

Parameters

session nint

The session to cancel processing on.

litert_lm_session_delete(nint)

Destroys a LiteRT LM Session.

public static extern void litert_lm_session_delete(nint session)

Parameters

session nint

The session to destroy.

litert_lm_session_generate_content(nint, nint[], nuint)

Generates content from the input prompt.

public static extern nint litert_lm_session_generate_content(nint session, nint[] inputs, nuint numInputs)

Parameters

session nint

The session to use for generation.

inputs nint[]

An array of InputData structs representing the multimodal input.

numInputs nuint

The number of InputData structs in the array.

Returns

nint

A pointer to the responses, or nint.Zero on failure. The caller is responsible for deleting the responses using litert_lm_responses_delete(nint).

litert_lm_session_generate_content_stream(nint, nint[], nuint, nint, nint)

Generates content from the input prompt and streams the response via a callback. This is a non-blocking call that will invoke the callback from a background thread for each chunk.

public static extern int litert_lm_session_generate_content_stream(nint session, nint[] inputs, nuint numInputs, nint callback, nint callbackData)

Parameters

session nint

The session to use for generation.

inputs nint[]

An array of InputData structs representing the multimodal input.

numInputs nuint

The number of InputData structs in the array.

callback nint

The callback function (NativeAPI.StreamCallback) to receive response chunks.

callbackData nint

A pointer to user data that will be passed to the callback.

Returns

int

0 on success, non-zero on failure to start the stream.

litert_lm_session_get_benchmark_info(nint)

Retrieves the benchmark information from the session. The caller is responsible for destroying the benchmark info using litert_lm_benchmark_info_delete(nint).

public static extern nint litert_lm_session_get_benchmark_info(nint session)

Parameters

session nint

The session to get the benchmark info from.

Returns

nint

A pointer to the benchmark info, or nint.Zero on failure.

litert_lm_session_rewind_to_checkpoint(nint, string)

Rewinds the session to the given checkpoint label.

public static extern int litert_lm_session_rewind_to_checkpoint(nint session, string label)

Parameters

session nint

The session to rewind.

label string

Label of the checkpoint to rewind to.

Returns

int

0 on success, non-zero on failure.

litert_lm_session_rewind_to_step(nint, int)

Rewinds the session to a specific step number.

public static extern int litert_lm_session_rewind_to_step(nint session, int step)

Parameters

session nint

The session to rewind.

step int

The step number to rewind to.

Returns

int

0 on success, non-zero on failure.

litert_lm_session_run_decode(nint)

Starts the decoding process for the model to predict the response based on the input prompt/query added after using litert_lm_session_run_prefill(nint, nint[], nuint). This is a blocking call and the function will return when the decoding process is done.

public static extern nint litert_lm_session_run_decode(nint session)

Parameters

session nint

The session to use.

Returns

nint

A pointer to the responses, or nint.Zero on failure. The caller is responsible for deleting the responses using litert_lm_responses_delete(nint).

litert_lm_session_run_decode_async(nint, nint, nint)

Starts the decoding process for the model to predict the response based on the input prompt/query added after using litert_lm_session_run_prefill(nint, nint[], nuint). This is a non-blocking call that will stream responses via a callback.

public static extern int litert_lm_session_run_decode_async(nint session, nint callback, nint callbackData)

Parameters

session nint

The session to use.

callback nint

The callback function (NativeAPI.StreamCallback) to receive response chunks.

callbackData nint

A pointer to user data that will be passed to the callback.

Returns

int

0 on success, non-zero on failure.

litert_lm_session_run_prefill(nint, nint[], nuint)

Adds the input prompt/query to the model for starting the prefilling process. This is a blocking call and the function will return when the prefill process is done.

public static extern int litert_lm_session_run_prefill(nint session, nint[] inputs, nuint numInputs)

Parameters

session nint

The session to use.

inputs nint[]

An array of InputData structs representing the multimodal input.

numInputs nuint

The number of InputData structs in the array.

Returns

int

0 on success, non-zero on failure.

litert_lm_session_run_text_scoring(nint, string[], nuint, bool)

Scores the target text after the prefill process is done.

public static extern nint litert_lm_session_run_text_scoring(nint session, string[] targetText, nuint numTargets, bool storeTokenLengths)

Parameters

session nint

The session to use.

targetText string[]

An array of target text strings to score.

numTargets nuint

The number of strings in the targetText array.

storeTokenLengths bool

Whether to store the token lengths of the target texts in the responses.

Returns

nint

A pointer to the responses, or nint.Zero on failure. The caller is responsible for deleting the responses using litert_lm_responses_delete(nint).

litert_lm_session_save_checkpoint(nint, string)

Saves the current state of the session to a checkpoint with the given label.

public static extern int litert_lm_session_save_checkpoint(nint session, string label)

Parameters

session nint

The session to save checkpoint for.

label string

Label for the checkpoint.

Returns

int

0 on success, non-zero on failure.