Class NativeAPI.Session

Namespace

Uralstech.UAI.LiteRTLM.Native

public static class NativeAPI.Session

Inheritance

object

NativeAPI.Session

Methods

litert_lm_session_cancel_process(nint)

Cancels the current processing in the session.

public static extern void litert_lm_session_cancel_process(nint session)

Parameters

session nint

The session to cancel processing on.

litert_lm_session_delete(nint)

Destroys a LiteRT LM Session.

public static extern void litert_lm_session_delete(nint session)

Parameters

session nint

The session to destroy.

litert_lm_session_generate_content(nint, nint[], nuint)

Generates content from the input prompt.

public static extern nint litert_lm_session_generate_content(nint session, nint[] inputs, nuint numInputs)

Parameters

session nint

The session to use for generation.

inputs nint[]

An array of InputData structs representing the multimodal input.

numInputs nuint

The number of InputData structs in the array.

Returns

nint

A pointer to the responses, or nint.Zero on failure. The caller is responsible for deleting the responses using litert_lm_responses_delete(nint).

litert_lm_session_generate_content_stream(nint, nint[], nuint, nint, nint)

Generates content from the input prompt and streams the response via a callback. This is a non-blocking call that will invoke the callback from a background thread for each chunk.

public static extern int litert_lm_session_generate_content_stream(nint session, nint[] inputs, nuint numInputs, nint callback, nint callbackData)

Parameters

session nint

The session to use for generation.

inputs nint[]

An array of InputData structs representing the multimodal input.

numInputs nuint

The number of InputData structs in the array.

callback nint

The callback function (NativeAPI.StreamCallback) to receive response chunks.

callbackData nint

A pointer to user data that will be passed to the callback.

Returns

int

0 on success, non-zero on failure to start the stream.

litert_lm_session_get_benchmark_info(nint)

Retrieves the benchmark information from the session. The caller is responsible for destroying the benchmark info using litert_lm_benchmark_info_delete(nint).

public static extern nint litert_lm_session_get_benchmark_info(nint session)

Parameters

session nint

The session to get the benchmark info from.

Returns

nint

A pointer to the benchmark info, or nint.Zero on failure.

litert_lm_session_run_decode(nint)

Starts the decoding process for the model to predict the response based on the input prompt/query added after using litert_lm_session_run_prefill(nint, nint[], nuint). This is a blocking call and the function will return when the decoding process is done.

public static extern nint litert_lm_session_run_decode(nint session)

Parameters

session nint

The session to use.

Returns

nint

A pointer to the responses, or nint.Zero on failure. The caller is responsible for deleting the responses using litert_lm_responses_delete(nint).

litert_lm_session_run_decode_async(nint, nint, nint)

Starts the decoding process for the model to predict the response based on the input prompt/query added after using litert_lm_session_run_prefill(nint, nint[], nuint). This is a non-blocking call that will stream responses via a callback.

public static extern int litert_lm_session_run_decode_async(nint session, nint callback, nint callbackData)

Parameters

session nint

The session to use.

callback nint

The callback function (NativeAPI.StreamCallback) to receive response chunks.

callbackData nint

A pointer to user data that will be passed to the callback.

Returns

int

0 on success, non-zero on failure.

litert_lm_session_run_prefill(nint, nint[], nuint)

Adds the input prompt/query to the model for starting the prefilling process. This is a blocking call and the function will return when the prefill process is done.

public static extern int litert_lm_session_run_prefill(nint session, nint[] inputs, nuint numInputs)

Parameters

session nint

The session to use.

inputs nint[]

An array of InputData structs representing the multimodal input.

numInputs nuint

The number of InputData structs in the array.

Returns

int

0 on success, non-zero on failure.

litert_lm_session_run_text_scoring(nint, string[], nuint, bool)

Scores the target text after the prefill process is done.

public static extern nint litert_lm_session_run_text_scoring(nint session, string[] targetText, nuint numTargets, bool storeTokenLengths)

Parameters

session nint

The session to use.

targetText string[]

An array of target text strings to score.

numTargets nuint

The number of strings in the targetText array.

storeTokenLengths bool

Whether to store the token lengths of the target texts in the responses.

Returns

nint

A pointer to the responses, or nint.Zero on failure. The caller is responsible for deleting the responses using litert_lm_responses_delete(nint).