Class BenchmarkInfo

Namespace

Uralstech.UAI.LiteRTLM

public sealed class BenchmarkInfo : LiteRTLMNativeHandle

Inheritance

object

LiteRTLMNativeHandle

BenchmarkInfo

Inherited Members

LiteRTLMNativeHandle.Dispose()

Methods

GetDecodeTurns()

Returns the benchmark information for each decode turn.

public IReadOnlyList<BenchmarkInfo.Turn> GetDecodeTurns()

Returns

IReadOnlyList<BenchmarkInfo.Turn>

GetPrefillTurns()

Returns the benchmark information for each prefill turn.

public IReadOnlyList<BenchmarkInfo.Turn> GetPrefillTurns()

Returns

IReadOnlyList<BenchmarkInfo.Turn>

GetTimeToFirstToken()

Returns the time to the first token in seconds.

public double GetTimeToFirstToken()

Returns

double

The time to the first token in seconds.

Remarks

Note that the time to first token does not include initialization time. It is the sum of the prefill time for the first turn and the time spent decoding the first token.

GetTotalInitTime()

Returns the total initialization time in seconds.

public double GetTotalInitTime()

Returns

double

The total initialization time in seconds.

ReleaseUnmanagedResources()

protected override void ReleaseUnmanagedResources()