Class swarmauri_core.evaluator_pools.IEvaluatorPool.IEvaluatorPool

swarmauri_core.evaluator_pools.IEvaluatorPool.IEvaluatorPool

Bases: ABC

Interface for evaluator pools that manage sets of evaluators.

This abstract class defines the contract for evaluator pools, providing methods for dynamic registration, execution, and aggregation of evaluators. Pools are responsible for coordinating the evaluation of programs across multiple evaluators and aggregating their results.

initialize abstractmethod

initialize()

Initialize the evaluator pool and its resources.

This method should be called before using the pool to set up any necessary resources, connections, or state.

RAISES	DESCRIPTION
RuntimeError	If initialization fails

Source code in swarmauri_core/evaluator_pools/IEvaluatorPool.py

@abstractmethod
def initialize(self) -> None:
    """
    Initialize the evaluator pool and its resources.

    This method should be called before using the pool to set up any
    necessary resources, connections, or state.

    Raises:
        RuntimeError: If initialization fails
    """
    pass

shutdown abstractmethod

shutdown()

Shut down the evaluator pool and release its resources.

This method should be called when the pool is no longer needed to clean up resources and ensure proper termination.

RAISES	DESCRIPTION
RuntimeError	If shutdown fails

Source code in swarmauri_core/evaluator_pools/IEvaluatorPool.py

@abstractmethod
def shutdown(self) -> None:
    """
    Shut down the evaluator pool and release its resources.

    This method should be called when the pool is no longer needed to
    clean up resources and ensure proper termination.

    Raises:
        RuntimeError: If shutdown fails
    """
    pass

add_evaluator abstractmethod

add_evaluator(evaluator, name=None)

Add an evaluator to the pool.

PARAMETER	DESCRIPTION
evaluator	The evaluator to add to the pool TYPE: IEvaluate
name	Optional name for the evaluator, if not provided a name will be generated TYPE: Optional[str] DEFAULT: None

RETURNS	DESCRIPTION
str	The name assigned to the evaluator

RAISES	DESCRIPTION
ValueError	If an evaluator with the same name already exists
TypeError	If the evaluator doesn't implement IEvaluate

Source code in swarmauri_core/evaluator_pools/IEvaluatorPool.py

@abstractmethod
def add_evaluator(self, evaluator: IEvaluate, name: Optional[str] = None) -> str:
    """
    Add an evaluator to the pool.

    Args:
        evaluator: The evaluator to add to the pool
        name: Optional name for the evaluator, if not provided a name will be generated

    Returns:
        The name assigned to the evaluator

    Raises:
        ValueError: If an evaluator with the same name already exists
        TypeError: If the evaluator doesn't implement IEvaluate
    """
    pass

remove_evaluator abstractmethod

remove_evaluator(name)

Remove an evaluator from the pool by name.

PARAMETER	DESCRIPTION
name	The name of the evaluator to remove TYPE: str

RETURNS	DESCRIPTION
bool	True if the evaluator was removed, False if it wasn't found

Source code in swarmauri_core/evaluator_pools/IEvaluatorPool.py

@abstractmethod
def remove_evaluator(self, name: str) -> bool:
    """
    Remove an evaluator from the pool by name.

    Args:
        name: The name of the evaluator to remove

    Returns:
        True if the evaluator was removed, False if it wasn't found
    """
    pass

evaluate abstractmethod

evaluate(programs, **kwargs)

Evaluate all programs with all registered evaluators.

This method runs each program through each evaluator and collects the results.

PARAMETER	DESCRIPTION
programs	The programs to evaluate TYPE: Sequence[IProgram]
**kwargs	Additional parameters to pass to evaluators DEFAULT: {}

RETURNS	DESCRIPTION
Sequence[IEvalResult]	A sequence of evaluation results, one for each program

RAISES	DESCRIPTION
RuntimeError	If evaluation fails

Source code in swarmauri_core/evaluator_pools/IEvaluatorPool.py

@abstractmethod
def evaluate(self, programs: Sequence[IProgram], **kwargs) -> Sequence[IEvalResult]:
    """
    Evaluate all programs with all registered evaluators.

    This method runs each program through each evaluator and collects the results.

    Args:
        programs: The programs to evaluate
        **kwargs: Additional parameters to pass to evaluators

    Returns:
        A sequence of evaluation results, one for each program

    Raises:
        RuntimeError: If evaluation fails
    """
    pass

evaluate_async abstractmethod async

evaluate_async(programs, **kwargs)

Asynchronously evaluate all programs with all registered evaluators.

This method runs each program through each evaluator concurrently and collects the results.

PARAMETER	DESCRIPTION
programs	The programs to evaluate TYPE: Sequence[IProgram]
**kwargs	Additional parameters to pass to evaluators DEFAULT: {}

RETURNS	DESCRIPTION
Sequence[IEvalResult]	A sequence of evaluation results, one for each program

RAISES	DESCRIPTION
RuntimeError	If evaluation fails

Source code in swarmauri_core/evaluator_pools/IEvaluatorPool.py

@abstractmethod
async def evaluate_async(
    self, programs: Sequence[IProgram], **kwargs
) -> Sequence[IEvalResult]:
    """
    Asynchronously evaluate all programs with all registered evaluators.

    This method runs each program through each evaluator concurrently and collects the results.

    Args:
        programs: The programs to evaluate
        **kwargs: Additional parameters to pass to evaluators

    Returns:
        A sequence of evaluation results, one for each program

    Raises:
        RuntimeError: If evaluation fails
    """
    pass

aggregate abstractmethod

aggregate(scores)

Aggregate multiple scores into a single score.

This method combines multiple evaluation scores into a single scalar value according to the pool's aggregation strategy.

PARAMETER	DESCRIPTION
scores	The scores to aggregate TYPE: Sequence[float]

RETURNS	DESCRIPTION
float	The aggregated score

RAISES	DESCRIPTION
ValueError	If scores is empty or contains invalid values

Source code in swarmauri_core/evaluator_pools/IEvaluatorPool.py

@abstractmethod
def aggregate(self, scores: Sequence[float]) -> float:
    """
    Aggregate multiple scores into a single score.

    This method combines multiple evaluation scores into a single scalar value
    according to the pool's aggregation strategy.

    Args:
        scores: The scores to aggregate

    Returns:
        The aggregated score

    Raises:
        ValueError: If scores is empty or contains invalid values
    """
    pass

set_aggregation_function abstractmethod

set_aggregation_function(func)

Set the function used to aggregate scores.

PARAMETER	DESCRIPTION
func	A function that takes a sequence of scores and returns an aggregated score TYPE: Callable[[Sequence[float]], float]

RAISES	DESCRIPTION
TypeError	If func is not callable or has an invalid signature

Source code in swarmauri_core/evaluator_pools/IEvaluatorPool.py

@abstractmethod
def set_aggregation_function(
    self, func: Callable[[Sequence[float]], float]
) -> None:
    """
    Set the function used to aggregate scores.

    Args:
        func: A function that takes a sequence of scores and returns an aggregated score

    Raises:
        TypeError: If func is not callable or has an invalid signature
    """
    pass

get_evaluator abstractmethod

get_evaluator(name)

Get an evaluator by name.

PARAMETER	DESCRIPTION
name	The name of the evaluator to retrieve TYPE: str

RETURNS	DESCRIPTION
Optional[IEvaluate]	The evaluator if found, None otherwise

Source code in swarmauri_core/evaluator_pools/IEvaluatorPool.py

@abstractmethod
def get_evaluator(self, name: str) -> Optional[IEvaluate]:
    """
    Get an evaluator by name.

    Args:
        name: The name of the evaluator to retrieve

    Returns:
        The evaluator if found, None otherwise
    """
    pass

get_evaluator_names abstractmethod

get_evaluator_names()

Get the names of all registered evaluators.

RETURNS	DESCRIPTION
List[str]	A list of evaluator names

Source code in swarmauri_core/evaluator_pools/IEvaluatorPool.py

@abstractmethod
def get_evaluator_names(self) -> List[str]:
    """
    Get the names of all registered evaluators.

    Returns:
        A list of evaluator names
    """
    pass

get_evaluator_count abstractmethod

get_evaluator_count()

Get the number of registered evaluators.

RETURNS	DESCRIPTION
int	The count of evaluators in the pool

Source code in swarmauri_core/evaluator_pools/IEvaluatorPool.py

@abstractmethod
def get_evaluator_count(self) -> int:
    """
    Get the number of registered evaluators.

    Returns:
        The count of evaluators in the pool
    """
    pass