agentlab.agents.dynamic_prompting

Functions

`fit_tokens`(shrinkable[, max_prompt_tokens, ...])	Shrink a prompt element until it fits max_prompt_tokens.
`make_obs_preprocessor`(flags)

Classes

`AXTree`(ax_tree, visible_elements_only[, ...])
`ActionFlags`([action_set, long_description, ...])
`ActionPrompt`(action_set, action_flags)
`BeCautious`([visible])
`ChatInstructions`(chat_messages[, visible, ...])
`Error`(error[, visible, prefix, limit_logs])
`Flags`()	Base class for flags.
`FocusedElement`(bid[, visible, prefix])
`GoalInstructions`(goal_object[, visible, ...])
`HTML`(html, visible_elements_only[, visible, ...])
`Hints`([visible])	Not super useful and stale.
`History`(history_obs, actions, memories, ...)
`HistoryStep`(previous_obs, current_obs, ...)
`MacNote`()
`ObsFlags`([use_html, use_ax_tree, use_tabs, ...])	A class to represent various flags used to control features in an application.
`Observation`(obs, flags)	Observation of the current step.
`PromptElement`([visible])	Base class for all prompt elements.
`Shrinkable`([visible])
`SystemPrompt`([visible])
`Tabs`(obs[, visible, prefix])
`Think`([visible])
`Trunkater`(visible[, shrink_speed, ...])	Shrinkable element that truncates the prompt element from the bottom after a certain number of iterations.

class agentlab.agents.dynamic_prompting.AXTree(ax_tree, visible_elements_only: bool, visible: bool = True, coord_type=None, visible_tag=True, prefix=''): Bases: Trunkater

class agentlab.agents.dynamic_prompting.ActionFlags(action_set: browsergym.experiments.benchmark.base.HighLevelActionSetArgs = None, long_description: bool = True, individual_examples: bool = False, multi_actions: bool = None, is_strict: bool = None)

Bases: Flags

action_set: HighLevelActionSetArgs = None

individual_examples: bool = False

is_strict: bool = None

long_description: bool = True

multi_actions: bool = None

class agentlab.agents.dynamic_prompting.ActionPrompt(action_set: AbstractActionSet, action_flags: ActionFlags): Bases: PromptElement

class agentlab.agents.dynamic_prompting.BeCautious(visible: bool = True): Bases: PromptElement

class agentlab.agents.dynamic_prompting.ChatInstructions(chat_messages, visible: bool = True, extra_instructions=None): Bases: PromptElement

class agentlab.agents.dynamic_prompting.Error(error: str, visible: bool = True, prefix='', limit_logs=True): Bases: PromptElement

class agentlab.agents.dynamic_prompting.Flags

Bases: object

Base class for flags. Mostly for backward compatibility.

asdict(): Helper for JSON serializable requirement.

copy()

classmethod from_dict(flags_dict): Helper for JSON serializable requirement.

class agentlab.agents.dynamic_prompting.FocusedElement(bid, visible: bool = True, prefix=''): Bases: PromptElement

class agentlab.agents.dynamic_prompting.GoalInstructions(goal_object, visible: bool = True, extra_instructions=None): Bases: PromptElement

class agentlab.agents.dynamic_prompting.HTML(html, visible_elements_only: bool, visible: bool = True, prefix=''): Bases: Trunkater

class agentlab.agents.dynamic_prompting.Hints(visible: bool = True)

Bases: PromptElement

Not super useful and stale.

class agentlab.agents.dynamic_prompting.History(history_obs, actions, memories, thoughts, flags: ObsFlags, shrink_speed=1)

Bases: Shrinkable

shrink(): Shrink individual steps

class agentlab.agents.dynamic_prompting.HistoryStep(previous_obs, current_obs, action, memory, thought, flags: ObsFlags, shrink_speed=1)

Bases: Shrinkable

shrink()

Implement shrinking of this prompt element.

You need to recursively call all shrinkable elements that are part of this prompt. You can also implement a shriking startegy for this prompt. Shrinking is can be called multiple times to progressively shrink the prompt until it fits max_tokens. Default max shrink iterations is 20.

class agentlab.agents.dynamic_prompting.MacNote: Bases: PromptElement

class agentlab.agents.dynamic_prompting.ObsFlags(use_html: bool = True, use_ax_tree: bool = False, use_tabs: bool = False, use_focused_element: bool = False, use_error_logs: bool = False, use_history: bool = False, use_past_error_logs: bool = False, use_action_history: bool = False, use_think_history: bool = False, use_diff: bool = False, html_type: str = 'pruned_html', use_screenshot: bool = True, use_som: bool = False, extract_visible_tag: bool = False, extract_clickable_tag: bool = False, extract_coords: Literal['False', 'center', 'box'] = 'False', filter_visible_elements_only: bool = False, openai_vision_detail: Literal['low', 'high', 'auto'] = 'auto', filter_with_bid_only: bool = False, filter_som_only: bool = False)

Bases: Flags

A class to represent various flags used to control features in an application.

use_html

Use the HTML in the prompt.

Type:: bool

use_ax_tree

Use the accessibility tree in the prompt.

Type:: bool

use_focused_element

Provide the ID of the focused element.

Type:: bool

use_error_logs

Expose the previous error in the prompt.

Type:: bool

use_history

Enable history of previous steps in the prompt.

Type:: bool

use_past_error_logs

If use_history is True, expose all previous errors in the history.

Type:: bool

use_action_history

If use_history is True, include the actions in the history.

Type:: bool

use_think_history

If use_history is True, include all previous chains of thoughts in the history.

Type:: bool

use_diff

Add a diff of the current and previous HTML to the prompt.

Type:: bool

html_type

Type of HTML to use in the prompt, may depend on preprocessing of observation.

Type:: str

use_screenshot

Add a screenshot of the page to the prompt, following OpenAI’s API. This will be automatically disabled if the model does not have vision capabilities.

Type:: bool

use_som

Add a set of marks to the screenshot.

Type:: bool

extract_visible_tag

Add a “visible” tag to visible elements in the AXTree.

Type:: bool

extract_clickable_tag

Add a “clickable” tag to clickable elements in the AXTree.

Type:: bool

extract_coords

Add the coordinates of the elements.

Type:: Literal[‘False’, ‘center’, ‘box’]

filter_visible_elements_only

Only show visible elements in the AXTree.

Type:: bool

extract_clickable_tag: bool = False

extract_coords: Literal['False', 'center', 'box'] = 'False'

extract_visible_tag: bool = False

filter_som_only: bool = False

filter_visible_elements_only: bool = False

filter_with_bid_only: bool = False

html_type: str = 'pruned_html'

openai_vision_detail: Literal['low', 'high', 'auto'] = 'auto'

use_action_history: bool = False

use_ax_tree: bool = False

use_diff: bool = False

use_error_logs: bool = False

use_focused_element: bool = False

use_history: bool = False

use_html: bool = True

use_past_error_logs: bool = False

use_screenshot: bool = True

use_som: bool = False

use_tabs: bool = False

use_think_history: bool = False

class agentlab.agents.dynamic_prompting.Observation(obs, flags: ObsFlags)

Bases: Shrinkable

Observation of the current step.

Contains the html, the accessibility tree and the error logs.

add_screenshot(prompt: BaseMessage) → BaseMessage

shrink()

Implement shrinking of this prompt element.

You need to recursively call all shrinkable elements that are part of this prompt. You can also implement a shriking startegy for this prompt. Shrinking is can be called multiple times to progressively shrink the prompt until it fits max_tokens. Default max shrink iterations is 20.

class agentlab.agents.dynamic_prompting.PromptElement(visible: bool = True)

Bases: object

Base class for all prompt elements. Prompt elements can be hidden.

property abstract_ex

Useful when this prompt element is requesting an answer from the llm. Provide an abstract example of the answer here. See Memory for an example.

Avoid overriding this method. Override _abstract_ex instead

Returns:: The abstract example
Return type:: str

property concrete_ex

Useful when this prompt element is requesting an answer from the llm. Provide a concrete example of the answer here. See Memory for an example.

Avoid overriding this method. Override _concrete_ex instead

Returns:: The concrete example
Return type:: str

property is_visible: Handle the case where visible is a callable.

parse_answer(text_answer) → dict

property prompt: str | BaseMessage: Avoid overriding this method. Override _prompt instead.

class agentlab.agents.dynamic_prompting.Shrinkable(visible: bool = True)

Bases: PromptElement, ABC

abstract shrink() → None

Implement shrinking of this prompt element.

You need to recursively call all shrinkable elements that are part of this prompt. You can also implement a shriking startegy for this prompt. Shrinking is can be called multiple times to progressively shrink the prompt until it fits max_tokens. Default max shrink iterations is 20.

class agentlab.agents.dynamic_prompting.SystemPrompt(visible: bool = True): Bases: PromptElement

class agentlab.agents.dynamic_prompting.Tabs(obs, visible: bool = True, prefix=''): Bases: PromptElement

class agentlab.agents.dynamic_prompting.Think(visible: bool = True): Bases: PromptElement

class agentlab.agents.dynamic_prompting.Trunkater(visible, shrink_speed=0.3, start_trunkate_iteration=10)

Bases: Shrinkable

Shrinkable element that truncates the prompt element from the bottom after a certain number of iterations.

shrink() → None

Implement shrinking of this prompt element.

You need to recursively call all shrinkable elements that are part of this prompt. You can also implement a shriking startegy for this prompt. Shrinking is can be called multiple times to progressively shrink the prompt until it fits max_tokens. Default max shrink iterations is 20.

agentlab.agents.dynamic_prompting.fit_tokens(shrinkable: Shrinkable, max_prompt_tokens=None, max_iterations=20, model_name='openai/gpt-4', additional_prompts=[''])

Shrink a prompt element until it fits max_prompt_tokens.

Parameters:

shrinkable (Shrinkable) – The prompt element to shrink.
max_prompt_tokens (int) – The maximum number of tokens allowed.
max_iterations (int, optional) – The maximum number of shrink iterations, by default 20.
model_name (str, optional) – The name of the model used when tokenizing.
additional_prompts (str or List[str], optional) – Additional prompts to account for when shrinking, by default [“”].

Returns:

the prompt after shrinking.

Return type:

str

Raises:

ValueError – Unrecognized type for prompt

agentlab.agents.dynamic_prompting.make_obs_preprocessor(flags: ObsFlags)