#API Reference

In the documentation below, you might see function calls prefixed with agent.. If you utilize destructuring in Playwright (e.g., async ({ ai, aiQuery }) => { /* ... */ }), you can call these functions without the agent. prefix. This is merely a syntactical difference.

#Constructors

Each Agent in Midscene has its own constructor.

• In Puppeteer, use PuppeteerAgent
• In Bridge Mode, use AgentOverChromeBridge
• In Android, use AndroidAgent

These Agents share some common constructor parameters:

• generateReport: boolean: If true, a report file will be generated. (Default: true)
• reportFileName: string: The name of the report file. (Default: generated by midscene)
• autoPrintReportMsg: boolean: If true, report messages will be printed. (Default: true)
• cacheId: string | undefined: If provided, this cacheId will be used to save or match the cache. (Default: undefined, means cache feature is disabled)
• actionContext: string: Some background knowledge that should be sent to the AI model when calling agent.aiAction(), like 'close the cookie consent dialog first if it exists' (Default: undefined)

In Playwright and Puppeteer, there are some common parameters:

• forceSameTabNavigation: boolean: If true, page navigation is restricted to the current tab. (Default: true)
• waitForNetworkIdleTimeout: number: The timeout for waiting for network idle between each action. (Default: 2000ms, set to 0 means not waiting for network idle)
• waitForNavigationTimeout: number: The timeout for waiting for navigation finished. (Default: 5000ms, set to 0 means not waiting for navigation finished)

#Interaction Methods

Below are the main APIs available for the various Agents in Midscene.

#agent.aiAction() or .ai()

This method allows you to perform a series of UI actions described in natural language. Midscene automatically plans the steps and executes them.

• Type

function aiAction(
  prompt: string,
  options?: {
    cacheable?: boolean;
  },
): Promise<void>;
function ai(prompt: string): Promise<void>; // shorthand form

• Parameters:

  • prompt: string - A natural language description of the UI steps.
  • options?: Object - Optional, a configuration object containing:
    • cacheable?: boolean - Whether cacheable when enabling caching feature. True by default.

• Return Value:

  • Returns a Promise that resolves to void when all steps are completed; if execution fails, an error is thrown.

• Examples:

// Basic usage
await agent.aiAction(
  'Type "JavaScript" into the search box, then click the search button',
);

// Using the shorthand .ai form
await agent.ai(
  'Click the login button at the top of the page, then enter "test@example.com" in the username field',
);

// When using UI Agent models like ui-tars, you can try a more goal-driven prompt
await agent.aiAction('Post a Tweet "Hello World"');

#agent.aiTap()

Tap something.

• Type

function aiTap(locate: string | Object, options?: Object): Promise<void>;

• Parameters:

  • locate: string | Object - A natural language description of the element to tap, or prompting with images.
  • options?: Object - Optional, a configuration object containing:
    • deepThink?: boolean - If true, Midscene will call AI model twice to precisely locate the element. False by default.
    • xpath?: string - The xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
    • cacheable?: boolean - Whether cacheable when enabling caching feature. True by default.

• Return Value:

  • Returns a Promise<void>

• Examples:

await agent.aiTap('The login button at the top of the page');

// Use deepThink feature to precisely locate the element
await agent.aiTap('The login button at the top of the page', {
  deepThink: true,
});

#agent.aiHover()

Only available in web pages, not available in Android.

Move mouse over something.

• Type

function aiHover(locate: string | Object, options?: Object): Promise<void>;

• Parameters:

  • locate: string | Object - A natural language description of the element to hover over, or prompting with images.
  • options?: Object - Optional, a configuration object containing:
    • deepThink?: boolean - If true, Midscene will call AI model twice to precisely locate the element. False by default.
    • xpath?: string - The xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
    • cacheable?: boolean - Whether cacheable when enabling caching feature. True by default.

• Return Value:

  • Returns a Promise<void>

• Examples:

await agent.aiHover('The version number of the current page');

#agent.aiInput()

Input text into something.

• Type

function aiInput(
  text: string | Object,
  locate: string,
  options?: Object,
): Promise<void>;

• Parameters:

  • text: string - The final text content that should be placed in the input element. Use blank string to clear the input.
  • locate: string | Object - A natural language description of the element to input text into, or prompting with images.
  • options?: Object - Optional, a configuration object containing:
    • deepThink?: boolean - If true, Midscene will call AI model twice to precisely locate the element. False by default.
    • xpath?: string - The xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
    • cacheable?: boolean - Whether cacheable when enabling caching feature. True by default.
    • autoDismissKeyboard?: boolean - If true, the keyboard will be dismissed after input text, only available in Android. (Default: true)

• Return Value:

  • Returns a Promise<void>

• Examples:

await agent.aiInput('Hello World', 'The search input box');

#agent.aiKeyboardPress()

Press a keyboard key.

• Type

function aiKeyboardPress(
  key: string,
  locate?: string | Object,
  options?: Object,
): Promise<void>;

• Parameters:

  • key: string - The web key to press, e.g. 'Enter', 'Tab', 'Escape', etc. Key Combination is not supported.
  • locate?: string | Object - Optional, a natural language description of the element to press the key on, or prompting with images.
  • options?: Object - Optional, a configuration object containing:
    • deepThink?: boolean - If true, Midscene will call AI model twice to precisely locate the element. False by default.
    • xpath?: string - The xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
    • cacheable?: boolean - Whether cacheable when enabling caching feature. True by default.

• Return Value:

  • Returns a Promise<void>

• Examples:

await agent.aiKeyboardPress('Enter', 'The search input box');

#agent.aiScroll()

Scroll a page or an element.

• Type

function aiScroll(
  scrollParam: PlanningActionParamScroll,
  locate?: string | Object,
  options?: Object,
): Promise<void>;

• Parameters:

  • scrollParam: PlanningActionParamScroll - The scroll parameter
    • direction: 'up' | 'down' | 'left' | 'right' - The direction to scroll. Whether it is Android or Web, the scrolling direction here all refers to which direction of the page's content will appear on the screen. For example, when the scrolling direction is down, the hidden content at the bottom of the page will gradually reveal itself from the bottom of the screen upwards.
    • scrollType: 'once' | 'untilBottom' | 'untilTop' | 'untilRight' | 'untilLeft' - Optional, the type of scroll to perform.
    • distance: number - Optional, the distance to scroll in px.
  • locate?: string | Object - Optional, a natural language description of the element to scroll on, or prompting with images. If not provided, Midscene will perform scroll on the current mouse position.
  • options?: Object - Optional, a configuration object containing:
    • deepThink?: boolean - If true, Midscene will call AI model twice to precisely locate the element. False by default.
    • xpath?: string - The xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
    • cacheable?: boolean - Whether cacheable when enabling caching feature. True by default.

• Return Value:

  • Returns a Promise<void>

• Examples:

await agent.aiScroll(
  { direction: 'up', distance: 100, scrollType: 'once' },
  'The form panel',
);

#agent.aiRightClick()

Only available in web pages, not available in Android.

Right-click on an element. Please note that Midscene cannot interact with the native context menu in browser after right-clicking. This interface is usually used for the element that listens to the right-click event by itself.

• Type

function aiRightClick(locate: string, options?: Object): Promise<void>;

• Parameters:

  • locate: string | Object - A natural language description of the element to right-click on, or prompting with images.
  • options?: Object - Optional, a configuration object containing:
    • deepThink?: boolean - If true, Midscene will call AI model twice to precisely locate the element. False by default.
    • xpath?: string - The xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
    • cacheable?: boolean - Whether cacheable when enabling caching feature. True by default.

• Return Value:

  • Returns a Promise<void>

• Examples:

await agent.aiRightClick('The file name at the top of the page');

// Use deepThink feature to precisely locate the element
await agent.aiRightClick('The file name at the top of the page', {
  deepThink: true,
});

#Data Extraction

#agent.aiAsk()

Ask the AI model any question about the current page. It returns the answer in string from the AI model.

• Type

function aiAsk(prompt: string | Object, options?: Object): Promise<string>;

• Parameters:

  • prompt: string | Object - A natural language description of the question, or prompting with images.
  • options?: Object - Optional, a configuration object containing:
    • domIncluded?: boolean | 'visible-only' - Whether to send simplified DOM information to the model, usually used for extracting invisible attributes like image links. If set to 'visible-only', only the visible elements will be sent. Default: False.
    • screenshotIncluded?: boolean - Whether to send screenshot to the model. Default: True.

• Return Value:

  • Return a Promise. Return the answer from the AI model.

• Examples:

const result = await agent.aiAsk('What should I do to test this page?');
console.log(result); // Output the answer from the AI model

Besides aiAsk, you can also use aiQuery to extract structured data from the UI.

#agent.aiQuery()

This method allows you to extract structured data from current page. Simply define the expected format (e.g., string, number, JSON, or an array) in the dataDemand, and Midscene will return a result that matches the format.

• Type

function aiQuery<T>(dataDemand: string | Object, options?: Object): Promise<T>;

• Parameters:

  • dataDemand: T: A description of the expected data and its return format.
  • options?: Object - Optional, a configuration object containing:
    • domIncluded?: boolean | 'visible-only' - Whether to send simplified DOM information to the model, usually used for extracting invisible attributes like image links. If set to 'visible-only', only the visible elements will be sent. Default: False.
    • screenshotIncluded?: boolean - Whether to send screenshot to the model. Default: True.

• Return Value:

  • Returns any valid basic type, such as string, number, JSON, array, etc.
  • Just describe the format in dataDemand, and Midscene will return a matching result.

• Examples:

const dataA = await agent.aiQuery({
  time: 'The date and time displayed in the top-left corner as a string',
  userInfo: 'User information in the format {name: string}',
  tableFields: 'An array of table field names, string[]',
  tableDataRecord:
    'Table records in the format {id: string, [fieldName]: string}[]',
});

// You can also describe the expected return format using a string:

// dataB will be an array of strings
const dataB = await agent.aiQuery('string[], list of task names');

// dataC will be an array of objects
const dataC = await agent.aiQuery(
  '{name: string, age: string}[], table data records',
);

// Use domIncluded feature to extract invisible attributes
const dataD = await agent.aiQuery(
  '{name: string, age: string, avatarUrl: string}[], table data records',
  { domIncluded: true },
);

#agent.aiBoolean()

Extract a boolean value from the UI.

• Type

function aiBoolean(prompt: string | Object, options?: Object): Promise<boolean>;

• Parameters:

  • prompt: string | Object - A natural language description of the expected value, or prompting with images.
  • options?: Object - Optional, a configuration object containing:
    • domIncluded?: boolean | 'visible-only' - Whether to send simplified DOM information to the model, usually used for extracting invisible attributes like image links. If set to 'visible-only', only the visible elements will be sent. Default: False.
    • screenshotIncluded?: boolean - Whether to send screenshot to the model. Default: True.

• Return Value:

  • Returns a Promise<boolean> when AI returns a boolean value.

• Examples:

const boolA = await agent.aiBoolean('Whether there is a login dialog');

// Use domIncluded feature to extract invisible attributes
const boolB = await agent.aiBoolean('Whether the login button has a link', {
  domIncluded: true,
});

#agent.aiNumber()

Extract a number value from the UI.

• Type

function aiNumber(prompt: string | Object, options?: Object): Promise<number>;

• Parameters:

  • prompt: string | Object - A natural language description of the expected value, or prompting with images.
  • options?: Object - Optional, a configuration object containing:
    • domIncluded?: boolean | 'visible-only' - Whether to send simplified DOM information to the model, usually used for extracting invisible attributes like image links. If set to 'visible-only', only the visible elements will be sent. Default: False.
    • screenshotIncluded?: boolean - Whether to send screenshot to the model. Default: True.

• Return Value:

  • Returns a Promise<number> when AI returns a number value.

• Examples:

const numberA = await agent.aiNumber('The remaining points of the account');

// Use domIncluded feature to extract invisible attributes
const numberB = await agent.aiNumber(
  'The value of the remaining points element',
  { domIncluded: true },
);

#agent.aiString()

Extract a string value from the UI.

• Type

function aiString(prompt: string | Object, options?: Object): Promise<string>;

• Parameters:

  • prompt: string | Object - A natural language description of the expected value, or prompting with images.
  • options?: Object - Optional, a configuration object containing:
    • domIncluded?: boolean | 'visible-only' - Whether to send simplified DOM information to the model, usually used for extracting invisible attributes like image links. If set to 'visible-only', only the visible elements will be sent. Default: False.
    • screenshotIncluded?: boolean - Whether to send screenshot to the model. Default: True.

• Return Value:

  • Returns a Promise<string> when AI returns a string value.

• Examples:

const stringA = await agent.aiString('The first item in the list');

// Use domIncluded feature to extract invisible attributes
const stringB = await agent.aiString('The link of the first item in the list', {
  domIncluded: true,
});

#More APIs

#agent.aiAssert()

Specify an assertion in natural language, and the AI determines whether the condition is true. If the assertion fails, the SDK throws an error that includes both the optional errorMsg and a detailed reason generated by the AI.

• Type

function aiAssert(assertion: string | Object, errorMsg?: string): Promise<void>;

• Parameters:

  • assertion: string | Object - The assertion described in natural language, or prompting with images.
  • errorMsg?: string - An optional error message to append if the assertion fails.

• Return Value:

  • Returns a Promise that resolves to void if the assertion passes; if it fails, an error is thrown with errorMsg and additional AI-provided information.

• Example:

await agent.aiAssert('The price of "Sauce Labs Onesie" is 7.99');

const items = await agent.aiQuery(
  '"{name: string, price: number}[], return product names and prices',
);
const onesieItem = items.find((item) => item.name === 'Sauce Labs Onesie');
expect(onesieItem).toBeTruthy();
expect(onesieItem.price).toBe(7.99);

#agent.aiLocate()

Locate an element using natural language.

• Type

function aiLocate(
  locate: string | Object,
  options?: Object,
): Promise<{
  rect: {
    left: number;
    top: number;
    width: number;
    height: number;
  };
  center: [number, number];
  scale: number; // device pixel ratio
}>;

• Parameters:

  • locate: string | Object - A natural language description of the element to locate, or prompting with images.
  • options?: Object - Optional, a configuration object containing:
    • deepThink?: boolean - If true, Midscene will call AI model twice to precisely locate the element. False by default.
    • xpath?: string - The xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
    • cacheable?: boolean - Whether cacheable when enabling caching feature. True by default.

• Return Value:

  • Returns a Promise when the element is located parsed as an locate info object.

• Examples:

const locateInfo = await agent.aiLocate(
  'The login button at the top of the page',
);
console.log(locateInfo);

#agent.aiWaitFor()

Wait until a specified condition, described in natural language, becomes true. Considering the cost of AI calls, the check interval will not exceed the specified checkIntervalMs.

• Type

function aiWaitFor(
  assertion: string,
  options?: {
    timeoutMs?: number;
    checkIntervalMs?: number;
  },
): Promise<void>;

• Parameters:

  • assertion: string - The condition described in natural language.
  • options?: object - An optional configuration object containing:
    • timeoutMs?: number - Timeout in milliseconds (default: 15000).
    • checkIntervalMs?: number - Interval for checking in milliseconds (default: 3000).

• Return Value:

  • Returns a Promise that resolves to void if the condition is met; if not, an error is thrown when the timeout is reached.

• Examples:

// Basic usage
await agent.aiWaitFor(
  'There is at least one headphone information displayed on the interface',
);

// Using custom options
await agent.aiWaitFor('The shopping cart icon shows a quantity of 2', {
  timeoutMs: 30000, // Wait for 30 seconds
  checkIntervalMs: 5000, // Check every 5 seconds
});

#agent.runYaml()

Execute an automation script written in YAML. Only the tasks part of the script is executed, and it returns the results of all .aiQuery calls within the script.

• Type

function runYaml(yamlScriptContent: string): Promise<{ result: any }>;

• Parameters:

  • yamlScriptContent: string - The YAML-formatted script content.

• Return Value:

  • Returns an object with a result property that includes the results of all .aiQuery calls.

• Example:

const { result } = await agent.runYaml(`
tasks:
  - name: search weather
    flow:
      - ai: input 'weather today' in input box, click search button
      - sleep: 3000

  - name: query weather
    flow:
      - aiQuery: "the result shows the weather info, {description: string}"
`);
console.log(result);

#agent.setAIActionContext()

Set the background knowledge that should be sent to the AI model when calling agent.aiAction().

• Type

function setAIActionContext(actionContext: string): void;

• Parameters:

  • actionContext: string - The background knowledge that should be sent to the AI model.

• Example:

await agent.setAIActionContext(
  'Close the cookie consent dialog first if it exists',
);

#agent.evaluateJavaScript()

Only available in web pages, not available in Android.

Evaluate a JavaScript expression in the web page context.

• Type

function evaluateJavaScript(script: string): Promise<any>;

• Parameters:

  • script: string - The JavaScript expression to evaluate.

• Return Value:

  • Returns the result of the JavaScript expression.

• Example:

const result = await agent.evaluateJavaScript('document.title');
console.log(result);

#agent.logScreenshot()

Log the current screenshot with a description in the report file.

• Type

function logScreenshot(title?: string, options?: Object): Promise<void>;

• Parameters:

  • title?: string - Optional, the title of the screenshot, if not provided, the title will be 'untitled'.
  • options?: Object - Optional, a configuration object containing:
    • content?: string - The description of the screenshot.

• Return Value:

  • Returns a Promise<void>

• Examples:

await agent.logScreenshot('Login page', {
  content: 'User A',
});

#agent.freezePageContext()

Freeze the current page context, allowing all subsequent operations to reuse the same page snapshot without retrieving the page state repeatedly. This significantly improves performance when executing a large number of concurrent operations.

Some notes:

• Usually, you do not need to use this method, unless you are certain that "context retrieval" is the bottleneck of your test script.
• You need to call agent.unfreezePageContext() in time to restore the real-time page state.
• Do not call this method in interaction operations, it will make the AI model unable to perceive the latest page state, causing confusing errors.

• Type

function freezePageContext(): Promise<void>;

• Return Value:

  • Promise<void>

• Examples:

// Freeze the page context
await agent.freezePageContext();

// Some queries...
const results = await Promise.all([
  await agent.aiQuery('Username input box value'),
  await agent.aiQuery('Password input box value'),
  await agent.aiLocate('Login button'),
]);
console.log(results);

// Unfreeze the page context, subsequent operations will use real-time page state
await agent.unfreezePageContext();

#agent.unfreezePageContext()

Unfreezes the page context, restoring the use of real-time page state.

• Type

function unfreezePageContext(): Promise<void>;

• Return Value:

  • Promise<void>

#Properties

#.reportFile

The path to the report file.

#Additional Configurations

#Setting Environment Variables at Runtime

You can override environment variables at runtime by calling the overrideAIConfig method.

import { overrideAIConfig } from '@midscene/web/puppeteer'; // or another Agent

overrideAIConfig({
  OPENAI_BASE_URL: '...',
  OPENAI_API_KEY: '...',
  MIDSCENE_MODEL_NAME: '...',
});

#Print usage information for each AI call

Set the DEBUG=midscene:ai:profile:stats to view the execution time and usage for each AI call.

export DEBUG=midscene:ai:profile:stats

#Customize the run artifact directory

Set the MIDSCENE_RUN_DIR variable to customize the run artifact directory.

export MIDSCENE_RUN_DIR=midscene_run # The default value is the midscene_run in the current working directory, you can set it to an absolute path or a relative path

#Customize the replanning cycle limit

Set the MIDSCENE_REPLANNING_CYCLE_LIMIT variable to customize the maximum number of replanning cycles allowed during action execution (aiAction).

export MIDSCENE_REPLANNING_CYCLE_LIMIT=10 # The default value is 10. When the AI needs to replan more than this limit, an error will be thrown suggesting to split the task into multiple steps

#Using LangSmith

LangSmith is a platform for debugging large language models. To integrate LangSmith, follow these steps:

# Set environment variables

# Enable debug mode
export MIDSCENE_LANGSMITH_DEBUG=1

# LangSmith configuration
export LANGSMITH_TRACING_V2=true
export LANGSMITH_ENDPOINT="https://api.smith.langchain.com"
export LANGSMITH_API_KEY="your_key_here"
export LANGSMITH_PROJECT="your_project_name_here"

After starting Midscene, you should see logs similar to:

DEBUGGING MODE: langsmith wrapper enabled

#Advanced features

#Prompting with images

You can use images as supplements in the prompt to describe things that cannot be expressed in natural language.

When prompting with images, the format of the prompt parameters is as follows:

{
  // Prompt text, in which images can be referred
  prompt: string,
  // The images referred in the prompt text
  images?: {
    // Image name, corresponding to the names referred in the prompt text
    name: string,
    // Image url, can be a local image path, Base64 string, or http link
    url: string
  }[]
  // When convertHttpImage2Base64 is true，the image links in the http format will be converted into Base64 encoding and sent to the LLM.
  // Which is applicable when the image links are not publicly accessible.
  convertHttpImage2Base64?: boolean
}

• Example 1: use images to inspect the tap position.

await agent.aiTap({
  prompt: 'The specific logo',
  images: [
    {
      name: 'The specific logo',
      url: 'https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png',
    },
  ],
});

• Example 2: use images to assert the page content.

await agent.aiAssert({
  prompt: 'Whether there is a specific logo on the page.',
  images: [
    {
      name: 'The specific logo',
      url: 'https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png',
    },
  ],
});

Notes on Image Size

When prompting with images, it is necessary to pay attention to the requirements of the AI model provider regarding the size and dimensions of the images. Images that are too large (such as exceeding 10M) or too small (such as being less than 10 pixels) may cause errors when the model is invoked. The specific restrictions should be based on the documentation provided by the AI model provider you are using.