Trails

Trails are fully supported in the JavaScript SDK.

Creating a Trail

A trail defines the following components:

• urls.ts: Exports an array of URL definitions
• selectors.ts: Exports an array of selector configurations
• actions.ts: Exports action classes that implement the TrailAction interface

A trail at the minimum has the following structure:

google-search/
    urls.ts
    selectors.ts
    actions.ts
    package.json

The package.json file defines the trail and its dependencies:

{
    "name": "google-search",
    "description": "Search google for webpages.",
    "version": "1.0.0",
    "dependencies": {
        "@monitoro/herd": "latest"
    }
}

You should never run the .ts files manually. Instead, use the Herd CLI to run, test, debug, and publish trails. Make sure to use version control to manage your trails, as publishing submits a built version to the Herd registry, not the source code.

Trail Implementation Guide

Step 1: Set up the trail structure

Create a new directory for your trail with the following files:

my-trail/
    urls.ts
    selectors.ts
    actions.ts
    package.json

Add the basic package.json configuration:

{
    "name": "my-trail",
    "version": "1.0.0",
    "dependencies": {
        "@monitoro/herd": "latest"
    }
}

Step 2: Define URLs

In urls.ts, export an array of URL definitions that your trail will interact with:

export default [{
    "id": "my-url",
    "template": "https://example.com/path?param1={param1}¶m2={param2}",
    "description": "Description of what this URL represents",
    "examples": [
        { "param1": "value1", "param2": "value2" }
    ],
    "params": {
        "param1": {
            "type": "string",
            "required": true,
            "description": "Description of param1"
        },
        "param2": {
            "type": "string",
            "required": false,
            "default": "defaultValue",
            "description": "Description of param2"
        }
    }
}];

Step 3: Define Selectors

In selectors.ts, export an array of selector configurations that define how to extract data from web pages:

export default [{
    "id": "my-selector",
    "value": {
        "dataKey": {
            "_$r": "#main-container",
            "title": { "_$": ".title" },
            "description": { "_$": ".description" },
            "link": { "_$": "a.link", "attribute": "href" }
        }
    },
    "description": "Selector for extracting specific data",
    "examples": [
        {
            "urlId": "my-url",
            "urlParams": { "param1": "value1", "param2": "value2" }
        }
    ]
}];

Step 4: Implement Actions

In actions.ts, define one or more action classes that implement the TrailAction interface:

import type { Device, TrailAction, TrailActionManifest, TrailRunResources } from "@monitoro/herd";

export class MyTrailAction implements TrailAction {
    manifest: TrailActionManifest = {
        name: "my-action",
        description: "Description of what this action does",
        params: {
            param1: {
                type: "string",
                description: "Description of param1"
            },
            param2: {
                type: "number",
                description: "Description of param2",
                default: 10
            }
        },
        result: {
            type: "array",
            description: "Description of the result",
            items: {
                type: "object",
                properties: {
                    property1: { type: "string" },
                    property2: { type: "number" }
                }
            }
        },
        examples: [
            {
                "param1": "value1",
                "param2": 10
            }
        ]
    }

    async test(device: Device, params: Record<string, any>, resources: TrailRunResources) {
        try {
            const result = await this.run(device, params, resources);
            // Validate result
            if (!result || result.length === 0) {
                return { status: "error", message: "No results found", result: [] };
            }
            return { status: "success", result };
        } catch (e) {
            return { status: "error", message: `Error: ${e}`, result: null };
        }
    }

    async run(device: Device, params: Record<string, any>, resources: TrailRunResources) {
        const { param1, param2 } = params;
        const page = await device.newPage();
        
        try {
            // Navigate to URL using the URL template from urls.ts
            await page.goto(resources.url('my-url', { param1, param2 }));
            
            // Extract data using selectors from selectors.ts
            const extracted = await page.extract(resources.selector('my-selector'));
            
            // Process extracted data
            const results = (extracted as any)?.dataKey || [];
            
            // Return processed results
            return results;
        } finally {
            await page.close();
        }
    }
}

Step 5: Testing and debugging

Use the Herd CLI to test your trail locally (make sure to define test cases as examples in the trail action manifest):

herd trail test --action my-trail

You can also watch for changes in the trail and re-run tests:

herd trail test --action my-trail --watch  # or herd trail test -a my-trail -w

And you can also test selectors only:

herd trail test --selector my-selector

And watch for changes in the selectors:

herd trail test --selector my-selector --watch  # or herd trail test -s my-selector -w

Step 6: Publish your trail

Publishing is coming soon!

Best Practices

1. Error handling: Implement robust error handling in your actions to handle network issues, missing elements, etc.
2. Performance: Minimize page loads and extract as much data as possible from each page.
3. Maintainability: Use descriptive names and add comments to make your trail easier to maintain.
4. Testing: Test your trail with different parameters to ensure it works in various scenarios.
5. Versioning: Increment your trail’s version in package.json when making changes.