MyDisct Solver API Docs - Captcha Solving & BBRE Guide

What is a Hybrid Workflow?

A hybrid workflow is a data collection or automation strategy that uses both passive and adaptive modes within the same pipeline. Instead of committing to a single mode for your entire project, you choose the right mode for each individual step based on what that step actually requires. The core idea is simple: use passive mode whenever you can, and switch to adaptive mode only when you must.

Passive mode is fast and inexpensive. It sends HTTP requests through a real browser environment, giving you authentic TLS fingerprints and browser-level headers, but it does not render JavaScript or provide browser automation. Adaptive mode launches a full Chromium browser instance that you can control programmatically. It handles JavaScript rendering, form filling, button clicking, screenshots, and any other browser interaction you need. The tradeoff is that adaptive mode is slower and costs more per request.

In most real-world projects, not every step requires full browser automation. A typical e-commerce scraping pipeline might need adaptive mode to log in and navigate past JavaScript challenges, but once authenticated, the actual product data pages can be fetched much faster with passive mode. A news aggregation system might use passive mode for 95% of its sources and only switch to adaptive mode for the handful of sites that require JavaScript rendering. By mixing modes strategically, you can reduce your costs by 40-70% compared to running everything in adaptive mode, while still handling every site reliably.

When to Use a Hybrid Approach

Not every project benefits from a hybrid workflow. If all your target pages are simple HTML with no JavaScript requirements, passive mode alone is sufficient. If every page requires full browser interaction, adaptive mode alone makes more sense. The hybrid approach shines when your workflow includes a mix of both types of pages, which is the case for the majority of real-world automation projects.

Scenario 1: Authentication Gates

Many websites require you to log in before accessing data. The login page typically involves JavaScript-rendered forms, CSRF tokens, and sometimes CAPTCHA challenges. Once you are authenticated, the actual data pages are often server-rendered HTML that can be fetched with simple HTTP requests. In this scenario, you use adaptive mode for the login step and then switch to passive mode for all subsequent data fetching, carrying the authentication cookies forward through a session.

Scenario 2: Mixed Content Sources

When you are collecting data from multiple sources, some sites will work perfectly with passive mode while others require adaptive mode. Rather than running everything in adaptive mode to cover the hardest cases, you can classify your sources and route each one to the appropriate mode. This is especially common in price comparison, news aggregation, and market research applications where you pull data from dozens or hundreds of different websites.

Scenario 3: Progressive Enhancement

Start every request in passive mode. If the response indicates that the page requires JavaScript rendering (empty body, redirect to a JavaScript challenge page, or missing expected content), retry the same URL in adaptive mode. This approach ensures you only pay the adaptive mode premium when it is actually needed, and it works well when you do not know in advance which pages will require browser automation.

Scenario 4: Session-Based Data Pipelines

Complex data pipelines often involve multiple steps: navigate to a search page, enter search terms, paginate through results, and then visit each result page to extract detailed data. The search and navigation steps might require adaptive mode for form interaction, while the individual result pages can be fetched in passive mode. Using sessions to maintain state between these steps lets you switch modes while preserving cookies and authentication.

Passive Mode Deep Dive

Passive mode is the workhorse of most BBRE integrations. It handles the majority of web requests efficiently and cost-effectively. Understanding its strengths and limitations is essential for designing effective hybrid workflows.

How Passive Mode Works

When you send a request in passive mode, BBRE routes it through a real browser environment to generate authentic TLS fingerprints, header ordering, and HTTP/2 characteristics. However, it does not launch a visible browser window or execute client-side JavaScript. The request goes out, the response comes back, and you get the raw HTML, headers, cookies, and status code. This makes passive mode significantly faster than adaptive mode because there is no page rendering, no DOM construction, and no JavaScript execution overhead.

When Passive Mode is Sufficient

Fetching server-rendered HTML pages where the content is in the initial response
Calling REST APIs that check TLS fingerprints or browser-like headers
Downloading files, images, or other static resources
Making authenticated requests where you already have valid cookies or tokens
High-volume data collection where speed and cost efficiency are priorities
Accessing pages with basic bot protection that checks headers but not JavaScript execution

Passive Mode Advantages

Advantage	Details
Speed	2-5x faster than adaptive mode because no browser rendering is needed
Cost	Lower per-request cost due to reduced server resource usage
Concurrency	Supports higher concurrent request volumes with the same account resources
Simplicity	No need to manage browser state, wait for elements, or handle page load events

Passive Mode Limitations

Cannot execute client-side JavaScript, so JavaScript-rendered content will be missing
Cannot interact with page elements (no clicking, typing, or form submission)
Cannot take screenshots of rendered pages
Cannot handle sites that require JavaScript execution to serve content
Cannot bypass JavaScript-based bot challenges that require code execution

JavaScript

const BBREClient = require("mydisctsolver-bbre");

const client = new BBREClient({
  apiKey: "YOUR_API_KEY"
});

async function fetchProductPage(productUrl) {
  const result = await client.get(productUrl, {
    mode: "passive",
    sensibility: "medium"
  });

  return {
    statusCode: result.statusCode,
    body: result.body,
    cookies: result.cookies
  };
}

async function main() {
  const urls = [
    "https://shop.example.com/product/12345",
    "https://shop.example.com/product/67890",
    "https://shop.example.com/product/11111"
  ];

  const results = await Promise.all(urls.map(fetchProductPage));
  results.forEach((r, i) => console.log("Product", i + 1, "status:", r.statusCode));
}

main();

Adaptive Mode Deep Dive

Adaptive mode is the heavy-duty tool in your BBRE toolkit. It launches a full Chromium browser instance that you control through the API, giving you the ability to do anything a real user can do in a browser. When passive mode hits a wall, adaptive mode breaks through it.

How Adaptive Mode Works

When you create a session in adaptive mode, the BBRE processor allocates a dedicated Chromium browser context with a unique fingerprint profile. This browser instance persists for the lifetime of the session, maintaining cookies, local storage, session storage, and all other browser state. You interact with the browser through the Browser API, sending commands like navigate, click, fill, type, wait, scroll, and screenshot. Each command is executed in the real browser and the result is returned to you.

When Adaptive Mode is Needed

Pages that require JavaScript execution to render their content (SPAs, React/Vue/Angular apps)
Login flows with JavaScript-rendered forms, CSRF tokens, or multi-step authentication
Sites that serve a JavaScript challenge before showing content
Workflows that require clicking buttons, filling forms, or selecting dropdown options
Pages where you need to scroll to trigger lazy-loaded content
Scenarios where you need screenshots for verification or monitoring
Complex multi-page workflows that require maintaining browser state

Adaptive Mode Capabilities

Capability	Description
Navigation	Navigate to URLs, go back/forward, reload pages
Form Interaction	Click buttons, fill input fields, select dropdowns, check checkboxes
Content Extraction	Get page HTML, text content, titles, URLs, find elements by selector or text
Waiting	Wait for elements, text, selectors, or fixed time periods
Scrolling	Scroll up, down, to top, to bottom, or to specific positions
Screenshots	Capture full-page or viewport screenshots as base64 images
JavaScript Execution	Execute arbitrary JavaScript in the page context
Cookie Management	Get, set, and clear cookies programmatically

JavaScript

const BBREClient = require("mydisctsolver-bbre");

const client = new BBREClient({
  apiKey: "YOUR_API_KEY"
});

async function loginToSite() {
  const session = await client.createSession({
    mode: "adaptive",
    sensibility: "high",
    timeout: 300
  });

  await session.start();
  const browser = session.browser();

  await browser.navigate("https://app.example.com/login");
  await browser.waitForSelector("#email");
  await browser.fill("#email", "[email protected]");
  await browser.fill("#password", "securePassword123");
  await browser.click("#login-button");
  await browser.waitForSelector(".dashboard-content");

  const title = await browser.getTitle();
  console.log("Logged in successfully:", title);

  await session.close();
}

loginToSite();

Example 1: E-Commerce Price Scraping

This example demonstrates a common hybrid workflow for e-commerce data collection. The target site requires a login to see wholesale prices, and the login page uses a JavaScript-rendered form with CSRF protection. Once logged in, the product catalog pages are server-rendered HTML that can be fetched efficiently with passive mode.

The workflow has three phases: first, use adaptive mode to log in and capture the authentication cookies. Second, use those cookies in passive mode to fetch product listing pages at high speed. Third, use passive mode to fetch individual product detail pages in parallel. This approach is typically 3-4x faster and significantly cheaper than running the entire pipeline in adaptive mode.

JavaScript (Node.js SDK)

JavaScript

const BBREClient = require("mydisctsolver-bbre");

const client = new BBREClient({
  apiKey: "YOUR_API_KEY"
});

async function loginAndGetCookies() {
  const session = await client.createSession({
    mode: "adaptive",
    sensibility: "high",
    timeout: 300
  });

  await session.start();
  const browser = session.browser();

  await browser.navigate("https://wholesale.example.com/login");
  await browser.waitForSelector("#login-form");
  await browser.fill("#email", "[email protected]");
  await browser.fill("#password", "wholesalePass123");
  await browser.click("#submit-login");
  await browser.waitForSelector(".account-dashboard");

  const cookies = await browser.getCookies();
  await session.close();

  return cookies;
}

async function fetchProductList(cookies, page) {
  const result = await client.get(
    "https://wholesale.example.com/products?page=" + page,
    {
      mode: "passive",
      sensibility: "medium",
      cookies: cookies
    }
  );

  return result.body;
}

async function fetchProductDetail(cookies, productId) {
  const result = await client.get(
    "https://wholesale.example.com/product/" + productId,
    {
      mode: "passive",
      sensibility: "medium",
      cookies: cookies
    }
  );

  return result.body;
}

async function main() {
  const cookies = await loginAndGetCookies();
  console.log("Login complete, got", Object.keys(cookies).length, "cookies");

  const totalPages = 10;
  for (let page = 1; page <= totalPages; page++) {
    const listHtml = await fetchProductList(cookies, page);
    console.log("Fetched product list page", page);

    const productIds = extractProductIds(listHtml);

    const details = await Promise.all(
      productIds.map(id => fetchProductDetail(cookies, id))
    );

    details.forEach((html, i) => {
      const price = extractPrice(html);
      console.log("Product", productIds[i], "price:", price);
    });
  }
}

function extractProductIds(html) {
  const matches = html.match(/data-product-id="(d+)"/g) || [];
  return matches.map(m => m.match(/d+/)[0]);
}

function extractPrice(html) {
  const match = html.match(/class="price">$?([d.]+)/);
  return match ? match[1] : "N/A";
}

main();

Python

import requests
import re

API_BASE = "https://bbre-solver-api.mydisct.com"
API_KEY = "YOUR_API_KEY"

headers = {
    "Content-Type": "application/json",
    "x-api-key": API_KEY
}

session_response = requests.post(
    API_BASE + "/session/create",
    headers=headers,
    json={
        "mode": "adaptive",
        "sensibility": "high",
        "timeout": 300
    }
)

session_data = session_response.json()
session_id = session_data["session"]["id"]

requests.post(
    API_BASE + "/browser/action",
    headers=headers,
    json={
        "sessionId": session_id,
        "action": "navigate",
        "params": {"url": "https://wholesale.example.com/login"}
    }
)

requests.post(
    API_BASE + "/browser/action",
    headers=headers,
    json={
        "sessionId": session_id,
        "action": "fill",
        "params": {"selector": "#email", "value": "[email protected]"}
    }
)

requests.post(
    API_BASE + "/browser/action",
    headers=headers,
    json={
        "sessionId": session_id,
        "action": "fill",
        "params": {"selector": "#password", "value": "wholesalePass123"}
    }
)

requests.post(
    API_BASE + "/browser/action",
    headers=headers,
    json={
        "sessionId": session_id,
        "action": "click",
        "params": {"selector": "#submit-login"}
    }
)

requests.post(
    API_BASE + "/browser/action",
    headers=headers,
    json={
        "sessionId": session_id,
        "action": "waitForSelector",
        "params": {"selector": ".account-dashboard"}
    }
)

cookie_response = requests.post(
    API_BASE + "/browser/action",
    headers=headers,
    json={
        "sessionId": session_id,
        "action": "getCookies",
        "params": {}
    }
)

auth_cookies = cookie_response.json()["result"]

requests.post(
    API_BASE + "/session/close",
    headers=headers,
    json={"sessionId": session_id}
)

for page in range(1, 11):
    list_response = requests.post(
        API_BASE + "/request/execute",
        headers=headers,
        json={
            "url": f"https://wholesale.example.com/products?page={page}",
            "method": "GET",
            "mode": "passive",
            "sensibility": "medium",
            "cookies": auth_cookies
        }
    )

    list_html = list_response.json()["task"]["result"]["body"]
    product_ids = re.findall(r'data-product-id="(d+)"', list_html)

    for product_id in product_ids:
        detail_response = requests.post(
            API_BASE + "/request/execute",
            headers=headers,
            json={
                "url": f"https://wholesale.example.com/product/{product_id}",
                "method": "GET",
                "mode": "passive",
                "sensibility": "medium",
                "cookies": auth_cookies
            }
        )

        detail_html = detail_response.json()["task"]["result"]["body"]
        price_match = re.search(r'class="price">$?([d.]+)', detail_html)
        price = price_match.group(1) if price_match else "N/A"
        print(f"Product {product_id}: ${price}")

cURL (Step-by-Step)

The hybrid workflow with cURL requires multiple sequential commands. First, create an adaptive session and perform the login, then use the captured cookies for passive mode requests.

Bash

curl -X POST https://bbre-solver-api.mydisct.com/session/create   -H "Content-Type: application/json"   -H "x-api-key: YOUR_API_KEY"   -d '{
    "mode": "adaptive",
    "sensibility": "high",
    "timeout": 300
  }'

Bash

curl -X POST https://bbre-solver-api.mydisct.com/browser/action   -H "Content-Type: application/json"   -H "x-api-key: YOUR_API_KEY"   -d '{
    "sessionId": "SESSION_ID_FROM_ABOVE",
    "action": "navigate",
    "params": {"url": "https://wholesale.example.com/login"}
  }'

Bash

curl -X POST https://bbre-solver-api.mydisct.com/request/execute   -H "Content-Type: application/json"   -H "x-api-key: YOUR_API_KEY"   -d '{
    "url": "https://wholesale.example.com/products?page=1",
    "method": "GET",
    "mode": "passive",
    "sensibility": "medium",
    "cookies": {"session_token": "abc123", "auth": "xyz789"}
  }'

Example 2: Data Collection with Authentication

This example shows a workflow where you need to authenticate through a complex login flow before collecting data. The login requires adaptive mode because the site uses a JavaScript-rendered form with dynamic CSRF tokens and a multi-step verification process. After authentication, the data endpoints return JSON responses that can be fetched efficiently with passive mode.

The key insight here is that the expensive adaptive mode session is only needed for the initial authentication. Once you have the session cookies, you close the adaptive session to free up resources and switch to passive mode for all data requests. This pattern is especially effective when you need to collect large volumes of data behind an authentication wall.

JavaScript (Node.js SDK)

JavaScript

const BBREClient = require("mydisctsolver-bbre");

const client = new BBREClient({
  apiKey: "YOUR_API_KEY"
});

async function authenticateWithAdaptive() {
  const session = await client.createSession({
    mode: "adaptive",
    sensibility: "high",
    timeout: 300
  });

  await session.start();
  const browser = session.browser();

  await browser.navigate("https://data-portal.example.com/auth/login");
  await browser.waitForSelector("#login-form");

  await browser.fill("#username", "datauser");
  await browser.fill("#password", "secureAccess456");
  await browser.click("#next-step");

  await browser.waitForSelector("#verification-code");
  await browser.fill("#verification-code", "123456");
  await browser.click("#verify-button");

  await browser.waitForSelector(".portal-home");

  const cookies = await browser.getCookies();
  const authToken = await browser.execute(
    "return document.cookie.split(';').find(c => c.trim().startsWith('auth_token=')).split('=')[1]"
  );

  await session.close();

  return { cookies, authToken };
}

async function fetchDataWithPassive(cookies, endpoint) {
  const result = await client.get(
    "https://data-portal.example.com/api/v2" + endpoint,
    {
      mode: "passive",
      sensibility: "low",
      cookies: cookies,
      headers: {
        "Accept": "application/json"
      }
    }
  );

  return JSON.parse(result.body);
}

async function main() {
  const { cookies } = await authenticateWithAdaptive();
  console.log("Authentication complete");

  const endpoints = [
    "/reports/daily",
    "/reports/weekly",
    "/reports/monthly",
    "/analytics/overview",
    "/analytics/trends"
  ];

  for (const endpoint of endpoints) {
    const data = await fetchDataWithPassive(cookies, endpoint);
    console.log("Fetched", endpoint, "- records:", data.results.length);
  }

  let page = 1;
  let hasMore = true;

  while (hasMore) {
    const data = await fetchDataWithPassive(
      cookies,
      "/transactions?page=" + page + "&limit=100"
    );

    console.log("Page", page, "- transactions:", data.results.length);
    hasMore = data.pagination.hasNext;
    page++;
  }
}

main();

Python

import requests

API_BASE = "https://bbre-solver-api.mydisct.com"
API_KEY = "YOUR_API_KEY"

headers = {
    "Content-Type": "application/json",
    "x-api-key": API_KEY
}

session_resp = requests.post(
    API_BASE + "/session/create",
    headers=headers,
    json={"mode": "adaptive", "sensibility": "high", "timeout": 300}
)
session_id = session_resp.json()["session"]["id"]

actions = [
    {"action": "navigate", "params": {"url": "https://data-portal.example.com/auth/login"}},
    {"action": "waitForSelector", "params": {"selector": "#login-form"}},
    {"action": "fill", "params": {"selector": "#username", "value": "datauser"}},
    {"action": "fill", "params": {"selector": "#password", "value": "secureAccess456"}},
    {"action": "click", "params": {"selector": "#next-step"}},
    {"action": "waitForSelector", "params": {"selector": "#verification-code"}},
    {"action": "fill", "params": {"selector": "#verification-code", "value": "123456"}},
    {"action": "click", "params": {"selector": "#verify-button"}},
    {"action": "waitForSelector", "params": {"selector": ".portal-home"}}
]

for act in actions:
    requests.post(
        API_BASE + "/browser/action",
        headers=headers,
        json={"sessionId": session_id, "action": act["action"], "params": act["params"]}
    )

cookie_resp = requests.post(
    API_BASE + "/browser/action",
    headers=headers,
    json={"sessionId": session_id, "action": "getCookies", "params": {}}
)
auth_cookies = cookie_resp.json()["result"]

requests.post(
    API_BASE + "/session/close",
    headers=headers,
    json={"sessionId": session_id}
)

endpoints = [
    "/reports/daily",
    "/reports/weekly",
    "/reports/monthly",
    "/analytics/overview",
    "/analytics/trends"
]

for endpoint in endpoints:
    resp = requests.post(
        API_BASE + "/request/execute",
        headers=headers,
        json={
            "url": f"https://data-portal.example.com/api/v2{endpoint}",
            "method": "GET",
            "mode": "passive",
            "sensibility": "low",
            "cookies": auth_cookies,
            "headers": {"Accept": "application/json"}
        }
    )

    data = resp.json()["task"]["result"]["body"]
    print(f"Fetched {endpoint}")

Example 3: Session-Based State Sharing

This example demonstrates how to use BBRE sessions to maintain state when switching between passive and adaptive approaches within the same workflow. Sessions preserve cookies, browser fingerprints, and other state data across multiple requests, making them the ideal mechanism for hybrid workflows that need continuity.

The scenario here is a real estate data aggregation pipeline. You need to search for properties on a site that uses JavaScript for its search interface, then fetch each property listing page. The search requires adaptive mode for form interaction, but the individual listing pages are server-rendered and work perfectly with passive session requests.

JavaScript (Node.js SDK)

JavaScript

const BBREClient = require("mydisctsolver-bbre");

const client = new BBREClient({
  apiKey: "YOUR_API_KEY"
});

async function hybridPropertySearch() {
  const session = await client.createSession({
    mode: "adaptive",
    sensibility: "high",
    timeout: 600
  });

  await session.start();
  const browser = session.browser();

  await browser.navigate("https://realestate.example.com/search");
  await browser.waitForSelector("#search-form");
  await browser.fill("#location", "San Francisco, CA");
  await browser.select("#property-type", "apartment");
  await browser.fill("#min-price", "500000");
  await browser.fill("#max-price", "1500000");
  await browser.click("#search-button");
  await browser.waitForSelector(".search-results");

  const resultsHtml = await browser.getHtml();
  const listingUrls = extractListingUrls(resultsHtml);
  console.log("Found", listingUrls.length, "listings");

  const listings = [];
  for (const url of listingUrls) {
    const result = await session.request({
      url: url,
      method: "GET"
    });

    listings.push({
      url: url,
      html: result.body,
      statusCode: result.statusCode
    });
  }

  let hasNextPage = true;
  let pageNum = 2;

  while (hasNextPage && pageNum <= 5) {
    const nextButton = await browser.find(".pagination .next-page");

    if (!nextButton) {
      hasNextPage = false;
      break;
    }

    await browser.click(".pagination .next-page");
    await browser.waitForSelector(".search-results");

    const pageHtml = await browser.getHtml();
    const pageUrls = extractListingUrls(pageHtml);

    for (const url of pageUrls) {
      const result = await session.request({
        url: url,
        method: "GET"
      });

      listings.push({
        url: url,
        html: result.body,
        statusCode: result.statusCode
      });
    }

    pageNum++;
  }

  await session.close();

  console.log("Total listings collected:", listings.length);
  return listings;
}

function extractListingUrls(html) {
  const matches = html.match(/href="(/listing/[^"]+)"/g) || [];
  return matches.map(m => "https://realestate.example.com" + m.match(/href="([^"]+)"/)[1]);
}

hybridPropertySearch();

Python

import requests
import re

API_BASE = "https://bbre-solver-api.mydisct.com"
API_KEY = "YOUR_API_KEY"

headers = {
    "Content-Type": "application/json",
    "x-api-key": API_KEY
}

session_resp = requests.post(
    API_BASE + "/session/create",
    headers=headers,
    json={"mode": "adaptive", "sensibility": "high", "timeout": 600}
)
session_id = session_resp.json()["session"]["id"]

search_actions = [
    {"action": "navigate", "params": {"url": "https://realestate.example.com/search"}},
    {"action": "waitForSelector", "params": {"selector": "#search-form"}},
    {"action": "fill", "params": {"selector": "#location", "value": "San Francisco, CA"}},
    {"action": "select", "params": {"selector": "#property-type", "value": "apartment"}},
    {"action": "fill", "params": {"selector": "#min-price", "value": "500000"}},
    {"action": "fill", "params": {"selector": "#max-price", "value": "1500000"}},
    {"action": "click", "params": {"selector": "#search-button"}},
    {"action": "waitForSelector", "params": {"selector": ".search-results"}}
]

for act in search_actions:
    requests.post(
        API_BASE + "/browser/action",
        headers=headers,
        json={"sessionId": session_id, "action": act["action"], "params": act["params"]}
    )

html_resp = requests.post(
    API_BASE + "/browser/action",
    headers=headers,
    json={"sessionId": session_id, "action": "getHtml", "params": {}}
)
results_html = html_resp.json()["result"]

listing_paths = re.findall(r'href="(/listing/[^"]+)"', results_html)
listing_urls = ["https://realestate.example.com" + path for path in listing_paths]

listings = []
for url in listing_urls:
    detail_resp = requests.post(
        API_BASE + "/session/request",
        headers=headers,
        json={
            "sessionId": session_id,
            "url": url,
            "method": "GET"
        }
    )

    result = detail_resp.json()["task"]["result"]
    listings.append({
        "url": url,
        "body": result["body"],
        "statusCode": result["statusCode"]
    })

requests.post(
    API_BASE + "/session/close",
    headers=headers,
    json={"sessionId": session_id}
)

print(f"Collected {len(listings)} property listings")

Session-Based Hybrid Workflow

Sessions are the backbone of hybrid workflows. They provide a persistent browser context that maintains cookies, fingerprint profiles, and other state data across multiple requests. When you switch between adaptive browser actions and passive-style session requests, the session ensures continuity. Here is a detailed look at how sessions enable hybrid workflows.

Creating a Session for Hybrid Use

When you plan to use both browser actions and regular HTTP requests within the same workflow, create your session in adaptive mode. Adaptive sessions support both browser automation commands and regular HTTP requests through the session request endpoint. A passive session only supports regular HTTP requests and cannot execute browser actions.

JavaScript

const BBREClient = require("mydisctsolver-bbre");

const client = new BBREClient({
  apiKey: "YOUR_API_KEY"
});

const session = await client.createSession({
  mode: "adaptive",
  sensibility: "medium",
  timeout: 600
});

await session.start();
const browser = session.browser();

await browser.navigate("https://app.example.com/login");
await browser.fill("#email", "[email protected]");
await browser.fill("#password", "password123");
await browser.click("#login-btn");
await browser.waitForSelector(".dashboard");

const dashboardData = await session.request({
  url: "https://app.example.com/api/dashboard",
  method: "GET",
  headers: {"Accept": "application/json"}
});

const reportsData = await session.request({
  url: "https://app.example.com/api/reports?range=30d",
  method: "GET",
  headers: {"Accept": "application/json"}
});

const usersData = await session.request({
  url: "https://app.example.com/api/users?limit=50",
  method: "GET",
  headers: {"Accept": "application/json"}
});

console.log("Dashboard:", JSON.parse(dashboardData.body));
console.log("Reports:", JSON.parse(reportsData.body));
console.log("Users:", JSON.parse(usersData.body));

await session.close();

Cookie Transfer Between Modes

One of the most powerful hybrid patterns is extracting cookies from an adaptive session and using them in standalone passive requests. This lets you authenticate once with adaptive mode, then make unlimited passive requests with those cookies without keeping the expensive adaptive session open. The cookies carry your authentication state, so the target site treats your passive requests as coming from an authenticated user.

JavaScript

const BBREClient = require("mydisctsolver-bbre");

const client = new BBREClient({
  apiKey: "YOUR_API_KEY"
});

async function extractCookiesFromAdaptive(loginUrl, credentials) {
  const session = await client.createSession({
    mode: "adaptive",
    sensibility: "high",
    timeout: 120
  });

  await session.start();
  const browser = session.browser();

  await browser.navigate(loginUrl);
  await browser.waitForSelector(credentials.formSelector);
  await browser.fill(credentials.emailSelector, credentials.email);
  await browser.fill(credentials.passwordSelector, credentials.password);
  await browser.click(credentials.submitSelector);
  await browser.waitForSelector(credentials.successSelector);

  const cookies = await browser.getCookies();
  await session.close();

  return cookies;
}

async function passiveRequestWithCookies(url, cookies) {
  return await client.get(url, {
    mode: "passive",
    sensibility: "medium",
    cookies: cookies
  });
}

async function main() {
  const cookies = await extractCookiesFromAdaptive(
    "https://platform.example.com/login",
    {
      formSelector: "#login-form",
      emailSelector: "#email",
      passwordSelector: "#password",
      submitSelector: "#submit",
      successSelector: ".user-profile",
      email: "[email protected]",
      password: "analysisPass789"
    }
  );

  const urls = [
    "https://platform.example.com/data/export/csv?type=sales",
    "https://platform.example.com/data/export/csv?type=inventory",
    "https://platform.example.com/data/export/csv?type=customers",
    "https://platform.example.com/data/export/csv?type=orders"
  ];

  const results = await Promise.all(
    urls.map(url => passiveRequestWithCookies(url, cookies))
  );

  results.forEach((result, i) => {
    console.log("Dataset", i + 1, "size:", result.body.length, "bytes");
  });
}

main();

Session Limits and Resource Management

Each account can have a maximum of 10 active sessions at any time. In hybrid workflows, it is critical to close sessions as soon as you no longer need them. A common mistake is keeping adaptive sessions open while running passive requests that do not need the session. If you have extracted the cookies you need, close the adaptive session immediately and use the cookies in standalone passive requests. This frees up your session slots for other workflows.

Session Timeout

Sessions have a configurable timeout between 60 and 3600 seconds. If a session remains inactive beyond its timeout period, it is automatically marked as expired. Set your session timeout based on the expected duration of your workflow. For short login-and-extract workflows, 120 seconds is usually sufficient. For longer multi-step processes, increase the timeout accordingly.

Performance Comparison

Understanding the performance characteristics of each mode is essential for designing efficient hybrid workflows. The numbers below are approximate and vary based on target site complexity, sensibility level, and network conditions, but they give you a reliable baseline for planning.

Speed Comparison

Metric	Passive Mode	Adaptive Mode
Simple page fetch (low sensibility)	1-3 seconds	5-10 seconds
Simple page fetch (medium sensibility)	2-5 seconds	8-15 seconds
Simple page fetch (high sensibility)	3-8 seconds	12-25 seconds
Session creation	1-2 seconds	3-8 seconds
Browser action (click, fill)	N/A	0.5-3 seconds per action
Full login workflow (5 actions)	N/A	10-30 seconds total

Cost Efficiency of Hybrid Approach

The following table illustrates the cost savings of a hybrid approach compared to running everything in adaptive mode. These numbers are based on a typical e-commerce scraping workflow that involves logging in, browsing 10 category pages, and fetching 200 product detail pages.

Approach	Adaptive Requests	Passive Requests	Relative Cost
All Adaptive	211	0	100%
Hybrid (login adaptive, rest passive)	6	205	35-45%
All Passive (if possible)	0	211	25-30%

Throughput Comparison

Because passive mode requests are faster and use fewer server resources, you can achieve significantly higher throughput with passive mode. In a hybrid workflow, the adaptive mode steps are typically the bottleneck. Design your pipeline so that the adaptive mode steps are as few and as fast as possible, and let passive mode handle the bulk of the data collection.

Scenario	Passive Throughput	Adaptive Throughput
Sequential requests (low sensibility)	20-30 requests/minute	4-8 requests/minute
Sequential requests (medium sensibility)	12-20 requests/minute	3-6 requests/minute
Parallel requests (5 concurrent)	60-100 requests/minute	15-30 requests/minute

Progressive Enhancement Pattern

The progressive enhancement pattern is a defensive strategy where you start every request in passive mode and only escalate to adaptive mode when passive mode fails or returns incomplete results. This approach is ideal when you are scraping a large number of URLs and you do not know in advance which ones will require browser automation.

The logic is straightforward: send the request in passive mode first. If the response body is empty, contains a JavaScript challenge page, or is missing the expected content markers, retry the same URL in adaptive mode. This ensures you only pay the adaptive mode premium for the pages that truly need it.

JavaScript Implementation

JavaScript

const BBREClient = require("mydisctsolver-bbre");

const client = new BBREClient({
  apiKey: "YOUR_API_KEY"
});

async function fetchWithFallback(url, contentValidator) {
  const passiveResult = await client.get(url, {
    mode: "passive",
    sensibility: "medium"
  });

  if (passiveResult.statusCode === 200 && contentValidator(passiveResult.body)) {
    return { mode: "passive", result: passiveResult };
  }

  const session = await client.createSession({
    mode: "adaptive",
    sensibility: "high",
    timeout: 120
  });

  await session.start();
  const browser = session.browser();
  await browser.navigate(url);
  await browser.wait(3000);

  const html = await browser.getHtml();
  const cookies = await browser.getCookies();
  await session.close();

  return {
    mode: "adaptive",
    result: { statusCode: 200, body: html, cookies: cookies }
  };
}

function hasProductData(html) {
  return html.includes("product-price") && html.includes("product-title");
}

async function main() {
  const urls = [
    "https://shop-a.example.com/product/1",
    "https://shop-b.example.com/item/abc",
    "https://shop-c.example.com/p/xyz"
  ];

  for (const url of urls) {
    const { mode, result } = await fetchWithFallback(url, hasProductData);
    console.log(url, "- fetched via", mode, "- size:", result.body.length);
  }
}

main();

Python Implementation

Python

import requests

API_BASE = "https://bbre-solver-api.mydisct.com"
API_KEY = "YOUR_API_KEY"

headers = {
    "Content-Type": "application/json",
    "x-api-key": API_KEY
}

def fetch_passive(url):
    resp = requests.post(
        API_BASE + "/request/execute",
        headers=headers,
        json={
            "url": url,
            "method": "GET",
            "mode": "passive",
            "sensibility": "medium"
        }
    )
    return resp.json()["task"]["result"]

def fetch_adaptive(url):
    session_resp = requests.post(
        API_BASE + "/session/create",
        headers=headers,
        json={"mode": "adaptive", "sensibility": "high", "timeout": 120}
    )
    session_id = session_resp.json()["session"]["id"]

    requests.post(
        API_BASE + "/browser/action",
        headers=headers,
        json={
            "sessionId": session_id,
            "action": "navigate",
            "params": {"url": url}
        }
    )

    requests.post(
        API_BASE + "/browser/action",
        headers=headers,
        json={
            "sessionId": session_id,
            "action": "wait",
            "params": {"timeout": 3000}
        }
    )

    html_resp = requests.post(
        API_BASE + "/browser/action",
        headers=headers,
        json={
            "sessionId": session_id,
            "action": "getHtml",
            "params": {}
        }
    )

    html = html_resp.json()["result"]

    requests.post(
        API_BASE + "/session/close",
        headers=headers,
        json={"sessionId": session_id}
    )

    return {"statusCode": 200, "body": html}

def has_product_data(html):
    return "product-price" in html and "product-title" in html

def fetch_with_fallback(url):
    result = fetch_passive(url)

    if result["statusCode"] == 200 and has_product_data(result["body"]):
        return "passive", result

    result = fetch_adaptive(url)
    return "adaptive", result

urls = [
    "https://shop-a.example.com/product/1",
    "https://shop-b.example.com/item/abc",
    "https://shop-c.example.com/p/xyz"
]

for url in urls:
    mode, result = fetch_with_fallback(url)
    print(f"{url} - fetched via {mode} - size: {len(result['body'])} bytes")

Choosing the Right Mode: Decision Guide

When designing a hybrid workflow, each step in your pipeline needs to be assigned to either passive or adaptive mode. Use the following decision guide to make the right choice for each step.

Use Passive Mode When:

The page content is available in the initial HTML response without JavaScript rendering
You are calling a REST API or JSON endpoint that checks TLS fingerprints
You already have valid authentication cookies from a previous adaptive session
You need to fetch many pages quickly and cost-effectively
The target site has basic or no bot protection
You are downloading files, images, or other static resources

Use Adaptive Mode When:

The page requires JavaScript execution to render its content
You need to fill forms, click buttons, or interact with page elements
The site presents a JavaScript challenge or CAPTCHA before showing content
You need to take screenshots of rendered pages
The login flow involves dynamic CSRF tokens or multi-step verification
You need to scroll to trigger lazy-loaded content
You need to execute custom JavaScript in the page context

Decision Table

Task	Recommended Mode	Reason
Fetch product page HTML	passive	Server-rendered content, no interaction needed
Log in to a website	adaptive	Form filling and button clicking required
Call a REST API endpoint	passive	Simple HTTP request, no browser interaction
Scrape a React SPA	adaptive	JavaScript rendering required for content
Download a CSV export	passive	Direct file download, no rendering needed
Submit a multi-step form	adaptive	Multiple form interactions across pages
Fetch authenticated API data	passive	Use cookies from adaptive login session
Take a page screenshot	adaptive	Browser rendering required for screenshots

Best Practices

Default to passive mode and escalate only when necessary. Start every new URL or endpoint with a passive mode request. If the response is incomplete, empty, or indicates a JavaScript challenge, then switch to adaptive mode for that specific URL. This approach minimizes cost and maximizes throughput across your entire pipeline.
Close adaptive sessions as soon as you have what you need. Adaptive sessions consume more resources than passive requests. Once you have extracted the cookies, tokens, or data you need from the adaptive session, close it immediately. Do not keep adaptive sessions open while running passive requests that do not need the session. Each account is limited to 10 active sessions, so freeing them up quickly is essential for parallel workflows.
Use cookie extraction for long-running data collection. When your workflow involves authenticating once and then fetching hundreds or thousands of pages, extract the authentication cookies from the adaptive session and use them in standalone passive requests. This lets you close the expensive adaptive session after just a few seconds and run the bulk of your data collection in fast, cheap passive mode.
Set appropriate sensibility levels for each mode. In hybrid workflows, you often need high sensibility for the adaptive login step (where bot detection is strictest) but can use medium or low sensibility for passive data fetching (where the requests are simpler). Do not use high sensibility everywhere. Match the sensibility to the actual threat level of each step.
Implement retry logic with mode escalation. Build your pipeline so that a failed passive request automatically retries in adaptive mode before giving up. This makes your system resilient to sites that occasionally serve JavaScript challenges or change their rendering behavior. Log which URLs required escalation so you can optimize your mode selection over time.
Monitor your balance before starting batch operations. Hybrid workflows can involve hundreds of requests. Check your account balance using the /account/balance endpoint before starting a large batch to ensure you have sufficient funds. Running out of balance mid-workflow can leave sessions in an inconsistent state and waste the work already completed.

Common Issues

Cookies Not Working in Passive Mode After Adaptive Login

If you extract cookies from an adaptive session and use them in passive mode requests but the target site does not recognize your authentication, there are several possible causes. First, make sure you are extracting all cookies, not just the session cookie. Some sites use multiple cookies for authentication (CSRF tokens, session IDs, user preferences). Second, check if the cookies have a Secure or HttpOnly flag that might affect how they are sent. Third, some sites tie their session cookies to specific browser fingerprints or IP addresses. If the passive request uses a different fingerprint than the adaptive session, the server may reject the cookies. In this case, keep the adaptive session open and use session requests instead of standalone passive requests.

SESSION_LIMIT_REACHED When Running Hybrid Pipelines

This error occurs when you try to create a new session but already have 10 active sessions. In hybrid workflows, this usually happens when adaptive sessions are not being closed properly. Common causes include: forgetting to close sessions after extracting cookies, error handling code that skips the session close step, and parallel workflows that each create their own sessions. Always wrap your session logic in try/finally blocks to ensure sessions are closed even when errors occur. Use the /account/sessions endpoint to list your active sessions and identify any that should be closed.

Adaptive Session Expires During Long Workflows

If your hybrid workflow takes longer than the session timeout, the adaptive session will expire and subsequent browser actions or session requests will fail with a SESSION_EXPIRED error. The default session timeout is 120 seconds, which may not be enough for complex workflows. Increase the timeout when creating the session (up to 3600 seconds maximum). Alternatively, restructure your workflow to extract cookies early and close the adaptive session, then use passive requests with those cookies for the remaining work.

Empty Response Body in Passive Mode

When a passive mode request returns a 200 status code but an empty or minimal body, the target page likely requires JavaScript rendering to display its content. This is the most common trigger for escalating to adaptive mode. Implement a content validation function that checks for expected markers in the response body (specific HTML elements, text patterns, or minimum content length). When validation fails, retry the URL in adaptive mode with a session that can render the JavaScript.

BBRE Hybrid Workflow Guide

What is a Hybrid Workflow?

When to Use a Hybrid Approach

Scenario 1: Authentication Gates

Scenario 2: Mixed Content Sources

Scenario 3: Progressive Enhancement

Scenario 4: Session-Based Data Pipelines

Passive Mode Deep Dive

How Passive Mode Works

When Passive Mode is Sufficient

Passive Mode Advantages

Passive Mode Limitations

Adaptive Mode Deep Dive

How Adaptive Mode Works

When Adaptive Mode is Needed

Adaptive Mode Capabilities

Example 1: E-Commerce Price Scraping

JavaScript (Node.js SDK)

Python

cURL (Step-by-Step)

Example 2: Data Collection with Authentication

JavaScript (Node.js SDK)

Python

Example 3: Session-Based State Sharing

JavaScript (Node.js SDK)

Python

Session-Based Hybrid Workflow

Creating a Session for Hybrid Use

Cookie Transfer Between Modes

Session Limits and Resource Management

Performance Comparison

Speed Comparison

Cost Efficiency of Hybrid Approach

Throughput Comparison

Progressive Enhancement Pattern

JavaScript Implementation

Python Implementation

Choosing the Right Mode: Decision Guide

Use Passive Mode When:

Use Adaptive Mode When:

Decision Table

Best Practices

Common Issues

Cookies Not Working in Passive Mode After Adaptive Login

SESSION_LIMIT_REACHED When Running Hybrid Pipelines

Adaptive Session Expires During Long Workflows

Empty Response Body in Passive Mode

Related Documentation