Skip to main content

How to Submit A Form With NodeJS

How To Submit A Form With NodeJS

Form submission is essential in web development and automation, enabling interaction with web apps and different types of automating tasks. In NodeJS, various methods exist that simplify form handling.

In this article, we'll go through some of the methods for submitting forms using NodeJS and we'll provide detailed steps along the way.

Need help scraping the web?

Then check out ScrapeOps, the complete toolkit for web scraping.

TL:DR - How To Submit A Form With NodeJS​

To submit a form programmatically in NodeJS, use the node-fetch library for making HTTP requests. Here's a TL:DR:

First, install node-fetch:

npm install node-fetch

Then, run the following code:

const fetch = require('node-fetch');

async function submitForm() {
    try {
        const response = await fetch('https://your-form-endpoint.com', {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json',
            },
            body: JSON.stringify({
                username: 'your_username',
                password: 'your_password'
            }),
        });

        if (response.ok) {
            const data = await response.json();
            console.log('Form submitted successfully:', data);
        } else {
            console.error('Failed to submit form:', response.statusText);
        }
    } catch (error) {
        console.error('Error submitting form:', error);
    }
}

submitForm();

The script above:

  • First, import node-fetch and define an async function submitForm() to handle the form submission.
  • Use fetch to send a POST request to your form endpoint (https://your-form-endpoint.com) with JSON data containing the username and password.
  • The function checks the response status with response.ok and logs a success message if the submission is successful or an error message if it fails.
  • To ensure robustness, error handling is used to manage potential network issues or server errors that might occur during the form submission process.

Understanding Form Submission​

The process of form submission typically involves sending HTML forms from the client's browser (front-end) to the server side (back-end). The server then handles and processes the submitted data.

Overview of HTML Forms and Their Elements​

HTML forms are essential user-interaction components facilitating data input and transmission to a server for processing.

HTML forms consist of various elements, each designed to collect specific user information:

1. Input Fields:​

These foundational elements allow users to input text, numbers, dates, and more. They come in various types such as text, password, checkbox, radio, and file upload, accommodating diverse data needs.

The following snippet demonstrates input fields for username, password, and email:

<form action="/submit" method="post">
  <label for="username">Username:</label><br>
  <input type="text" id="username" name="username"><br><br>

  <label for="password">Password:</label><br>
  <input type="password" id="password" name="password"><br><br>

  <label for="email">Email:</label><br>
  <input type="email" id="email" name="email"><br><br>

  <input type="submit" value="Submit">
</form>

Here's how the form looks like when rendered:

HTML form

2. Buttons:​

Within forms, buttons serve as action triggers. They play a crucial role in initiating the transfer of form data to the server.

<button>Submit</button>

A button is created with the button keyword, i.e. its opening and closing tags.

3. Dropdowns and Selection Boxes:​

Dropdown menus and selection boxes provide users with predefined choices, ensuring data input consistency and user convenience.

4. Labels and Descriptions:​

Labels provide descriptive text alongside form elements, improving accessibility. Placeholders and tooltips further aid users in completing forms accurately.

5. Form Controls:​

Forms include checkboxes, radio buttons, and switches. These enable users to make selections or toggle options within the form.

The HTTP POST Method and Its Role in Form Submission​

When a user fills out a form and submits it, the HTTP POST method securely transmits the form data to a designated server endpoint.

Let's see the above form again as an example:

<form action="/submit" method="post">
  <label for="username">Username:</label><br>
  <input type="text" id="username" name="username"><br><br>

  <label for="password">Password:</label><br>
  <input type="password" id="password" name="password"><br><br>

  <label for="email">Email:</label><br>
  <input type="email" id="email" name="email"><br><br>

  <input type="submit" value="Submit">
</form>
  • Once a form is created, we add the method option to the form element.
  • The POST shown here discreetly sends data within the body of the HTTP request.
  • This is different from the GET method, which appends form data to the URL.
  • Unlike GET, POST ensures privacy and security, making it suitable for transmitting sensitive information such as passwords or payment details.

Exploring Methods for NodeJS Form Submission​

We'll explore different methods for handling form submissions in NodeJS. These approaches are tailored to different needs, ranging from straightforward HTTP requests to complex browser automation tasks:

  • Using the Axios library: Making HTTP requests and submitting forms directly using the Axios library.
  • Browser Automation with Puppeteer: Automating browser interactions to locate form elements, fill them out, and submit them programmatically.
  • Automating Browser Interactions with NightmareJS: Using NightmareJS to automate browser interactions and manage form submissions.
  • Web Scraping Approach with Cheerio and Axios: Using Cheerio and Axios to parse HTML forms and submit form data during web scraping tasks.
  • Headless Browsing with Playwright: Exploring headless browser automation using the Playwright library for form submissions.
  • Advanced Techniques and Libraries: Exploring additional methods and libraries available for handling form submissions in NodeJS.

Using the Axios library​

Axios is a powerful library in NodeJS for making HTTP requests, favored for its simplicity and versatility. This section will demonstrate how to effectively submit forms using Axios, covering fundamental concepts such as POST requests, form data handling, headers, and authentication strategies.

Submitting Forms with POST Requests​

To submit a form using Axios, we primarily use the axios.post() method.

Here's a basic example:

const axios = require('axios');

const url = 'https://example.org/post';
const formData = {
    field1: 'value1',
    field2: 'value2'
};

axios.post(url, formData)
    .then(response => {
        console.log('Status Code:', response.status);
        console.log('Response Data:', response.data);
    })
    .catch(error => {
        console.error('Error submitting form:', error);
    });

In this example, axios.post() sends a POST request to the specified url with the formData object. The response object contains the server's response to the HTTP request.

Error-catching is added to make sure any potential error is written to the console.

Using Axios Interceptors​

Axios interceptors allow to globally intercept requests or responses before they are handled by axios. This can be particularly useful for modifying headers or handling errors consistently across requests.

Here's how to utilize interceptors:

const axios = require('axios');

axios.interceptors.request.use(config => {
    config.headers.Authorization = `Bearer ${getToken()}`;
    return config;
}, error => {
    return Promise.reject(error);
});

const url = 'https://example.com/api/submit';
const formData = {
    field1: 'value1',
    field2: 'value2'
};

axios.post(url, formData)
    .then(response => {
        console.log('Form submitted successfully:', response.data);
    })
    .catch(error => {
        console.error('Error submitting form:', error);
    });

In the example above, Axios interceptors are utilized to globally modify outgoing requests. The interceptor adds a Bearer token to the Authorization header using the getToken() function, ensuring authenticated access to the API endpoint. We'll take a deeper look at authentication in the next section.

After adding the auth header, the axios.post() method sends a POST request to https://example.com/api/submit with form data contained in formData. Upon successful submission, the response data is logged to the console.

Error handling is implemented to catch and log any request errors. This approach shows how Axios simplifies HTTP request handling and enables secure and efficient communication with APIs in NodeJS applications.

Handling Authentication​

Authentication is crucial for securing web applications. Axios simplifies integrating authentication methods such as Basic Authentication and Bearer Token Authentication.

Basic Authentication​

Basic Authentication is a straightforward method where the username and password are encoded in Base64 and sent with the HTTP request. This method is suitable for scenarios where simple credential verification is required.

In the code below, we first define the endpoint URL and the form data to be submitted. We use axios.post() to send a POST request to the specified URL, including the form data and authentication credentials.

The auth option in Axios allows us to include the username and password, which Axios automatically encodes and sends in the Authorization header. This header ensures that the server can authenticate the client before processing the form data.

The .then() method is used to handle a successful response from the server, logging the status code and response data. Conversely, the .catch() method catches any errors that occur during the request and prints it to the console.

const axios = require('axios');

const url = 'https://example.com/secure-form';
const formData = {
    field1: 'value1',
    field2: 'value2'
};
const username = 'your_username';
const password = 'your_password';

axios.post(url, formData, {
    auth: {
        username,
        password
    }
})
.then(response => {
    console.log('Status Code:', response.status);
    console.log('Response Data:', response.data);
})
.catch(error => {
    console.error('Error submitting form:', error);
});

Bearer Token Authentication​

Bearer Token Authentication involves issuing a token to the user after a successful login, which is then used to authenticate subsequent requests.

This method is commonly used in APIs to verify that the request is coming from an authorized user. Bearer tokens offer a more secure way of handling authentication as they can be set to expire and can be easily revoked if compromised.

const axios = require('axios');

const url = 'https://api.example.com/data';
const formData = {
    field1: 'value1',
    field2: 'value2'
};
const authToken = 'your_auth_token';

axios.post(url, formData, {
    headers: {
        Authorization: `Bearer ${authToken}`
    }
})
.then(response => {
    console.log('Status Code:', response.status);
    console.log('Response Data:', response.data);
})
.catch(error => {
    console.error('Error submitting form:', error);
});
  • Here, we define the URL for the API endpoint and the form data to be submitted. We store the authentication token in a variable called authToken.
  • When making the POST request with Axios, we include this token in the Authorization header by setting it to Bearer <AUTH_TOKEN>. This header allows the server to validate the token and authenticate the request, ensuring that only authorized users can access the data.
  • The .then() method handles the successful response from the server, logging the status code and the response data. The .catch() method catches any errors that occur during the request and logs an error message, helping to identify issues such as expired or invalid tokens.

Using Bearer Token Authentication with Axios is a secure and efficient way to manage authenticated requests, especially for APIs that require strong security measures.

Handling File Uploads and Multi-part Form Data​

When dealing with file uploads, Axios supports sending files as multipart/form-data. Here's an example of how to upload a file:

const axios = require('axios');
const fs = require('fs');

const url = 'https://example.com/upload';
const formData = new FormData();
formData.append('file', fs.createReadStream('/path/to/file'));

axios.post(url, formData, {
    headers: formData.getHeaders()
})
.then(response => {
    console.log('File uploaded successfully:', response.data);
})
.catch(error => {
    console.error('Error uploading file:', error);
});
  • Here, a file is read from the local disk using fs and its createReadStream method.
  • After appending the file stream to the form data object using formData.append(), we create a POST request with axios and pass the headers from the form data object with formData.getHeaders().
  • After the .post method call, we are using then and catch to either log response data or the returned error from the example.com's server.

Browser Automation with Puppeteer​

Puppeteer is a robust tool for automating web browser interactions in NodeJS, providing a high-level API over the Chrome DevTools Protocol. This makes it a versatile choice for tasks such as form submission, web scraping, and UI testing. In this section, we will explore how to automate browser interactions with Puppeteer, with a particular focus on form interactions.

Setting Up Puppeteer​

Before we dive into form interactions, we need to set up Puppeteer. We start by installing it via npm:

npm install puppeteer

Once installed, we can create a basic script to launch a browser and navigate to a webpage:

const puppeteer = require('puppeteer');

(async () => {
    const browser = await puppeteer.launch({ headless: false });
    const page = await browser.newPage();
    await page.goto('https://example.com');

    // Perform actions . . .

    await browser.close();
})();

In this script, puppeteer.launch() initializes a new browser instance, and page.goto() navigates to the specified URL. Finally, we close the browser to make sure everything is cleaned up.

Locating Form Elements​

To interact with form elements, we first need to locate them on the webpage. Puppeteer offers several methods to find elements, such as page.$ for a single element and page.$$ for multiple elements.

Here's how to locate and interact with a text input field:

const puppeteer = require('puppeteer');

(async () => {
    const browser = await puppeteer.launch({ headless: false });
    const page = await browser.newPage();
    await page.goto('https://practice.expandtesting.com/login');

    const usernameInput = await page.$('#username');
    await usernameInput.type('practice');

    await browser.close();
})();

In this example, page.$('#username') finds the element with the ID username, and type('practice') inputs the specified text into the field. This allows us to simulate user input effectively.

Filling Out and Submitting Forms​

Once the form elements are located, filling out and submitting the form is straightforward. We use the type method shown previously to input data into fields and the click method to submit the form.

const puppeteer = require('puppeteer');

(async () => {
    const browser = await puppeteer.launch({ headless: false });
    const page = await browser.newPage();
    await page.goto('https://practice.expandtesting.com/login');

    const usernameInput = await page.$('#username');
    await usernameInput.type('practice');

    const passwordInput = await page.$('#password');
    await passwordInput.type('DarthVaderIsCool');

    const submitButton = await page.$('#login button');
    await submitButton.click();

    await browser.close();
})();

Here, we locate the username and password input fields, fill them with data, and then click the submit button to send the form. Puppeteer's methods make it easy to automate these actions.

Scheduling Form Submissions with Puppeteer​

Puppeteer enables scheduling of form submissions by simulating user interactions at specified intervals. This capability is useful for scenarios requiring periodic data updates or interactions with time-sensitive web forms.

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({ headless: true });
  const page = await browser.newPage();

  await page.goto('https://practice.expandtesting.com/login');

  setInterval(async () => {
    await page.type("input[name='username']", 'practice');
    await page.type("input[name='password']", 'DarthVaderIsCool');
    await page.click("button[type='submit']");
  }, 60000); // Submit form every 60 seconds

  // Do other actions . . .

  await browser.close();
})();

The above code demonstrates scheduling form submissions at a regular interval (every 60 seconds in this example) using setInterval(). It continuously fills out the form and submits it to the server, simulating periodic actions.

Handling Dynamic Forms and Elements​

Modern web applications often use JavaScript to dynamically generate form elements. Puppeteer can handle these scenarios by waiting for specific conditions before interacting with the elements.

Using Puppeteer's waitForSelector method, we can wait for elements to appear before proceeding:

const puppeteer = require('puppeteer');

(async () => {
    const browser = await puppeteer.launch({ headless: false });
    const page = await browser.newPage();
    await page.goto('https://example.com');

    await page.waitForSelector('#dynamic-element');

    const dynamicElement = await page.$('#dynamic-element');
    await dynamicElement.type('dynamic data');

    await browser.close();
})();

Here, waitForSelector('#dynamic-element') ensures that the script waits until the element with the ID dynamic-element appears before interacting with it.

Dealing with Complex Forms​

Sometimes forms require interaction with multiple elements or iframes. Puppeteer provides methods to switch between frames and interact with nested elements.

Here's an example of interacting with elements inside an iframe:

const puppeteer = require('puppeteer');

(async () => {
    const browser = await puppeteer.launch({ headless: false });
    const page = await browser.newPage();
    await page.goto('https://example.com');

    const iframeElement = await page.$('#iframe_id');
    const iframe = await iframeElement.contentFrame();

    const iframeInput = await iframe.$('#element_inside_iframe');
    await iframeInput.type('data inside iframe');

    await browser.close();
})();

In this script, we locate the iframe, then switch to its content frame using contentFrame(). After receiving the iframe object, we can then access the frame itself and interact with its elements like we would for regular elements.

Handling Asynchronous Form Submissions​

In Puppeteer, managing asynchronous form submissions involves executing JavaScript within the context of a web page using page.evaluate(). This capability is particularly useful when dealing with forms that submit data asynchronously via AJAX or similar mechanisms.

Here’s an example of how Puppeteer can handle an asynchronous form submission:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  await page.goto('https://example.com/login');

  await page.type('input[name="username"]', 'your_username');
  await page.type('input[name="password"]', 'your_password');

  const response = await page.evaluate(async () => {
    const formData = new FormData(document.querySelector('form'));
    const fetchOptions = {
      method: 'POST',
      body: formData
    };
    const fetchResponse = await fetch('https://example.com/login', fetchOptions);
    return fetchResponse.json();
  });

  console.log('Server response:', response);

  await browser.close();
})();

Here, Puppeteer navigates to a login page and asynchronously fills out the username and password fields using page.type(). The form submission is then simulated within page.evaluate() using the Fetch API. This approach ensures that the form data is sent to the server asynchronously, mimicking real user behavior.

The use of await inside page.evaluate() allows Puppeteer to wait for the asynchronous fetch operation to complete and receive the server's response. This response is then processed further within the script, such as parsing JSON data or handling error messages returned by the server.

Handling asynchronous form submissions in Puppeteer is crucial for scenarios where web applications rely on AJAX or JavaScript-based interactions. By leveraging Puppeteer's capabilities to execute JavaScript in the context of a page, developers can automate complex workflows involving form submissions and ensure reliable interaction with dynamic web pages.

Using NightmareJS​

NightmareJS is a powerful browser automation library built on Electron. It is ideal for tasks like UI testing, web scraping, and automating complex interactions with web applications.

This chapter will guide you through using NightmareJS, focusing on handling form submissions, managing authentication and redirects, dealing with form validation errors, and handling CSRF protection.

Handling Form Submission​

NightmareJS simplifies the process of filling out and submitting forms. We can locate form elements using CSS selectors and interact with them as a user would.

const Nightmare = require('nightmare');
const nightmare = Nightmare({ show: true });

nightmare
  .goto('https://practice.expandtesting.com/login')
  .type('#username', 'practice')
  .type('#password', 'DarthVaderIsCool')
  .click('#login button')
  .end()
  .then(() => console.log('Form submitted successfully'))
  .catch(error => console.error('Error:', error));

Here, we initialize Nightmare, navigate to the login page, and use the type method to enter text into the username and password fields. The click method submits the form. Finally, end closes the browser instance.

Handling Authentication and Redirects​

Nightmare can manage authentication challenges and handle redirects automatically. This is particularly useful when dealing with secure web pages that require user credentials.

const Nightmare = require('nightmare');
const nightmare = Nightmare({ show: true });

nightmare
  .goto('https://example.com/secure-form')
  .authentication('your_username', 'your_password')
  .type('#username', 'practice')
  .type('#password', 'DarthVaderIsCool')
  .click('#login button')
  .wait('#welcome')
  .end()
  .then(() => console.log('Authenticated and form submitted successfully'))
  .catch(error => console.error('Error:', error));

In this code, authentication('your_username', 'your_password') sets the basic authentication credentials. We type the corresponding values into #username and #password fields, and then "click" the login button.

The wait method ensures that we wait for a specific element (e.g., a welcome message) to appear before closing the browser. This approach ensures that our script handles authentication and any subsequent redirects seamlessly.

Handling Form Validation Errors​

Dealing with form validation errors is a common challenge in web automation. NightmareJS allows us to detect and handle these errors gracefully by inspecting the response after form submission.

const Nightmare = require('nightmare');
const nightmare = Nightmare({ show: true });

nightmare
  .goto('https://practice.expandtesting.com/login')
  .type('#username', 'practice')
  .type('#password', 'wrongpassword')
  .click('#login button')
  .wait('.error-message')
  .evaluate(() => document.querySelector('.error-message').innerText)
  .end()
  .then(errorMessage => {
    if (errorMessage.includes('Your password is invalid!')) {
      console.log('Login failed: Invalid credentials');
    } else {
      console.log('Login successful');
    }
  })
  .catch(error => console.error('Error:', error));

After attempting to log in with incorrect credentials, we wait for an error message to appear and then extract its text content using evaluate. Based on the error message, we can determine whether the login attempt was successful or failed due to invalid credentials.

Handling CSRF Protection​

Cross-Site Request Forgery (CSRF) protection is a security measure that prevents unauthorized actions on behalf of a user. NightmareJS can handle CSRF tokens by extracting them from forms and including them in the submission.

const Nightmare = require('nightmare');
const nightmare = Nightmare({ show: true });
const { JSDOM } = require('jsdom');

nightmare
  .goto('http://example.com/login')
  .evaluate(() => document.documentElement.innerHTML)
  .then(html => {
    const dom = new JSDOM(html);
    const csrfToken = dom.window.document.querySelector('input[name="csrf_token"]').value;

    return nightmare
      .type('#username', 'myusername')
      .type('#password', 'mypassword')
      .insert('input[name="csrf_token"]', csrfToken)
      .click('button[type="submit"]')
      .wait('#welcome')
      .end();
  })
  .then(() => console.log('Form with CSRF protection submitted successfully'))
  .catch(error => console.error('Error:', error));

In the code above, we first navigate to the login page and extract the CSRF token using evaluate and JSDOM. We then insert the token into the form before submitting it. This approach ensures that our automation script correctly handles CSRF protection, mimicking legitimate user behavior.

Cheerio and Axios​

Cheerio and Axios are powerful tools for web scraping and interacting with web pages in NodeJS applications. We have already seen Axios, which is a library for handling HTTP requests.

Cheerio, on the other hand, simplifies HTML parsing and manipulation. We can use both in tandem for parsing HTML forms, submitting forms, and navigating complex HTML structures.

Parsing HTML Forms​

Cheerio provides a jQuery-like API for parsing and manipulating HTML documents. Combined with Axios, we can fetch HTML content from a web page and extract form data easily. Here’s a basic example of parsing HTML forms:

const axios = require('axios');
const cheerio = require('cheerio');

const url = 'https://example.com/login';

axios.get(url)
  .then(response => {
    const $ = cheerio.load(response.data);
    const form = $('form').first();

    const formData = {};
    form.find('input').each((index, element) => {
      const name = $(element).attr('name');
      const value = $(element).val();
      formData[name] = value;
    });

    console.log('Form data:', formData);
  })
  .catch(error => {
    console.error('Error fetching and parsing HTML:', error);
  });

In this example, Axios fetches the HTML content from url, and Cheerio parses it. We extract form data by finding all input elements within the first form and storing them in formData. This approach is useful for scraping and analyzing HTML forms programmatically.

Submitting Forms​

Once we've parsed form data using Cheerio and Axios, we can submit forms by sending HTTP POST requests. Here’s an example demonstrating form submission using Axios:

const axios = require('axios');

const url = 'https://example.com/login';
const formData = {
  username: 'user123',
  password: 'password123'
};

axios.post(url, formData)
  .then(response => {
    console.log('Form submitted successfully');
    console.log('Response:', response.data);
  })
  .catch(error => {
    console.error('Error submitting form:', error);
  });

In this snippet, Axios sends a POST request to url with formData, simulating form submission. This method allows to automate interactions with web applications that require user input.

Cheerio simplifies navigating through complex HTML structures, making it easier to locate and interact with specific elements. Here’s an example of navigating nested elements:

const axios = require('axios');
const cheerio = require('cheerio');

const url = 'https://example.com/products';

axios.get(url)
  .then(response => {
    const $ = cheerio.load(response.data);
    const products = [];

    $('div.product').each((index, element) => {
      const name = $(element).find('h2').text();
      const price = $(element).find('span.price').text();
      products.push({ name, price });
    });

    console.log('Products:', products);
  })
  .catch(error => {
    console.error('Error fetching and parsing HTML:', error);
  });

In the example above, Cheerio is used to parse an HTML page containing product information. We navigate through nested elements to extract product names and prices, demonstrating how to handle complex HTML structures effectively.

By combining Cheerio for HTML parsing and manipulation with Axios for HTTP requests, we can create robust web scraping and automation scripts in NodeJS applications.

Headless Browsing with Playwright​

Within the NodeJS ecosystem, Playwright stands out as a versatile and powerful tool for automating interactions with web pages. Whether you're performing automated testing, web scraping, or form submission,

Playwright provides a robust framework that supports multiple browser engines including Chromium, Firefox, and WebKit.

Setting Up Headless Browsers​

Playwright simplifies the setup of headless browsers by providing easy installation of browser binaries. Here’s how to get started with Playwright:

# Install Playwright
npm install playwright

# Install browser dependencies
npx playwright install

The above commands install Playwright and download the necessary browser binaries for Chromium, Firefox, and WebKit, making them ready for headless operations.

Submitting Forms Programmatically​

Submitting forms programmatically is straightforward with Playwright. The page.fill method allows us to input data into form fields, and the page.click method is used to simulate clicking the submit button:

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: true });
  const page = await browser.newPage();

  await page.goto('https://demo.opencart.com/admin');

  await page.fill("input[name='username']", "demo");
  await page.fill("input[name='password']", "demo");

  await page.click("button[type='submit']");

  await browser.close();
})();

These methods ensure that the form is filled and submitted just as if a user were interacting with it. The headless mode of the browser means these actions happen in the background without any GUI.

Handling Browser-Specific Behavior and Quirks​

Different browsers can behave differently due to variations in their rendering engines and JavaScript implementations. Playwright’s support for multiple browser engines helps us address these quirks by allowing us to test our automation scripts across various browser environments.

For instance, to test the same form submission in Firefox, we only need to change the browser initialization:

const { firefox } = require('playwright');

(async () => {
  const browser = await firefox.launch({ headless: true });
  const page = await browser.newPage();

  await page.goto('https://demo.opencart.com/admin');

  await page.fill("input[name='username']", "demo");
  await page.fill("input[name='password']", "demo");
  await page.click("button[type='submit']");

  await browser.close();
})();

By running our scripts in code browsers, we can identify and handle browser-specific issues early in the development cycle.

Debugging with Slow Motion​

Playwright provides a slowMo option to slow down actions, making it easier to debug and observe automated interactions in real-time. This is particularly useful for troubleshooting and understanding the flow of automated tasks:

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: false, slowMo: 100 });
  const page = await browser.newPage();

  await page.goto('https://demo.opencart.com/admin');

  await page.fill("input[name='username']", "demo");
  await page.fill("input[name='password']", "demo");
  await page.click("button[type='submit']");

  await browser.close();
})();

Setting headless to false and adding slowMo slows down actions by 100 milliseconds, allowing one to observe each step of the automation process visually.

Advanced Techniques and Libraries​

In this section, we'll cover advanced techniques for form submission in NodeJS. By exploring these methods, we'll help you choose the most suitable approach for your projects based on performance and complexity.

Using node-fetch for Asynchronous Form Submission​

Node-fetch is a minimalistic, lightweight library for making HTTP requests in NodeJS using a fetch-like interface. It's particularly useful for asynchronous operations such as form submissions. Below is an example demonstrating how to use node-fetch for asynchronous form submission in NodeJS:

To begin, we import node-fetch using require('node-fetch'). This library provides the fetch function, which we utilize to send a POST request to a specific URL (https://practicetestautomation.com/practice-test-login/). The request includes headers specifying Content-Type: application/json and a JSON stringified body containing the username and password.

const fetch = require('node-fetch');

async function submitForm() {
    try {
        const response = await fetch('https://practicetestautomation.com/practice-test-login/', {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json',
            },
            body: JSON.stringify({
                username: 'student',
                password: 'Password123'
            }),
        });

        if (response.ok) {
            const data = await response.text();
            if (data.includes("Congratulations student. You successfully logged in!")) {
                console.log("User logged in successfully!");
            } else {
                console.log("Failed to log in.");
            }
        } else {
            console.log(`Failed to submit form. Status code: ${response.status}`);
        }
    } catch (error) {
        console.error('Error submitting form:', error);
    }
}

submitForm();

In this example, node-fetch is used to send a POST request to the login endpoint with the specified form data (username and password). The response is checked to determine if the login was successful based on the response content.

Utilizing node-fetch for Concurrent Form Submissions​

For scenarios requiring concurrent form submissions in NodeJS, node-fetch can handle multiple asynchronous requests effectively. Here’s an example demonstrating concurrent form submissions using node-fetch:

const fetch = require('node-fetch');

async function submitForm(username, password) {
    try {
        const response = await fetch('https://practicetestautomation.com/practice-test-login/', {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json',
            },
            body: JSON.stringify({ username, password }),
        });

        if (response.ok) {
            const data = await response.text();
            if (data.includes("Congratulations")) {
                console.log(`User ${username} logged in successfully!`);
            } else {
                console.log(`Login failed for ${username}`);
            }
        } else {
            console.log(`Failed to submit form for ${username}. Status code: ${response.status}`);
        }
    } catch (error) {
        console.error(`Error submitting form for ${username}:`, error);
    }
}

async function main() {
    const users = [
        { username: 'user1', password: 'Password1' },
        { username: 'user2', password: 'Password2' },
        { username: 'user3', password: 'Password3' }
    ];

    const promises = users.map(user => submitForm(user.username, user.password));
    await Promise.all(promises);
}

main();

In this example, node-fetch is used within a loop to concurrently submit form data for multiple users. In the main function, each user's credentials are passed to the submitForm function, which sends a POST request to the login endpoint. Concurrent execution is managed using JavaScript's Promise.all, ensuring all asynchronous operations have been processed.

Analysis of node-fetch​

Pros of node-fetch:

  • Provides a straightforward API similar to the browser's fetch API, making it easy to understand and use for developers familiar with JavaScript.
  • Handles concurrency well for moderate tasks. It efficiently manages multiple asynchronous requests, ensuring optimal performance without significant overhead.

Cons of node-fetch:

  • No native support for HTTP/2, which can be a limitation if you're looking to take advantage of HTTP/2's performance improvements, such as multiplexing and header compression.
  • Fewer built-in features (compared to, say, Axios). For instance, it lacks advanced timeout configuration, request cancellation, and interceptors.
  • Does not throw an error for HTTP responses with status codes 4xx or 5xx by default. Manual check is needed of the response status and errors, which can add extra boilerplate code.

For projects requiring simple yet effective asynchronous form submissions in NodeJS, node-fetch offers a reliable alternative to Axios. It strikes a balance between ease of use and performance, making it suitable for a wide range of web development tasks.

Conclusion​

In this guide, we explored various methods for submitting forms using NodeJS, each offering unique approach for different use case.

From the libraries we used, Axios stands out with its rich feature set and intuitive API, making it a popular choice for handling HTTP requests. Cheerio, when combined with Axios, excels at parsing and extracting data from HTML, making it a great tool for web scraping tasks involving form interactions. Playwright offers headless browser automation, ideal for simulating complex user behaviors and interactions with web forms.

By understanding the specific advantages and limitations of each method, you can decide which option to use for your specific needs.

For more information, visit the official documentation of these libraries:

More NodeJS Web Scraping Guides​

For more Node.JS resources, feel free to check out the NodeJS Web Scraping Playbook or some of our in-depth guides: