Contents

Web Scraping with PowerShell and Selenium

Automating Web Data Extraction and Analysis with PowerShell and Selenium WebDriver

Website Visitors:

Selenium is a popular open-source testing framework used to automate web browsers. It allows developers and testers to write scripts in various programming languages to interact with web pages, simulate user actions, and perform various testing scenarios. Selenium provides language-specific bindings for several programming languages, including Python, Java, C#, Ruby, and JavaScript.

PowerShell is a powerful automation and scripting tool for Windows operating systems. It provides a rich set of command-line utilities and scripting capabilities, making it an ideal tool for system administration, automation, and testing. In this article, we’ll discuss how to use Selenium with PowerShell, including how to import the module and use DLL files.

Prerequisites

Before we start, there are a few things that you will need:

  • A computer running Windows operating system

  • PowerShell installed on your computer

  • A web browser that is supported by Selenium (such as Google Chrome, Mozilla Firefox, or Microsoft Edge)

Installing Selenium with PowerShell

Before we can start using Selenium with PowerShell, we need to install the Selenium module. The easiest way to do this is by using the PowerShell Gallery. Here’s how to do it:

Open PowerShell as an administrator. Run the following command to install the Selenium module:

1
Install-Module -Name Selenium

Wait for the installation to complete.

Importing and Using Selenium module

Once we have installed the Selenium module, we need to import it into our PowerShell session. Here’s how to do it:

Open PowerShell. Import the Selenium module using the following command:

1
Import-Module Selenium

Now that we have imported the Selenium module, we can start using it to automate web browsers.

Here are examples demonstrating how to use the Selenium module in PowerShell:

1. Installing the Selenium Module:

1
Install-Module Selenium (or Import-Module Selenium)

2. Starting a Browser Driver:

1
2
3
4
5
6
7
8
# Start a Chrome driver
$Driver = Start-SeChrome

# Start a Firefox driver
$Driver = Start-SeFirefox

# Start an Edge driver
$Driver = Start-SeEdge

3. Navigating to a URL:

1
Enter-SeUrl https://www.example.com -Driver $Driver

4. Finding Elements:

1
2
3
4
5
6
7
8
# Find an element by ID
$Element = Find-SeElement -Driver $Driver -Id "myElementId"

# Find an element by name
$Element = Find-SeElement -Driver $Driver -Name "myElementName"

# Find an element by XPath
$Element = Find-SeElement -Driver $Driver -XPath "//button[text()='Click me']"

5. Clicking on Elements:

1
Click-SeElement -Element $Element

6. Sending Keystrokes:

1
Send-SeKeys -Element $Element -Text "Hello, world!"

7. Waiting for Elements:

1
Wait-SeElement -Driver $Driver -Id "myElementId"

8. Closing the Browser:

1
Stop-SeDriver -Driver $Driver

Using DLL files with Selenium

You can also import selenium DLL file and work with Selenium commands in PowerShell. To use a DLL file in PowerShell, you should download Selenium webdriver.dll file and use the “Add-Type” cmdlet. Here’s how you can do it:

Downloading WebDriver.dll file

Nagivate to https://www.nuget.org/packages?q=selenium+webdriver url. It will search for selenium webdriver and show you results. Download the first package which says Selenium.WebDriver. It will endup in .nupkg file extension. Using softwares like 7zip, extract all contents of that file to a folder. Next open the folder you’ve extracted the .nupkg file contents and navigate to Folder\lib\net5.0 (or whichever is latest in the list).

Here you will find WebDriver.dll file. If dotnet is installed in your machine, select same dotnet version folder from nupkg file and use the WebDriver.dll file. You have to experiment with which WebDriver.dll file works for your machine. In our testing, we’ve used net45\WebDriver.dll file which is for dotnet 4.5.

Downloading driver.exe file

Goto Selenium downloads page: Downloads | Selenium and download the driver file as per your browser version. If you’d like to work with chrome, download chrome driver file which is chromedriver_win32.zip. Extract it to the same folder you’ve used for WebDriver.dll file in above step. So, WebDriver.dll file and chromedriver.exe file are in same folder.

Open PowerShell on your computer. Type the following command to add the DLL file:

1
2
Add-Type -Path "C:\path\to\WebDriver.dll"
# Replace "C:\path\to\file.dll" with the actual path to the DLL file.

Press Enter to execute the command.

Examples

Now that you have added the dll file, let’s take a look at some examples of how to use Selenium with PowerShell. We’ve shown examples for chrome, and edge browsers. You have to download appropriate driver.exe files as per the browser you use.

Example 1: Filling out a Form

In this example, we will use Selenium to fill out a form on a webpage. Here’s the code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# Import Selenium DLL file 
Add-Type -Path "C:\path\to\file.dll"

# Create a new Chrome driver
$driver = New-Object OpenQA.Selenium.Chrome.ChromeDriver

# Navigate to the webpage
$driver.Navigate().GoToUrl("https://www.example.com/form")

# Find the text boxes and fill them out
$usernameBox = $driver.FindElementById("username")
$passwordBox = $driver.FindElementById("password")
$usernameBox.SendKeys("myusername")
$passwordBox.SendKeys("mypassword")

# In the website you open, username might be referred as user_name or id_user etc.. Hit F12, goto develper tools and check the correct id and use it in the commands above.
# Similarly follow the same for password column as well.

# Find the submit button and click it
$submitButton = $driver.FindElementByTagName("button")
$submitButton.Click()

# Close the browser
$driver.Close()

This code opens a webpage with aform that contains text boxes for username and password. It then uses Selenium to find these text boxes and fill them out with the values “myusername” and “mypassword”, respectively. Finally, it finds the submit button and clicks it to submit the form.

Example 2: Interacting with JavaScript

In this example, we will use Selenium to interact with JavaScript on a webpage. Here’s the code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
# Import Selenium DLL file 
Add-Type -Path "C:\path\to\file.dll"

# Create a new Edge driver
$driver = New-Object OpenQA.Selenium.Edge.EdgeDriver

# Navigate to the webpage
$driver.Navigate().GoToUrl("https://www.example.com")

# Execute JavaScript
$script = "document.getElementById('my-element').innerHTML = 'Hello, World!'"
$driver.ExecuteScript($script)

# Close the browser
$driver.Close()

This code opens a webpage and executes a JavaScript script that changes the innerHTML of an element with the ID “my-element” to “Hello, World!”.

Suggested Article

If you’d like to continue reading, checkout our other tech articles here or browse all our topics here.

Conclusion

In this article, we discussed how to use Selenium with PowerShell, including how to install the Selenium WebDriver module, import the module, and use DLL files. We also provided a simple example of how to use Selenium with PowerShell to automate the different browsers and extract information from a web page.

Selenium provides a powerful set of tools for automating web browsers, and PowerShell provides a rich set of automation and scripting capabilities. By combining these two technologies, developers and testers can create powerful and flexible testing scenarios to automate web-based applications.

As always, it’s important to use Selenium and PowerShell responsibly and ethically. Be sure to follow best practices for testing and automation, and always test in a controlled and isolated environment to avoid unintended consequences. With that in mind, happy testing!

We hope you have learned something new in this article. Please feel free to share your thoughts about this article in the comments section below.

Your inbox needs more DevOps articles.

Subscribe to get our latest content by email.