Effortlessly Mastering subprocess Popen
此内容尚未翻译成您的语言。
The subprocess Module: Wrapping Programs With Python
by Ian Currie
If you’ve ever wanted to simplify your command-line scripting or use Python alongside command-line applications—or any applications for that matter—then the Python subprocess
module can help. From running shell commands and command-line applications to launching GUI applications, the Python subprocess
module can help.
By the end of this tutorial, you’ll be able to:
- Understand how the Python
subprocess
module interacts with the operating system - Issue shell commands like
ls
ordir
- Feed input into a process and use its output
- Handle errors when using
subprocess
- Understand the use cases for
subprocess
by considering practical examples
In this tutorial, you’ll get a high-level mental model for understanding processes, subprocesses, and Python before getting stuck into the subprocess
module and experimenting with an example. After that, you’ll start exploring the shell and learn how you can leverage Python’s subprocess
with Windows and UNIX-based shells and systems. Specifically, you’ll cover communication with processes, pipes, and error handling.
Note: subprocess
isn’t a GUI automation module or a way to achieve concurrency. For GUI automation, you might want to look at PyAutoGUI. For concurrency, take a look at this tutorial’s section on modules related to subprocess
.
Once you have the basics down, you’ll be exploring some practical ideas for how to leverage the subprocess
module in your projects. You’ll learn about creating a new project and changing extended attributes. Additionally, this tutorial will introduce you to other Python modules associated with subprocess
, and walk you through using the Popen
class to connect two processes together with pipes and interact dynamically with a process.
Processes and Subprocesses
Processes and the Operating System
Before diving into Python’s subprocess
module, it’s important to understand what processes and subprocesses are, and how they relate to the operating system.
A process refers to a running program on your computer. Each process has a unique ID, known as a process ID, which is assigned by the operating system. You can think of a process as an instance of an application running on your computer. When you open a text editor or a web browser, each of these instances is a different process.
The operating system is responsible for managing processes. It oversees the creation, execution, and termination of processes. It also schedules the execution of multiple processes to ensure optimal use of system resources.
Process Lifetime
The lifetime of a process can be divided into different stages:
- Creation: A process is created when you start an application or run a command. The operating system creates a new process to execute the application or command.
- Execution: Once a process is created, it starts executing the instructions provided by the application or command.
- Termination: A process may terminate when it has finished executing its instructions or when it is terminated forcefully by the operating system.
Active Processes on Your System
To get a list of active processes on your system using Python, you can utilize the subprocess
module along with the appropriate command for your operating system.
Here’s an example that uses the ps
command on UNIX-based systems (like macOS and Linux) to list the active processes:
This code uses the run()
function from the subprocess
module to execute the ps
command (which lists the active processes) with the aux
option. The capture_output
parameter is set to True
to capture the command’s output, and the text
parameter is set to True
to indicate that the captured output should be returned as a string.
On Windows, the tasklist
command can be used instead:
This code uses the tasklist
command to list the active processes on Windows. The capture_output
and text
parameters have the same purpose as in the previous example.
By running these code snippets, you’ll see the output containing information about all the currently active processes on your system.
Overview of the Python subprocess Module
The subprocess
module in Python provides a way to create new processes, interact with them, and handle their input and output streams. It allows you to execute shell commands, run command-line applications, and even launch GUI applications.
To use the subprocess
module, you’ll need to import it at the beginning of your script:
Once you have imported the module, you can start exploring its various functions and classes.
Basic Usage of the Python subprocess Module
The subprocess
module provides several functions and classes for executing and managing processes. In this section, you’ll learn about some basic usage of the module.
The Timer Example
To get started with the subprocess
module, let’s create a simple example that demonstrates the run()
function. This function allows you to run a command and capture its output.
Suppose you want to measure the time it takes to execute a command. One way to do this is by creating a timer script that uses the run()
function to execute the command and captures the elapsed time.
Here’s an example that shows how to use the run()
function to measure the time it takes to execute the ls
command (which lists the contents of a directory):
In this code, the run()
function is used to execute the ls
command with the -l
option. The capture_output
and text
parameters are set to True
to capture the output of the command and return it as a string.
The start_time
variable is assigned the current time using the time.time()
function before the run()
function is called. Similarly, the end_time
variable is assigned the current time after the run()
function completes.
The elapsed time is calculated by subtracting the start_time
from the end_time
. Finally, the output of the ls
command and the elapsed time are printed.
When you run this code, you’ll see the output of the ls
command and the elapsed time it took to execute the command.
The Use of subprocess to Run Any App
The subprocess
module is not limited to executing shell commands like ls
. It can also run any program or application installed on your system, including Python scripts, compiled executables, and even GUI applications.
Here’s an example that demonstrates how to use the subprocess
module to execute a Python script:
In this example, the run()
function is used to execute the script.py
Python script. The python
command is used to run the script. The capture_output
and text
parameters are set to True
to capture the output of the script and return it as a string.
Similarly, you can use the subprocess
module to run other programs or applications by replacing 'python'
with the desired program or application name and providing any additional command-line arguments as needed.
The CompletedProcess Object
The run()
function returns a CompletedProcess
object that represents the completed process. This object contains useful information about the executed command, such as the return code, the captured output, and more.
You can access this information by assigning the result of the run()
function to a variable and accessing the attributes of the CompletedProcess
object.
Here’s an example that demonstrates how to access the return code and the captured output of the run()
function:
In this code, the returncode
attribute of the result
object is accessed to get the return code of the ls
command. The stdout
attribute is accessed to get the captured output of the command.
By running this code, you’ll see the return code and the captured output of the ls
command printed.
subprocess Exceptions
When working with the subprocess
module, it’s important to handle potential exceptions that may occur during the execution of processes. The module provides several exception classes to handle specific types of errors.
CalledProcessError for Non-Zero Exit Code
One common exception that you may encounter is the CalledProcessError
. This exception is raised when a process exits with a non-zero exit code, indicating an error.
To handle this exception, you can use a try
…except
block. Here’s an example that demonstrates how to handle the CalledProcessError
:
In this code, the run()
function is used to execute the ls
command with the nonexistent_dir
argument. Since the directory doesn’t exist, the command will fail, and a CalledProcessError
exception will be raised.
The check
parameter is set to True
to raise an exception if the command fails. If the check
parameter is set to False
(the default value), the exception won’t be raised, and you’ll have to manually check the return code to determine if the command succeeded or failed.
Inside the except
block, the attributes of the CalledProcessError
object are accessed to get the return code and the error output.
By running this code, you’ll see the error message and the error output printed.
TimeoutExpired for Processes That Take Too Long
Another useful exception class provided by the subprocess
module is TimeoutExpired
. This exception is raised when a process takes too long to complete.
You can specify a timeout value using the timeout
parameter of the run()
function. If the process doesn’t complete within the specified timeout, a TimeoutExpired
exception will be raised.
Here’s an example that demonstrates how to use the timeout
parameter and handle the TimeoutExpired
exception:
In this code, the sleep
command is used to pause the execution for 10 seconds. However, the timeout
parameter is set to 5
seconds. Therefore, the process will exceed the specified timeout, and a TimeoutExpired
exception will be raised.
Inside the except
block, the timeout
attribute of the TimeoutExpired
object is accessed to get the timeout value specified in the run()
function.
By running this code, you’ll see the timeout message printed.
FileNotFoundError for Programs That Don’t Exist
Sometimes, you may encounter a situation where you’re trying to execute a program or application that doesn’t exist on your system. In such cases, the subprocess
module raises a FileNotFoundError
exception.
To handle this exception, you can use a try
…except
block. Here’s an example that demonstrates how to handle the FileNotFoundError
:
In this code, the run()
function is used to execute a nonexistent_program
. Since the program doesn’t exist, a FileNotFoundError
exception will be raised.
Inside the except
block, a simple error message is printed.
By running this code, you’ll see the “Program Not Found” message printed.
An Example of Exception Handling
Now that you’re familiar with handling exceptions in the subprocess
module, let’s put it all together with a more comprehensive example:
In this code, the run()
function is used to execute the ls
command with the nonexistent_dir
argument. The capture_output
, text
, and check
parameters are set to capture the output, return it as a string, and raise an exception if the command fails.
The code includes except
blocks to handle the CalledProcessError
, TimeoutExpired
, and FileNotFoundError
exceptions.
By running this code, you can see the appropriate error message and output based on the specific exception that is raised.
Introduction to the Shell and Text-Based Programs With subprocess
In addition to executing shell commands, the subprocess
module can also interact with text-based programs and scripts. This allows you to leverage the power and flexibility of the shell and use Python to automate tasks, process data, and more.
Use Cases for the Shell and subprocess
There are many use cases where using the shell alongside Python can be beneficial:
- Automation: You can automate repetitive tasks by scripting command-line operations. For example, you can write a Python script that runs a series of commands to update files, process data, or perform system administration tasks.
- Integration: You can integrate third-party command-line tools or libraries into your Python applications. This allows you to leverage the functionality of these tools without having to reinvent the wheel.
- Data Processing: You can use text-based programs or scripts to process and manipulate data. Python can be used to automate the execution of these programs and handle their input and output.
To interact with the shell and text-based programs, the subprocess
module provides functions and classes that allow you to execute commands, handle input and output, and even automate user interactions.
Basic Usage of subprocess With UNIX-Based Shells
When working with UNIX-based shells (like the Bash shell on macOS and Linux), you can execute shell commands directly using the run()
function.
Here’s an example that demonstrates how to use the run()
function with a UNIX-based shell:
In this code, the echo
command is used to print the string "Hello, World!"
. The run()
function is called with the command as a list and the capture_output
and text
parameters set to capture the output as a string.
By running this code, you’ll see the output of the echo
command printed.
Basic Usage of subprocess With Windows Shells
When working with Windows shells (like Command Prompt or PowerShell), you can use the run()
function with a slight modification. Instead of passing the command as a list, you need to pass it as a single string.
Here’s an example that demonstrates how to use the run()
function with a Windows shell:
In this code, the echo
command is used to print the string "Hello, World!"
. The command is passed as a single string to the run()
function. The capture_output
and text
parameters are set to capture the output as a string, and the shell
parameter is set to True
to use the Windows shell to execute the command.
By running this code, you’ll see the output of the echo
command printed.
A Security Warning
When using the run()
function with a shell, be cautious when passing user-supplied input as part of the command. Never use string concatenation or interpolation to build the command string, as it can potentially lead to security vulnerabilities.
For example, consider the following code:
In this code, the user is prompted to enter a filename. The supplied filename is then passed directly to the cat
command using string interpolation. This code is dangerous because it allows the user to execute arbitrary commands by manipulating the input.
To prevent this, you should sanitize and validate any user input that is used in commands. One way to do this is by using the shlex.quote()
function to properly quote any user input, ensuring it is treated as a single argument. Here’s an example that demonstrates how to sanitize user input:
In this code, the shlex.quote()
function is used to properly quote the user input. This ensures that the user input is treated as a single argument and prevents any potential command injection attacks.
By using proper input sanitation and validation techniques, you can reduce the risk of security vulnerabilities when interacting with shells and text-based programs.
Communication With Processes
The subprocess
module provides several ways to interact with processes, including reading and writing to their standard I/O streams. This allows you to communicate with processes and automate tasks that require input and receive output from external programs.
The Standard I/O Streams
When a program or application is executed, it has three standard I/O streams associated with it:
- Standard Input (
stdin
): This stream accepts input to the program. By default, it is connected to the keyboard, allowing the user to provide input interactively. However, it can also be redirected to read input from a file or a pipe. - Standard Output (
stdout
): This stream represents the normal output of the program. By default, it is connected to the console, allowing the program to print output that the user can see. However, it can also be redirected to write output to a file or a pipe. - Standard Error (
stderr
): This stream is used to print error messages and diagnostic output. By default, it is also connected to the console, separate from the normal output. However, likestdout
, it can be redirected to write output to a file or a pipe.
The subprocess
module provides ways to access and communicate with these standard I/O streams of a process, allowing you to automate tasks and handle input and output.
The Magic Number Generator Example
To demonstrate communication with processes, let’s create a simple example that generates a magic number. The magic number is a random integer between 1 and 100, and the program waits for the user to guess the number. It provides feedback based on whether the guess is too high, too low, or correct.
Here’s an example that uses the subprocess
module to communicate with the magic number generator:
In this code, the magic_number_generator.py
script generates the magic number. The subprocess.Popen()
function is used to create the process, specifying the command to run as a list.
Inside the while
loop, the user is prompted to enter a guess. The guess is sent to the process using process.stdin.write()
and process.stdin.flush()
. The response from the process is read using process.stdout.readline()
, stripped of leading and trailing whitespace using strip()
, and printed.
The loop continues until the guess is correct, as indicated by the response.
Finally, the process is closed by closing the stdin
and stdout
pipes, and waiting for the process to finish using process.wait()
.
When you run this code, you’ll be prompted to enter guesses until you guess the correct number, and you’ll see feedback from the process indicating whether your guess is too high, too low, or correct.
The Decoding of Standard Streams
When interacting with processes in Python, it’s important to properly handle the encoding and decoding of the standard I/O streams. By default, these streams use the system default encoding, which may not be suitable for all situations.
To specify a different encoding for the standard I/O streams, you can use the encoding
parameter of the subprocess.Popen()
function. This parameter specifies the text encoding to use, such as 'utf-8'
or 'latin1'
. If you’re unsure which encoding to use, you can use 'utf-8'
, which is commonly used and supports a wide range of characters.
Here’s an example that demonstrates how to specify the encoding for the standard I/O streams:
In this code, the encoding
parameter is set to 'utf-8'
to use the UTF-8 encoding for the standard I/O streams of the process.
By specifying the correct encoding, you can ensure that the input and output of the process are properly encoded and decoded, preventing any potential encoding errors.
Reaction Game Example
To further illustrate communication with processes, let’s create another example: a reaction game. In this game, the program waits for a ready signal and measures the time it takes for the user to press a key. The program then provides feedback on the reaction time.
Here’s an example that demonstrates how to use the subprocess
module to communicate with the reaction game:
In this code, the reaction_game.py
script implements the reaction game. The subprocess.Popen()
function is used to create the process, specifying the command to run as a list.
The program waits for the ready signal from the process using process.stdout.readline()
, strips leading and trailing whitespace using strip()
, and prints the signal.
The user is prompted to press the Enter key to start the game. The start time is recorded using time.time()
.
The start signal ("start\n"
) is sent to the process using process.stdin.write()
and process.stdin.flush()
.
The result of the game is read from the process using process.stdout.readline()
, stripped of leading and trailing whitespace, and printed.
The end time is recorded using time.time()
, and the reaction time is calculated by subtracting the start time from the end time.
Finally, the process is closed by closing the stdin
and stdout
pipes, and waiting for the process to finish.
When you run this code, you’ll see the ready signal, prompted to press Enter, and provided with feedback on your reaction time.
Pipes and the Shell
The subprocess
module provides ways to create and manage pipes, allowing you to connect processes together and transfer data between them.
Introduction to Pipes
A pipe is a communication channel between two processes. It allows the output of one process to be connected to the input of another process. Data written to the output of the first process is received as input by the second process.
Pipes are commonly used to connect processes together to form pipelines. A pipeline is a series of connected processes, where the output of one process is piped directly into the input of the next process.
By using pipes, you can combine the functionality of multiple processes and perform complex tasks by chaining them together sequentially, each process consuming the output of the previous process and producing output for the next process.
The Pipes of subprocess
The subprocess
module provides several functions and classes to create, manage, and connect processes with pipes.
To create a pipe, you can use the subprocess.PIPE
constant as the value for the stdin
, stdout
, or stderr
parameter of the subprocess.Popen()
or subprocess.run()
functions. This designates that the respective standard I/O stream should be connected to a pipe.
Here’s an example that demonstrates how to create a pipe and connect processes together:
In this code, two processes are created: process1
and process2
.
process1
is created using subprocess.Popen()
with the stdout=subprocess.PIPE
parameter, which creates a pipe for the standard output of the process. This allows the output of process1
to be received as input by process2
.
process2
is created using subprocess.Popen()
with the stdin=process1.stdout
parameter. This specifies that the standard input of process2
should be connected to the output of process1
. This effectively directs the output of process1
to the input of process2
.
The stdout
pipe of process1
is closed using the stdout.close()
method, as it won’t be used anymore.
The communicate()
method of process2
is called to wait for process2
to finish and get its output. The [0]
index is used to access the output, as communicate()
returns a tuple containing the output and the error output.
Finally, the output is printed.
When you run this code, you’ll see the output of process2
, which contains the line "Hello, World!"
as it matches the pattern "World"
.
Pipe Simulation With run()
The subprocess
module also provides the subprocess.run()
function, which is a simplified interface for executing commands and capturing their output. This function can also be used to simulate pipes by connecting processes together.
Here’s an example that demonstrates how to use subprocess.run()
to simulate pipes:
In this code, the echo
command is executed using subprocess.run()
with the capture_output=True
and text=True
parameters. This captures the output of the echo
command and returns it as a string.
The captured output is then passed as the input to the grep
command using the input
parameter of subprocess.run()
. This effectively simulates a pipe, directing the output of the echo
command to the input of the grep
command.
The output of the grep
command is captured and printed.
When you run this code, you’ll see the output of the grep
command, which contains the line "Hello, World!"
as it matches the pattern "World"
.
Practical Ideas
In addition to the basic usage of the subprocess
module, there are many practical ideas for leveraging it in your projects. Here are a couple of examples:
Creating a New Project: An Example
When starting a new project, you often need to set up a directory structure and create initial files. Instead of manually creating each file and directory, you can automate this process using the subprocess
module.
Here’s an example that demonstrates how to create a new project using the subprocess
module:
In this code, the user is prompted to enter a project name using input()
. The project directory is created using os.mkdir()
, and the process’s working directory is changed using os.chdir()
.
The src
directory is created inside the project directory using os.mkdir()
.
The main Python file is created using open()
in write mode, and the README.md
file is created in a similar way. The contents of the files are written using the file object’s write()
method, which creates the initial content of the files.
Finally, the project directory is opened in the default file manager using subprocess.run()
and the open
command.
When you run this code, you’ll see a new project directory created with the specified name. Inside the directory, you’ll find a src
directory containing a main.py
file, and a README.md
file with the project name as the heading.
Changing Extended Attributes
Extended attributes are additional metadata associated with files and directories on certain filesystems. They provide a way to attach additional information to files and directories beyond the standard attributes like permissions and timestamps.
The subprocess
module can be used to interact with the xattr
command-line tool on macOS and Linux systems, allowing you to manage extended attributes programmatically.
Here’s an example that demonstrates how to change extended attributes using the subprocess
module:
In this code, the xattr
command is used to set an extended attribute on a file using subprocess.run()
with the appropriate parameters.
The value of the extended attribute is then retrieved using subprocess.run()
with the -p
option.
The output is captured and printed.
To run this code, you need to have the xattr
command-line tool installed on your system. You can install it using the appropriate package manager for your system (brew
for macOS, apt
for Ubuntu, etc.).
When you run this code, you’ll see the output of the xattr
command, which is the value of the extended attribute associated with the file.
Python Modules Associated With subprocess
The subprocess
module is a powerful tool for executing and interacting with external processes. However, it’s not the only Python module available for these tasks. There are several other modules you can explore for more specific use cases or expanded functionality.
Here are a few notable Python modules associated with subprocess
:
- shlex: The
shlex
module provides a way to parse shell-style strings into tokens, making it easier to work with shell commands and scripts. - argparse: The
argparse
module allows you to define and parse command-line arguments and options, making it easier to create command-line interfaces for your Python scripts. - pty: The
pty
module provides a way to run a command in a pseudo-terminal, which can be useful when interacting with programs that expect to be run in a terminal environment.
By exploring these modules and the subprocess
module, you can expand your capabilities when working with external processes and creating command-line interfaces in Python.
The Popen Class
The Popen
class is another important component of the subprocess
module. It provides more control and flexibility when working with processes, allowing you to perform advanced process management and handle complex scenarios.
Using Popen()
To create a new process using the Popen
class, you need to instantiate the class and pass the command to run as a list to the constructor. You can also specify additional parameters to customize the behavior of the process.
Here’s an example that demonstrates how to use the Popen
class to create a new process:
In this code, the Popen
class is instantiated with ['echo', 'Hello, World!']
as the command to run. The stdout=subprocess.PIPE
parameter specifies that the standard output of the process should be captured and returned as a string. Similarly, the stderr=subprocess.PIPE
parameter specifies that the standard error of the process should also be captured.
The communicate()
method of the process
object is called to wait for the process to finish and get its output. The output and error output are returned as tuples, with the output assigned to output
and the error output assigned to error
. In this example, since the command echo
doesn’t produce any error output, error
will be an empty string.
Finally, the output is printed.
When you run this code, you’ll see the output of the echo
command printed.
Connecting Two Processes Together With Pipes
One powerful feature of the Popen
class is the ability to connect two processes together with pipes. This allows you to build complex pipelines and pass data between processes.
Here’s an example that demonstrates how to connect two processes together with pipes using the Popen
class:
In this code, two Popen
instances are created: process1
and process2
.
process1
is created with ['echo', 'Hello, World!']
as the command to run and stdout=subprocess.PIPE
to create a pipe for the standard output of the process.
process2
is created with ['grep', 'World']
as the command to run and stdin=process1.stdout
to connect the standard input of process2
to the output of process1
’s pipe.
The stdout
pipe of process1
is closed using the stdout.close()
method, as it won’t be used anymore.
The communicate()
method of process2
is called to wait for process2
to finish and get its output. The error output is discarded using the _
variable, as the echo
command doesn’t produce any error output.
Finally, the output is printed.
When you run this code, you’ll see the output of process2
, which contains the line "Hello, World!"
as it matches the pattern "World"
.
Interacting Dynamically With a Process
The Popen
class also allows you to interact with a process dynamically, sending input and receiving output as needed. This can be useful for long-running processes or interactive programs.
Here’s an example that demonstrates how to interact dynamically with a process using the Popen
class:
In this code, the Popen
class is instantiated with ['python', '-i']
as the command to run. The -i
option starts the Python interpreter in interactive mode, allowing multiple commands to be executed.
After creating the process, input is sent to the process using process.stdin.write()
and process.stdin.flush()
. These methods write the input to the process’s stdin pipe and ensure the data is flushed and sent immediately.
The output of the process is read using process.stdout.readline()
, which reads a line of output until a newline character is encountered. The leading and trailing whitespace is stripped using strip()
.
Finally, the output is printed. The stdin pipe is closed using process.stdin.close()
, and the process is waited for using process.wait()
.
When you run this code, you’ll see the output of the process, which contains the line "Hello, World!"
as it was executed in the interactive mode.
Conclusion
The subprocess
module in Python provides a powerful way to interact with external processes, allowing you to execute shell commands, run command-line applications, and even launch GUI applications. With the knowledge and techniques covered in this tutorial, you can now leverage the subprocess
module to simplify your command-line scripting, automate tasks, and handle input and output for external programs.
By understanding the basics of processes and subprocesses, exploring the Python subprocess
module, and examining practical examples, you can enhance your Python skills and become more proficient in working with external processes in your projects.